[hennaz]

There are 3 categories of "mainstream" languages taught, category 1 having the easiest languages for an English-speaker to learn, category 3 having the hardest.
Category 1 includes: Afrikaans, Danish, Dutch, French, Italian, Norwegian, Portuguese, Romanian, Spanish and Swedish. They have more similarities to English, and take about 24 weeks (600 class hours) to gain at least some proficiency. The hardest languages of this category are German, Indonesian, Malay and Swahili, and may take up to 36 weeks.

Category 2 includes: Albanian, Armenian, Azeri, Bengali, Bulgarian, Czech, Finnish, Greek, Hebrew, Hindi, Hungarian, Icelandic, Latvian, Lithuanian, Mongolian, Persian, Polish, Russian, Serbian, Tagalog, Thai, Turkish, Ukrainian, Urdu and Vietnamese. These have significant linguistic and/or cultural differences from English, and 44 weeks of learning are needed for any proficiency.

Category 3 includes: Arabic, Chinese (Mandarin/Cantonese), Japanese and Korean. They are considered highly difficult for speakers of Western languages to learn, and may take up to 88 weeks for basic proficiency (preferably half the time spent studying in the country which speaks that language).
Don't let this discourage you. If you want to learn Japanese, go for it! Yes there are difficulties, but the grammar and conjugations aren't as hard as say, French, because unlike French, there are no genders, only 2 irregular verbs (suru and kuru) and no imperfect/subjunctive/pluperfect etc. I don't know whether the kanji or the completely-alien vocabulary is the hardest part. In any case, Japanese is certainly not impossible to learn.