I have studied a variety of Asian languages and would say that Chinese is difficult when it comes to the characters and pronunciation. Chinese grammar is similar to English so it is not that difficult.
"Wo de zhongwen bu tai hau" ^^
Reading and memorizing characters takes time, but if you do it everyday practicing reading and writing, it will become easier, but you have to do it everyday.
Pronunciation is the most difficult as there are 4-5 different tones for the same sound so if you say a particular sound in the wrong tone, it can have a completely different meaning (eg: "ma" could mean horse, mother, etc). However once you train your tongue, it should be no problem.
Listen to lots of Mandarin/Taiwanese pop music and watch movies/TV to improve your listening.
I would say Japanese is quite easy except for grammar.
Korean is one of the easiest to learn but grammar is similar to Japanese and pronunciation is very difficult, even more than Chinese.