Video can only be in an .avi container. XviD and DivX 3, 4, 5, and 6 (I believe) are known to be compatible video codecs. I believe the audio has to be mp3; I'm not sure if there are any other audio codecs supported like PCM or ADPCM, but I doubt they are. AC3 (Dolby Digital) is popular for movies and isn't supported.
With the current firmware (1.2.1), there are audio sync problems with most files, and DivX 3 has had color issues, but these problems will be fixed in the next firmware update (the beta 1.4.0 has them fixed, I believe).