As a GP2X owner, but far from a fanboy (I'm looking forward to the XGP, and if the hardware in the PSP appealed to me more, I might want to buy one as well) .. I feel I should post my experiences with the machine and help set the record straight between the 'extremists' on both sides.
Raw Performance: The most important bit to me. The machine has a MagicEyes SoC (System on a Chip) containing 2 ARM9 cores- an ARM920T and an ARM940T, both rated for 200 MHz. I am able to run my ARM920T core, the one that runs the vast majority of software, at up to 312 MHz. At this speed, the performance is astouding-- and it's amazing at 200 MHz as well. Nearly all emulators run at or near full speed-- only the most extreme platforms to emulate usually experience any serious slowdown at all. I'm developing for the machine, and I'm blown away by what it can handle. It also bears mentioning the ARM9 is one of my absolute favorite processor architectures, and this was the ideal choice for the machine as far as I am concerned as far as balance betweeen performance and battery life is concerned.
The second CPU, the ARM940T, has no MMU and less cache so it's fairly specialized in what it can do- it takes a lot of programming finesse but it has some great uses, like assisting video encoding or some emulation assistance. Developers are figuring out how to use it, and I am extremely confident libraries will soon exist to run common tasks on the 940T like audio/image decoding/handling to lift some burden off the main ARM920T. Overclocking the 920T is simple and painless and can be done in software at any time. You can also underclock for better battery life.
Memory: One of the biggest advantages the GP2X has-- it has an enormous amount of RAM available. There is 64MB total, and only about 8MB is used by the OS. So you have about 56MB of RAM to play with as a developer-- that's incredible, and a big part of why NeoGeo Arcade emulation is possible on the GP2X despite how huge such games are. This is a godsend for me, as a developer using a LOT of memory-intensive graphics AND music! When your handheld has as much memory as an XBox, you've got something impressive.
Storage: The onboard NAND memory is 64 MB, 32 MB of which can be used for storage and 32 MB is reserved for the basic kernel and programs. The fact that it uses such a cheap and readily available format, for which I have a reader on every one of my machines (and my camera uses SD!) this is really ideal. My 1GB SD card cost me very little ($50.) This is nice compared to Memory Stick especially.
Graphics/GFX Performance: The '2x contains a 2D blitter which speeds up 2D graphics operations (Drawing, filling, etc.) dramatically. It also contains 2 scaling units that can resize/zoom and flip the display so applications that are not the native resolution of the display (320x240) can run just fine. There is no hardware 3D support, but given the lack of good, gameplay-rich 2D games as of late compared to the industrywide influx of less creative, less inspired 3D titles, I can do without the pretty realistic visuals for impressive and immersive gameplay. Like the good old days. Lack of a 3D GPU also aids battery life.
Speaking of graphics-- the LCD. Only the earliest MK1 units had the 'flickering' issue, it was rapidly fixed and resellers offered to replace units with the issue. Later MK1 units like mine have it fixed and come from a much better factory, with higher quality fabrication. MK2 units have a whole new LCD with a crisper appearance. There is a 'scanline' effect on MK1 displays (not present on MK2s) but it can be eliminated with a quick 10-second adjustment in the settings menu, which never needs to be done again. The display is bright, crisp and not blurry at all. Much better than I expected for the price.
Audio: Fantastic audio quality; even the 2 onboard speakers sound good. You can of course attach your own headphones or speakers easily and they sound superb; as good as my PC does easily. The built-in audio player is simple and to-the-point and plays all my Mp3 and OGG files without a hitch. It's not very fancy but it does offer some seeking and equalizer functionality; and apparently the new version coming out with the next firmware update adds such things as playlist support (excellent.)
Video: The machine has built-in MPEG4 decoder hardware that supplements the ARM920T, and works with the ARM940T. Even high-definition resolution video (e.g. 1080, 720 lines tall video) plays at a smooth framerate, usually 30 FPS. The machine can play MPEG4 video or DivX and XViD, which is about all I encode to anyway. You can play AVI containers or OGM ones (renamed to AVI). The player is open source so new formats and codecs are being added by community members at the moment. Battery life is much decreased when using the video player due to the extra hardware that's engaged, but you can put a big dent in that by just using videos encoded to 320x240 so the battery-hungry scaler doesn't need to do anything- you can comfortably view videos for quite a few hours on 2 rechargable NiMH AAs.
Photos: The built-in photo viewer is nice and has an integrated explorer, can flip images to view them at different angles, zoom in or out. This is especially nice for me since my camera stores JPEG photos on SD cards, so I can see a big, high-quality copy of my photos the moment I take them! (My laptop has an integrated SD card reader too, so this is extra-convenient.) There are some bugs where the player crashes reading some files, but the majority work fine. It even supports very large images (I think it tops out around 3000px across. But that's pretty reasonable.)
Controls: As has been said the control stick is on the sketchy side and isn't as precise as I'd like, but is much improved with the custom cap that I ordered and with the new one that is provided with all new units (you can also order one separately.) I'm replacing my stick with a D-Pad shortly with a kit provided by a community member, and other replacement kits are in the works. For most games though, the stick is really just fine, and you get used to it amazingly fast. I'm just a picky bastard. The rest of the buttons are great; very comfortable with good response-- the shoulder buttons could use to 'click' more but this is fine; it doesn't hinder me in any way.
Battery Life: I get about 5-6 hours of life playing very stressful emulators like NeoGeo or SNES, and can cram in some audio and photo viewing in there too-- this is on some cheap $15-for-a-pack-of-4 2800MAh Duracell NiMHs or 2500 MAh Energizer NiMHs. On a pair of Lithium AAs it lasts at least 9 hours on average, I stock up on those when I can't recharge. Where there's a wall outlet, I use my tiny compact AC adapter, works perfectly. The cord plugs into the side where it's a bit odd, but it acts like a bit of a 'grip' for me so I don't mind it. As long as I have 4 AAs with me I never have any sort of power issue.
Software: The emulators and default apps for media playback are great, but that's just the tip of the iceberg. There are lots of great ports like classic FPS'es-- Quake, Wolfenstein 3D, Doom, Rise of the Triad, Duke Nukem 3D and Marathon for example. There are ports of great 2D PC games like Noiz2sa and the "ScummVM"-supported Lucasarts adventure titles. There are LOTS of homebrew games in development and released, like the fantastic arcade-style Vektar or a nice selection of others. The best stuff is in development and shaping up nicely.
Homebrew: Developing for the machine is a breeze; I had never used SDL or coded for a handheld Linux machine prior to my existing projects, but it was a piece of cake to get used to it. You just compile your applications on your PC with easy-to-use SDK tools (or an IDE) and copy your executable to your SD card, or run it over the USB cable via Telnet, copy it over via Samba (yes, it CAN share files with Windows.. or act as an FTP or HTTP server) and run it. Very, very easy.
At the moment I'm the programmer for a team developing a very intense 'danmaku-style' top-scrolling shooter called Stargazer (a.k.a. Project Luminance), and an FPS, Armor of Acheron G, as well as an air combat sim (based on Quake), both with single-player and multi-player support; Stargazer to support multiple players via USB controller connectivity, especially useful with the GP2X's TV output!
I really couldn't be much happier with the machine; it's few quirks are mostly minor issues. I can see the negativity from some people who weren't ready for the risks inherent to being an early adopter and got one of the crummier models from the first production run which didn't last very long, but that really comes with the territory. Nowadays, things are pretty peachy in GP2X-land. If you are considering getting a GP2X, you'd be doing yourself a favor to do so.