Much like Trooper, the initial draw to the GP32, was emulation. It can emulate 8bits, and 16bits for the most part, and being an Atari nut means the XL/XE 8bit line up, and ST 16bit is taken care of quite well. Which also means, I don't have to chase after newer open source hand helds, because the GP32, has already done what I want.
My fetish aside, and in addition to Trooper's too, there is MAME, the ROM list is not too bad, as I like older games, I am happy with a few I have on a dedicated SMC. Also I have similar for the Gameboy emulator as well.
Amongst others there is Windups, a nice little front end. We also have GPBook (ebook reader), it reads text file, and is compatible with Notepad, you can put alot of reading material even on a small card. Almost forgot, there is Little GP Tracker (LGPT), a sequencer for you creative side (you can use this with a custom MIDI cable for contolling MIDI instruments too). You can listen to MP3, but why worry spending money on an SMC card for that if you could spend the money better on a dedicated MP3 player (but it is nice to have the ability). Also there is GPCinema for watching the old DIVX 4.12 format encoded video (I never tried it myself, but did see one playing and it was good).
On my current SMC card (this one is a 64MB), I have:
Tetris (Puzzle - Im sure you already know this game)
Sokoban (Puzzle - an old favourite of mine)
GP32 War (War game - wish I had more time for this one)
GP32 Sopwith (Side ways scroller - old but I like it for a quick sortie)
NetHack (RPG - old skool)
Aka Noid (Breakout - for a quick fix)
Bolcataxian (Vertical shooter - great blasting game)
Bolcatoid (Breakout - I like breakout)
Arcano (Breakout - hold the unit sideways)
Newkind (Elite - an unfinished game, but hey it's Elite)
Pinball Dreams (Pinball - like on the old Amiga 16bit)
Trooper listed a few of the must have games to try. They show the quality the GP32 can put out.
Must agree about the SMC prices.
EDIT: Typos (Torpor 2 Trooper).
RE-EDIT: Another one!