Djwillis recently documented using OE to build a few official firmware options; one of the templates he's included is a basic command-line only firmware, if you're interested.
Its actually pretty easy to boot up alternate distributions, but it takes a lot of work to tweak them to be useful.
ie: You could get a recent Debian ARM or Ubuntu ARM (for example), drop in the Pandora kernel and have _Something_ going pretty fast. But removing the unapplicable software so you're not eating up all your RAM etc, thats where the work begins.
There should be a number of guides for BeagleBoard around which will help out.
Are you aware of .. when booting, hold down right-trigger to get a bootmenu, which lets you specify 'boot from SD' as one of the options; with a properly set up SD, you'll be able to tweak and edit your firmware on SD, pop it into your pandora and boot, and see how it goes, without risking your NAND firmware. Great stuff.
If you like, we cna dig up some boot-scripts and such so you cna just boot your kernel off one partition (a small one), and then have it go right into your firmware on another partition of the same SD, and just unpack your big firmware into there, for tweaking away. It works really well.
I've got to run right now, but ask lots of questions, we'll fill you in
(heck, you could go right minimal, a few of us have, especially notaz; boot kernel off the SD, and have it drop right to a shell, with your OS only being 'ls' and a copy basic tools; I've even done that, booting right into Minimenu
jeff