All I am saying here is, you can DO it with the right amount of effort, but:
1. Ubuntu will be big. Ubuntu will be slow to start up. And the usual software suite it uses, like Gnome, KDE etc. will hog lots of RAM. These are all very bad things for a handheld's battery life and ability to run...