With the average size of roms as well as homebrew, you don't exactly need huge storage space, SD cards offer plenty of space. PSX would be the exception but it's quite a long shot for that ever to really work well.
Besides that, hard drives are still quite vulnerable to shock, even though they've been improving. Now I wouldn't suggest you drop your GP2X in the first place, but when you do, chances of survival are a lot bigger without a hard drive in it.
Also, you may not care about cost, but in the big picture GPH has to keep the cost down since a lot of people will care, and it's not like they can sell the handheld at a loss and profit from the games (what games) like other companies do.