Well, it depends on what you do with X. I don't use GNOME or KDE, and X starts up pretty quickly. However, then again, that's on a 1.6 Ghz processor.
My dedicated server is 500Mhz, 96 MB of RAM. I hooked up a head to it and ran X on it at one point, just for kicks, and it ran Fluxbox just...