I am typing this comment from a Debian Squeeze box with XFCE (although I'm actually typing from a VNC session on Fluxbox). This installation does not and never did have GNOME installed. GNOME is only the default environment in Debian. You can also pick KDE or XFCE from the installation CD. The GNOME version is no more customized than the XFCE one. Alternatively, you can do a command line installation and only add X-Window plus whatever window manager or desktop environment suits your fancy.
Interestingly, the Debian installation that has been put together for the hacked Zipit Z2 includes only Fluxbox as a window manager unless you install something else later. Debian is pretty flexible, but as Letalius Sonus said, it's not targeted specifically toward Armv7-A, but toward running on just about any Arm processor.