Although both the Original Quake and DN3D on the PC were rendered in software, one of the huge differences between Quake and Duke, was that Duke was not really 100% true 3D. I say this in the way that Duke still used sprites for lots of things such as the wee baddies, weapons and various items. Quake on the other hand rendered the lot, everything on screen was a true 3D object and that's why Quake requires more power to run.
You can see what I mean about sprites by comparing the following: In Duke 3D, look at something you can pick up such as a weapon, you will see it always faces you. In Quake you can walk around them properly. Mind you, the original Quake on the PC allowed you to run it in different resolutions so it might have looked a bit better than on the GP2X screen. There is an option on the GP2X Quake called "Double Size" Make sure this is set to off, or things will look really blocky.
Comparing the GP2X Quake to the X-Box or PSP or even newer ports of Quake on the PC such as GL-Quake isn't really fair, as the newer ports are using 3D hardware and not rendering it in software. On newer ports you're getting not only higher resolutions, but better lighting and atmosphere effects, it's bound to look better. What you're getting on the GP2X is the original Quake as it was released on the PC, and to be honest, it runs a lot smoother on the 2X than on my old PC when I first got Quake, especially if you overclock the 2X.