I tried that, it's pretty tightly bound to the whole process.
Looks like what it does is uses imake to build a world makefile which does a bunch of stuff, builds some headers and compiles some code, then crafts a configure script in the Xvnc directory. You then run the ./configure which uses imake to build a Makefile. I tried just copying the Makefile but it's riddled with x86 defines and requires some 64 bit headers.
I've seen automakeception, where you'll get 2 or 3 layers of different (or sometimes the same) make makers that must all be called sequentially, but this is the first time I've seen a circular make system. It's impossible to break out of and is doing nothing to make me re-evaluate my stance on automakes.
Ok, did take a quick look. Imake is still part of X and the xorg version is straightforward to compile.
Unfortunately it is too new for the 6.3 based tigntvnc 1.3. But that actually already includes a better suited version, though the documentation is suboptimal. The build finishes, but I don't have time to test the result. Here is the recipe if someone wants to follow up, though mcobit's buid is probavbly better. I used the very nice C dev tools PND from freamon.
1. cd into Xvnc
2. let configure create config/cf/platform.def (the rest can be disabled)
3. cd into config/imake
3a. fix the embedded cpp program name in imake.c or imakemdef.h
4. build imake using the bootstrap Makefile with 'make -f Makefile.ini'
5. cd into ../cf
6. fix up linux.cf (at least the architecture detection and DefaultGccIncludeDir)
7. cd into ../makedepend
8. do '../imake/imake -I../cf -DTOPDIR=../.. -DCURDIR=.', then 'make'
9. cd into ../../../libvncauth
10. do '../Xvnc/config/imake/imake -I../Xvnc/config/cf -DTOPDIR=../Xvnc -DCURDIR=.', 'make Makefiles', 'make includes', 'make depend', and finally 'make'
11. cd into ../Xvnc
12. do 'config/imake/imake -Iconfig/cf -DTOPDIR=. -DCURDIR=.', 'make Makefiles', 'make includes', 'make depend', and finally 'make'
13. wait
It builds fine for me and leaves Xvnc in programs/Xserver. Don't know what is required to run it. If there is interest, I could tar the whole directory tree.
edit: just remembered the missing step 3a.