Did some research. X11VNC is only capable of connecting to real displays. You'll need to install vncserver if you want a virtual X server with a different resolution.If you're running off SD card this is trivial as doing:
sudo opkg update
sudo opkg install vncserver
vncpasswd
vncserver :1 -geomertry 1024x768
This will start a VNC/X session server on display 1 with a resolution of 1024x768. You could choose any resolution you want of course. There will be no window manager, but just an xterm to begin with. If you want a full XFCE experience. You'll need to edit the ~/.vnc/.xstartup file and include the line:
kill your current session and restart it to have a full dekstop.
Code:
vncserver -kill:1
vncserver :1 -geomertry 1024x768
While untested, I am almost 100% certain this will work. Have fun!