Cannot connect to Xen domU via VNC if X isn’t installed on domU

I'm trying to build a Xen domU that can be connected through the Xen's VNC server. Below is the template (actually it's generated by OpenNebula):

name = 'one-153'
#O CPU_CREDITS = 256
memory  = '128'
bootloader = "/usr/bin/pygrub"
disk = ['tap:aio:/home/oneadmin/cloud/one/var/153/images/disk.0,xvda,w',]
vif = ['mac=02:00:c0:a8:00:03,bridge=virbr0',]
vfb = ['type=vnc,vnclisten=slave1,vncdisplay=1,vncpasswd=v98KXdFN']

The problem is that I can't connect to the domU if it doesn't have X. In this case all I got is a blank screen. Besides, if the domU has X, the screen is still blank until the login prompt appears, while it should be like this.

Some information that may be useful:

• The domU and the dom0 both run CentOS 5.5.
• If the domU has X, it can be connected even when both X and the domU's VNC server isn't running.
• The VNC client is RealVNC.