Linux – How to properly setup X forwarding on (Ubuntu) linux

VNC will do this for you. It's the Unix version of remote desktop.

It's been a while since I asked this question, so I thought I'd mention the solution we ultimately used.

Hijacking the local X screen

In the end, I just ran the remote opengl programs on the server's local X screen. The machine was running Ubuntu server edition and it wasn't running an xserver be default, so I had to set up an xserver to run at startup (I just installed Ubuntu's ubuntu-desktop package, killing a mosquito with a sledgehammer), and then gave myself access to the X screen using these commands as root: "export DISPLAY=:0.0; xhost + local:". Then I could ssh into the machine, call "export DISPLAY=:0.0" and then run my opengl programs as normal. Anyone sitting at the remote machine would see a window pop-up and watch my program running, but we don't have a monitor connected so this wasn't a problem.

It's important to use some form of offscreen rendering, because reading pixels directly from the onscreen color buffer could result in garbage data if the window becomes obscured by another window. Since you can't see the X screen, it's difficult to know if this has happened. Offscreen rendering (e.g. Framebuffer objects (fbo) or pbuffers) doesn't have this problem.

Hijacking the server's local Xscreen isn't an ideal solution, so here are a few alternatives I found along the way:

Virtual Framebuffers

Xvfb is an option, but it didn't work for me, because OpenGL wasn't benefitting from hardware acceleration, and framebuffer objects weren't supported, which are necessary for CUDA interoperability with OpenGL. Nevertheless, this might be a workable option where hijacking the local screen isn't acceptable, or where user's can't get xhost privileges.

VirtualGL

From the VirtualGL website:

VirtualGL is an open source package which gives any Unix or Linux remote display software the ability to run OpenGL applications with full 3D hardware acceleration.

This is exactly what I want, and it looks very promising, but I didn't have the time to deal with a new library dependency, so I haven't tested it. My guess is that this is the ideal solution once I can get it compiled, installed, and configured. This is what VirtualBox and some VNC servers use to support hardware accelerated 3D.