Building a 32-bit app in 64-bit Ubuntu

After hours of googling, I decide to give up and ask you experts. I am trying to build a 32-bit application (xgap if anyone interested) in my 64 Ubuntu 11.10. I added the CFLAGS=-m32 and the LDFLAGS=-L/usr/lib32 in the makefile. The objects are built into 32 bit fine. The last step is to link all the objects and libraries for X windows into this executable—xgap. Somehow it keeps giving me this error:

gcc -o xgap xcmds.o utils.o gapgraph.o gaptext.o pty.o popdial.o xgap.o selfile.o   -L/usr/lib32 -lXaw -lXmu -lXt -lXext -lX11  -lSM -lICE

/usr/bin/ld: skipping incompatible /usr/lib32/libXmu.so when searching for -lXmu
...

/usr/bin/ld: i386 architecture of input file `xcmds.o' is incompatible with i386:x86-64 output
...

I have installed ia32-libs and mutilib support. I think I just need to force the linker to generate a i386 output. I tried to put two ld flags in my gcc command as shown above: -melf_i386 and -oformat elf32-i386. But what happens is that gcc doesn't search for the 32 bit library in /usr/lib32 anymore. I wonder if I need to put those flags in some fixed order?

Thanks for any idea and help!

EDIT: when I add the -m32 flag in my last gcc command (the linking stage I believe), even if I have the -L/usr/lib32 flag in place, gcc doesn't search in /usr/lib32 anymore ( really weird…) and generates the following error:

/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../libXaw.so when searching for -lXaw
/usr/bin/ld: skipping incompatible /usr/lib/libXaw.so when searching for -lXaw
/usr/bin/ld: cannot find -lXaw
collect2: ld returned 1 exit status

Any one has any idea why this happens? I am using the auto tool to configure and make. I am really good at modifying those script files.