The answer is found in the Linux source, specifically, /usr/src/linux/mm/shmem.c, starting around line 70 on my system (Gentoo 2.6.31-ish):
/*
* The maximum size of a shmem/tmpfs file is limited by the maximum size of
* its triple-indirect swap vector - see illustration at shmem_swp_entry().
*
* With 4kB page size, maximum file size is just over 2TB on a 32-bit kernel,
* but one eighth of that on a 64-bit kernel. With 8kB page size, maximum
* file size is just over 4TB on a 64-bit kernel, but 16TB on a 32-bit kernel,
* MAX_LFS_FILESIZE being then more restrictive than swap vector layout.
One-eighth of 2 TB is exactly 256 GB. Larger sizes are possible with a 32-bit kernel, as you discovered with your 32-bit FC6 test system.
It appears that changing the page size may be related to enabling HugeTLB filesystem support in the kernel. However, I don't know enough about the guts of the kernel to say how or why, or what steps you need to take to take advantage of it, or what other implications it might have. To enable it, run make menuconfig, navigate to File systems, and then Pseudo filesystems. The option in question is HugeTLB file system support. The online help for it says:
CONFIG_HUGETLBFS:
hugetlbfs is a filesystem backing for HugeTLB pages, based on
ramfs. For architectures that support it, say Y here and read
<file:Documentation/vm/hugetlbpage.txt> for details.
If unsure, say N.
It might be worth running this by StackOverflow, too. I hope this helps.
Best Answer
You need any file system that supports "punch" to create holes in files where data previously existed. All files systems that support inodes (ext2/3/4, xfs, btfs, etc) support files with holes while creating the files but only recently has there been a way to remove (zero out) data in a way that creates holes. See http://lwn.net/Articles/415889/
If you are concerned with "TRIM" support for virtual disks, that's a related but different matter. That is related to the virtual disk system you use.