Why disable swap on kubernetes

The very idea of pods is that of co-location, that is, being able to define compute-locality where it makes sense (or where it's needed). So, no, per definition, since all containers in a pod will be scheduled on the same node, a pod can not span nodes.

See also my critique of pods here. Having said that, we've introduced the pods concept in DC/OS now with v1.9 as well. There seems to be a certain demand, but my hunch (since I don't have data on it) is that it's mostly a sort of safety blanket for cases where one doesn't or can't go all in, cloud-native wise.

As you said, it's because the data will not be synced across multiple nodes. It's probably ok to do some weird syncing solution across multiple nodes if only one pod will access the data at a time, but this quickly becomes a headache with multiple pods.

Instead of syncing the directory with something like nfs, just use that as your volume instead of a HostPath volume. Kubernetes supports nfs as a volume already, and many many more solutions as well.