slurm – How to Set Up Interactive-Job-Only or Batch-Job-Only Partition on a SLURM Cluster?

It has been a while since I touched SLURM so take the following with a grain of salt. Also, the design of your cluster will be determined by your workload. Generally, you start with a head node and a number of compute nodes and you build up from there. A package like Rocks can be a good place to start.

I can see the shared directory being a problem. Depending on your workload you may already have a lot of traffic going over NFS so I would install SLURM locally. You can make a copy of your slurm config available on an NFS exported volume and copy in place with a 'fornodes' script or use a scripted scp. If you are doing a lot of changes to your slurm config you could even add the slurmd restart to the script.

In regards to the backup control node, I wouldn't worry about it. Your head node is probably a single point of failure so if you lose that you are already going to have problems with your jobs. I'm also not sure how the backup mechanism works for SLURM accounting if it is enabled since that normally uses a database like MySQL.

For exports, I normally export the /home directory and /opt on smaller clusters. Depending on your data needs, you could consider a separate storage node with additional storage which would distribute your NFS load. Since you mention that you are having issues with stability you may consider using a package like Ganglia which will monitor node load, memory utilization, network throughput, and other values and present them in a series of graphs. You can also learn quite a bit with command line tools like top running on your compute nodes. You will also want to test scaling of your jobs. If your jobs run horribly when they span nodes(MPI?) you may need a faster, lower latency interconnect like Infiniband or 10Gb Ethernet.

Good luck with SLURM. I liked using it before I changed jobs but since it isn't as popular as Torque/Maui or Sun/Oracle Grid Engine answers to my odd questions were always hard to find.

I think this is mostly solved - still need to do some more testing, but things are mostly stable at this point.

The first issue was a hardware issue. The segfaults described above indicated RAM issues, so I started a suite of tests using memtest86+. Tests revealed lots of failures when all sticks were inserted, but individually each stick was fine. After reseating the CPU, memtest passed with all sticks seated. So, I think the initial problem was a poorly seated CPU.

The second issue was related to these types of errors:

[2023-12-07T16:51:03.492] debug:  job_container/tmpfs: _restore_ns: _restore_ns: Job 54 not found, deleting the namespace
[2023-12-07T16:51:03.492] error: _delete_ns: umount2 /var/nvme/storage/cn4/54/.ns failed: Invalid argument
[2023-12-07T16:51:03.492] debug:  job_container/tmpfs: _restore_ns: _restore_ns: Job 59 not found, deleting the namespace
[2023-12-07T16:51:03.492] error: _delete_ns: umount2 /var/nvme/storage/cn4/59/.ns failed: Invalid argument
[2023-12-07T16:51:03.492] error: Encountered an error while restoring job containers.
[2023-12-07T16:51:03.492] error: Unable to restore job_container state.

Note that these errors are appearing during a job with ID=64. The directories that were left over for jobs 54 and 59 in the job container directory /var/nvme/storage/cn4/ were zombies from failed jobs that failed to clear automatically. I think this was confusing Slurm who was trying to restore their state, but they were long dead, which caused confusion... or that's the best I can surmise with my naive understanding.

Clearing the /var/nvme/storage/cn*/ directories on all nodes ended these types of errors, and this node in particular doesn't drop anymore (at least as frequently? Need more testing.). The overall question is why these zombie directories hang around, which is a nuisance. But, at least it is easily addressed.