HPC cluster: select the number of CPUs and threads in SLURM sbatch

Question

The terminology used in the sbatch man page might be a bit confusing. Thus, I want to be sure I am getting the options set right. Suppose I have a task to run on a single node with N threads. Am I correct to assume that I would use --nodes=1 and --ntasks=N?

I am used to thinking about using, for example, pthreads to create N threads within a single process. Is the result of that what they refer to as "cores" or "cpus per task"? CPUs and threads are not the same things in my mind.

Solution

Depending on the parallelism you are using: distributed or shared memory

--ntasks=# : Number of "tasks" (use with distributed parallelism).

--ntasks-per-node=# : Number of "tasks" per node (use with distributed parallelism).

--cpus-per-task=# : Number of CPUs allocated to each task (use with shared memory parallelism).

---

From this question: if every node has 24 cores, is there any difference between these commands?

sbatch --ntasks 24 [...]
sbatch --ntasks 1 --cpus-per-task 24 [...]

Answer: (by Matthew Mjelde)

Yes there is a difference between those two submissions. You are correct that usually ntasks is for mpi and cpus-per-task is for multithreading, but let’s look at your commands:

For your first example, the sbatch --ntasks 24 […] will allocate a job with 24 tasks. These tasks in this case are only 1 CPUs, but may be split across multiple nodes. So you get a total of 24 CPUs across multiple nodes.

For your second example, the sbatch --ntasks 1 --cpus-per-task 24 [...] will allocate a job with 1 task and 24 CPUs for that task. Thus you will get a total of 24 CPUs on a single node.

In other words, a task cannot be split across multiple nodes. Therefore, using --cpus-per-task will ensure it gets allocated to the same node, while using --ntasks can and may allocate it to multiple nodes.

---

Another good Q&A from CÉCI's support website: Suppose you need 16 cores. Here are some use cases:

• you use mpi and do not care about where those cores are distributed: --ntasks=16
• you want to launch 16 independent processes (no communication): --ntasks=16
• you want those cores to spread across distinct nodes: --ntasks=16 and --ntasks-per-node=1 or --ntasks=16 and --nodes=16
• you want those cores to spread across distinct nodes and no interference from other jobs: --ntasks=16 --nodes=16 --exclusive
• you want 16 processes to spread across 8 nodes to have two processes per node: --ntasks=16 --ntasks-per-node=2
• you want 16 processes to stay on the same node: --ntasks=16 --ntasks-per-node=16
• you want one process that can use 16 cores for multithreading: --ntasks=1 --cpus-per-task=16
• you want 4 processes that can use 4 cores each for multithreading: --ntasks=4 --cpus-per-task=4

Comments

[Tanash]

Thank you for your reply, but in your example sbatch --ntasks 24 […], what if I choose '--nodes=1' ? is it in this case will be same as your second example sbatch --ntasks 1 --cpus-per-task 24 [...] which will get a total of 24 CPUs on a single node?

[Tung]

For your question: you will need --nodes=1 --ntasks=1 --cpus-per-task=N where N is the # of (OpenMP) parallel threads. See example here stackoverflow.com/a/50654725/786542

[Tanash]

Is sbatch --ntasks 1 indicates to the number of threads in the sbatch script?

[Tung]

No. The --cpus-per-task=N is

[Tanash]

Does your last comment (about OpenMP) apply when no OpenMP is in use, just pthreads?