How can I tell the maximum threads my server can run?

Question

Here is the machine spec:

CPU(s):                20
Thread(s) per core:    1
Core(s) per socket:    10
Socket(s):             2

Based on what I've read so far, these numbers mean that I can run 20 parallel jobs because I have 20 CPUs.

However, how many threads can I run within each of those CPUs?

Solution

Welcome to ServerFault!

You have two CPU sockets, that is two physical processors. Each processor has 10 cores, each core being basically equivalent to a classic single-core CPU on its own. Each core can only run 1 thread at a time, i.e. hyperthreading is disabled.

So, you can have a total maximum of 20 threads executing in parallel, one thread per CPU/core. That can mean 20 single-threaded jobs, 1 multi-threaded job with 20 threads, or anything in between.

But that is only for threads that are expected to be 100% busy at work all the time, almost every millisecond of their life. If your threads are going to spend any significant time waiting for something else to happen, the system will automatically be able to run other threads while your thread(s) are waiting.

Unless you are in an environment that is heavily optimized for number crunching, you aren't going to get all those 20 threads for yourself for 100% of the time: the background processes (daemons) of the OS are going to occupy some of them for some (small) proportion of time. Run ps -ef (or equivalent) on any modern unix-like system when it's idle: you are going to see way more than 20 processes, each containing at least one thread, and that is just for the OS itself.

Assuming that you'll get the most out of the system by allocating exactly the number of threads the hardware has available may be oversimplifying the optimization problem. Depending on exactly what you're doing, you might want to use more or less threads. For example, if you are planning for heavy calculation jobs that are inherently CPU-bound, you might actually get better results by allocating slightly less than 20 threads for your job, so that one or two CPU cores will remain available for the OS's background tasks, so your job threads will get interrupted less often.

But if you are setting up a J2EE server environment, the JVM will usually have a large number of threads, and most of which will spend most of their life waiting for input, so the total number of threads used by the J2EE server's JVM on a 20-CPU system can easily be way higher than 20.

Comments

[tera_789]

That is a very helpful explanation; thanks. My threads are going to be actually I/O bound. I have 2 tasks: 1) I will be getting the size of a file on UNIX system and 2) writing something to disk. I believe both are I/O bound operations. For both, my plan was to run multiple processes (i.e., on separate CPUs) and then run multiple threads within each process (CPU). Am I clear? If so, do you think this approach is reasonable?

[telcoM]

Your tasks are clearly I/O bound. But you are mistaken in thinking that threads happen within a single CPU. A classic single-threaded process can only use one CPU at a time; a multithreaded process is more complex, but can execute on several CPUs in parallel, up to the number of CPUs it has. If a process is multithreaded, its threads will "naturally" share the memory space of the process, so passing data from one thread to another within a process will be easy. Processes are more strictly separated from each other: they'll need the OS's Inter-Process Communication services to share memory.

[tera_789]

Hmmm...Let me try to repeat what you said: There is no need to run multiple processes and then, multiple threads within them. It is enough to start a multi-threaded program on a single CPU, and it will naturally use the resources of other CPUs. Did I get it correctly?

[telcoM]

Yes, you understood correctly. For example, the top command will show a process's %CPU value as 100% when a single-threaded process is fully utilizing a single core. A multi-threaded process can show percentages higher than 100%, as its threads are using multiple cores. A value of 400% would indicate that the process has at least four threads and on average, the full capacity of 4 CPU cores was needed to run it on the last statistics refresh interval (typically 1 second by default).

[tera_789]

Ok, thanks. I think I got it now. I am not sure if you are familiar with Python and its GIL, but as I understand, only 1 thread can run with Python program at a time - as a result, CPU usage may never go higher than 100% there. Anyway, so going back to my question: on my machine (based on the spec provided), I can run 20 threads max if each thread runs at 100%. If not, then I can potentially have even 100 threads where 99 are waiting and 1 is running - is this correct? Also, if my machine supported hyper threading, this means at least 2 threads can run on a single CPU at the same time, right?

[telcoM]

Python's GIL is a limitation that is specific to the common implementation of Python only. If using normal Python, you might want to have multiple processes after all, as you won't be able to use threads effectively. If you can split the tasks into 20 sub-tasks that can be done independently, your program could then fork into a total of 20 processes to fully utilize your machine, and then put the partial results from each process together. Then those 20 processes could run in parallel. More complicated than with just one process, but with a 20-to-1 difference, might well be worth it.

[telcoM]

And yes, you're correct on hyperthreading, but that won't help Python with its GIL. Your system currently can and will run 20 threads in parallel, and all those threads don't need to be from the same process: your system can execute up to 20 different single-threaded processes in parallel. If you have more than that (as usual), they will have to take turns, and processes/threads waiting for I/O or some other event will automatically give up their turn for ones that can do some work immediately.