Does mmap or malloc allocate RAM?

Question

I know this is probably a stupid question but i've been looking for awhile and can't find a definitive answer. If I use mmap or malloc (in C, on a linux machine) does either one allocate space in RAM? For example, if I have 2GB of RAM and wanted to use all available RAM could I just use a malloc/memset combo, mmap, or is there another option I don't know of?

I want to write a series of simple programs that can run simultaneously and keep all RAM used in the process to force swap to be used, and pages swapped in/out frequently. I tried this already with the program below, but it's not exactly what I want. It does allocate memory (RAM?), and force swap to be used (if enough instances are running), but when I call sleep doesn't that just lock the memory from being used (so nothing is actually being swapped in or out from other processes?), or am I misunderstanding something.

For example, if I ran this 3 times would I be using 2GB (all) of RAM from the first two instances, and the third instance would then swap one of the previous two instances out (of RAM) and the current instance into RAM? Or would instance #3 just run using disk or virtual memory?

This brings up another point, would I need to allocate enough memory to use all available virtual memory as well for the swap partition to be used?

Lastly, would mmap (or any other C function. Hell, even another language if applicable) be better for doing this?

#include <stdio.h>
#include <stdlib.h>
#include <string.h> 

#define MB(size) ( (size) * 1024 * 1024)
#define GB(size) ( (size) * 1024 * 1024 * 1024)


int main(){
    char *p;
    p = (char *)malloc(MB(512));
    memset(p, 'T', MB(512));
    printf(".5 GB allocated...\n");

    char *q;
    q = (char *)malloc(MB(512));
    memset(q, 'T', MB(512));
    printf("1 GB allocated...\n");
    printf("Sleeping...\n");

    sleep(300);
}

** Edit: I am using CentOS 6.4 (with 3.6.0 kernel) for my OS, if that helps any.

Solution 1

This is very OS/machine dependent.

In most OSes neither allocates RAM. They both allocate VM space. They make a certain range of your processes virtual memory valid for use. RAM is normally allocated later by the OS on first write. Until then those allocations do not use RAM (aside from the page table that lists them as valid VM space).

If you want to allocate physical RAM then you have to make each page (sysconf(_SC_PAGESIZE) gives you the system pagesize) dirty.

In Linux you can see your VM mappings with all details in /proc/self/smaps. Rss is your resident set of that mapping (how much is resident in RAM), everything else that is dirty will have been swapped out. All non-dirty memory will be available for use, but won't exist until then.

You can make all pages dirty with something like

size_t mem_length;
char (*my_memory)[sysconf(_SC_PAGESIZE)] = mmap(
      NULL
    , mem_length
    , PROT_READ | PROT_WRITE
    , MAP_PRIVATE | MAP_ANONYMOUS
    , -1
    , 0
    );

int i;
for (i = 0; i * sizeof(*my_memory) < mem_length; i++) {
    my_memory[i][0] = 1;
}

On some Implementations this can also be achieved by passing the MAP_POPULATE flag to mmap, but (depending on your system) it may just fail mmap with ENOMEM if you try to map more then you have RAM available.

Solution 2

Theory and practice differ greatly here. In theory, neither mmap nor malloc allocate actual RAM, but in practice they do.

mmap will allocate RAM to store a virtual memory area data structure (VMA). If mmap is used with an actual file to be mapped, it will (unless explicitly told differently) further allocate several pages of RAM to prefetch the mapped file's contents.
Other than that, it only reserves address space, and RAM will be allocated as it is accessed for the first time.

malloc, similarly, only logically reserves amounts of address space within the virtual address space of your process by telling the operating system either via sbrk or mmap that it wants to manage some (usually much larger than you request) area of address space. It then subdivides this huge area via some more or less complicated algorithm and finally reserves a portion of this address space (properly aligned and rounded) for your use and returns a pointer to it.
But: malloc also needs to store some additional information somewhere, or it would be impossible for free to do its job at a later time. At the very least free needs to know the size of an allocated block in addition to the start address. Usually, malloc therefore secretly allocates a few extra bytes which are immediately preceding the address that you get -- you don't know about that, it doesn't tell you.

Now the crux of the matter is that while in theory malloc does not touch the memory that it manages and does not allocate physical RAM, in practice it does. And this does indeed cause page faults and memory pages to be created (i.e. RAM being used).
You can verify this under Linux by keeping to call malloc and watch the OOP killer blast your process out of existence because the system runs out of physical RAM when in fact there should be plenty left.

Comments

[cHam]

what is the format of the headers in smaps? For example, the first line is "00400000-0040b000 r-xp 00000000 fd:00 1048608" I assume this first two are start/end addresses in physical memory, if so, what are the others?

[Sergey L.]

@cHam All addresses are VM addresses. You will not find physical addresses anywhere as those can change over time. r-xp are the permissions (read/execute/private in this case). The zeroes are the offset of an underlying mapped file (if applicable), device descriptor of an underlying file/device mapping. The last one is the inode of a mapped device, 0 if not applicable. Consult linux.die.net/man/5/proc for more details.

[cHam]

Do you have any suggestions on how to make each page dirty? I'm not sure what i'm supposed to do with (sysconf(_SC_PAGESIZE))...

[Sergey L.]

@cHam added an edit. You essentially have to write to each page. sysconf(_SC_PAGESIZE) is your system's page size. OSes allocate memory only in chunks of pages. mmap always allocates memory aligned to the page size.

[TheAhmad]

As a side note, malloc() RAM consumption is more severe when allocations are more fine-grained.