program stack size

Question

I learned default stack size for each process is limited to 8MB and mmap_base is calculated based on stack size in rlimit and random value. Code below is mmap_base function which calculates mmap_base address in x86(linux/include/uapi/asm-generic/resource.h).

static unsigned long mmap_base(unsigned long rnd)
{
    unsigned long gap = rlimit(RLIMIT_STACK);

    if (gap < MIN_GAP)
        gap = MIN_GAP;
    else if (gap > MAX_GAP)
        gap = MAX_GAP;

    return PAGE_ALIGN(TASK_SIZE - gap - rnd);
}

I am wondering what if program stack size is greater than 8MB+rnd value? I mean what if stack size grows above mmap_base ? If I allocate stack memory above 8MB is it just fail with segmentation fault? If kernel enlarge stack size automatically is it possible to move contents in mmap_base to other spaces?

Solution

The process main thread stack size cannot grow larger than the set limit. The default value of this limit is 8 MB. Exceeding this limit will result in a segmentation fault and the process will be sent a SIGSEGV signal, by default killing it. The maximum size of the stack can be changed with ulimit -s before starting the program. The kernel does not move around memory areas (like the mmap area) after the program has been started, and could not do so, because there are usually pointers pointing into this area that would point to wrong addresses after the move.

However, the check for stack overflow is performed when the stack memory is accessed, so just performing a large allocation on the stack, or otherwise changing the value of the stack pointer, does not necessarily trigger a fault.

There was some talk in the summer of 2017 about the possibility to exploit this behaviour. If some attacker can trick a program to allocate a large amount of memory, this can result in the stack pointer skipping a guard area and point into a valid, but different area instead. This opens up opportunities for some clever tricks to take control of the process. See this lwn.net article for a discussion of the issue.