Linux Kernel logical address space organisation

Question

According to "Write Great Code" in almost all OS run time memory is organized into following regions:

OS | Stack | Heap | Text | Static | Storage/BSS

[In increasing address fashion]

User space process uses higher memory region for its different types of data objects.

Kernel space process also have different types of data objects. Do these objects share the user space memory regions (stack, heap etc) or do they have there own separate sub-sections(heap, stack etc) located in OS region.And, if so, what is the order in which they are arranged. Thanks,

Solution 1

It is wrong about the ordering. The OS is located at the top of memory, which is generally above the 3 GB mark ( 0xC0000000 ) in the 32bit kernel, and in the 64bit kernel it is the half way point of 0x8000000000000000 IIRC.

The location of the stack and heap are randomized. There is no real rule about the ordering of the text/data/bss segments within the main program, and every dynamic library has its own set of these, so there are many of them scattered all over memory.

Back when dinosaurs ruled the earth ( 20+ years ago ), the program address space was linear ( no holes ) and the order was text, data, bss, stack, heap. There were also no dynamic libraries or threading back then. That all changed with virtual memory.

Kernel processes are entirely contained within the kernel part of the address space; the user portion is ignored. This allows the kernel to speed up context switching between kernel threads beacuse it does not have to update the page tables since all processes share the same kernel portion of the page tables.

Solution 2

This is not true of “almost all OS”. The kinds of memory areas represented are fairly typical, but there's no reason why they should be in any particular order, and there can be more than one piece of a given kind.

Under Linux, you can look at a process's address space with cat /proc/$pid/maps where $pid is the process ID, e.g. cat /proc/$$/maps to look at the shell you're running cat from, or cat /proc/self/maps to look at the cat process's own mappings. The command pmap produces slightly nicer output.

08048000-08054000 r-xp 00000000 08:01 828061     /bin/cat
08054000-08055000 r--p 0000b000 08:01 828061     /bin/cat
08055000-08056000 rw-p 0000c000 08:01 828061     /bin/cat
08c7f000-08ca0000 rw-p 00000000 00:00 0          [heap]
b755a000-b7599000 r--p 00000000 08:01 273200     /usr/lib/locale/en_US.utf8/LC_CTYPE
b7599000-b759a000 rw-p 00000000 00:00 0 
b759a000-b76ed000 r-xp 00000000 08:01 269273     /lib/tls/i686/cmov/libc-2.11.1.so
b76ed000-b76ee000 ---p 00153000 08:01 269273     /lib/tls/i686/cmov/libc-2.11.1.so
b76ee000-b76f0000 r--p 00153000 08:01 269273     /lib/tls/i686/cmov/libc-2.11.1.so
b76f0000-b76f1000 rw-p 00155000 08:01 269273     /lib/tls/i686/cmov/libc-2.11.1.so
b76f1000-b76f4000 rw-p 00000000 00:00 0 
b770b000-b7712000 r--s 00000000 08:01 271618     /usr/lib/gconv/gconv-modules.cache
b7712000-b7714000 rw-p 00000000 00:00 0 
b7714000-b7715000 r-xp 00000000 00:00 0          [vdso]
b7715000-b7730000 r-xp 00000000 08:01 263049     /lib/ld-2.11.1.so
b7730000-b7731000 r--p 0001a000 08:01 263049     /lib/ld-2.11.1.so
b7731000-b7732000 rw-p 0001b000 08:01 263049     /lib/ld-2.11.1.so
bfbec000-bfc01000 rw-p 00000000 00:00 0          [stack]

You can see the code and the read-write data (text and BSS) from the executable, then the heap, then a memory-mapped file, then a little more read-write data, then code, read-only data and read-write data from a shared library (text and BSS again), more read-write data, another shared library (more precisely, the dynamic linker), and finally the sole thread's stack.

Kernel code uses its own address ranges. On many platforms, Linux uses the upper part of the address space for the kernel, often the upper 1GB. Ideally, this space would be enough to map kernel code, kernel data, and the system memory (RAM) and every memory-mapped device. On typical 32-bit PCs of today, this isn't possible, which requires contortions that are only of interest to kernel hackers.

While kernel code is handling a system call, ideally (when the aforementioned contortions are not in place) the process's memory is mapped at the same addresses. This allows processes to pass data to the kernel, and the kernel can read from the pointer directly. It's not a big gain, though, since the pointers need to be validated anyway (so that the process can't trick the kernel into reading from memory that the process isn't supposed to have access to).

The memory zones inside the Linux kernel space are fairly complex. There are several different memory pools, and the main distinctions aren't about where the memory comes from but rather who it's shared with. If you're curious about them, start with LDD3.

Solution 3

Not an answer, but an FYI that needs more space.

I don't think your conception of logical address layout is at all correct.

You can compile and run this program to see what a userland process has for addresses:

#include <stdio.h>
long global_initialized = 119234;
long global_uninitialized;
extern int _end, _edata, _etext;
int
main(int ac, char **av)
{
        long local;

        printf("main at 0x%lx\n", main);
        printf("ac at   0x%lx\n", &ac);
        printf("av at   0x%lx\n", &av);
        printf("av has  0x%lx\n", av);
        printf("initialized global at 0x%lx\n", &global_initialized);
        printf("global at             0x%lx\n", &global_uninitialized);
        printf("local at              0x%lx\n", &local);
        printf("_end at               0x%lx\n", &_end);
        printf("_edata at             0x%lx\n", &_edata);
        printf("_etext at             0x%lx\n", &_etext);
        return 0;
}

The Red Hat Enterprise Server I have running, has readelf, which can be used to say where the kernel would (logically) load an executable file:

readelf -S where

Shows me a lot of the same addressing information that the output of where gives.

I don't think readelf will easily work on a Linux kernel (/boot/vmlinuz or some such), and I think that the kernel by default starts at 0x80000000 in its own address space: it's not mapped in a userland process, despite using an address above the userland stack top at 0x7fffffff (x86, 32 bit addressing).

Comments

[sdaau]

Thanks for the example! Just my note on the Linux part - I just tried this example as where.c, on Ubuntu 11.04 using gcc where.c -o where; reports "main at 0x80483c4". Tried readelf -S where, and it reports, say "[13] .text PROGBITS 08048310 ... " which looks about right? Although I also get "ac at 0xbfb035a0" and "local at 0xbfb0358c", and that address range (0xbf...) seems not to reported by readelf -S.