How is the linux graphics stack organised?

Question

Can anybody explain(hopefully with a picture), how is the linux graphics stack organised? I hear all the time about X/GTK/GNOME/KDE etc., but I really don't have any idea what they actually do and how they interact with each other and other portions of the stack. How do Unity and Wayland fit in?

Solution 1

The X Window System uses a client-server architecture. The X server runs on the machine that has the display (monitors + input devices), while X clients can run on any other machine, and connect to the X server using the X protocol (not directly, but rather by using a library, like Xlib, or the more modern non-blocking event-driven XCB). The X protocol is designed to be extensible, and has many extensions (see xdpyinfo(1)).

The X server does only low level operations, like creating and destroying windows, doing drawing operations (nowadays most drawing is done on the client and sent as an image to the server), sending events to windows, ... You can see how little an X server does by running X :1 & (use any number not already used by another X server) or Xephyr :1 & (Xephyr runs an X server embedded on your current X server) and then running xterm -display :1 & and switching to the new X server (you may need to setup X authorization using xauth(1)).

As you can see, the X server does very little, it doesn't draw title bars, doesn't do window minimization/iconification, doesn't manage window placement... Of course, you can control window placement manually running a command like xterm -geometry -0-0, but you will usually have an special X client doing the above things. This client is called a window manager. There can only be one window manager active at a time. If you still have open the bare X server of the previous commands, you can try to run a window manager on it, like twm, metacity, kwin, compiz, larswm, pawm, ...

As we said, X only does low level operations, and doesn't provide higher level concepts as pushbuttons, menus, toolbars, ... These are provided by libraries called toolkits, e.g: Xaw, GTK, Qt, FLTK, ...

Desktop environments are collections of programs designed to provide a unified user experience. So desktop environments typically provides panels, application launchers, system trays, control panels, configuration infrastructure (where to save settings). Some well known desktop environments are KDE (built using the Qt toolkit), Gnome (using GTK), Enlightenment (using its own toolkit libraries), ...

Some modern desktop effects are best done using 3d hardware. So a new component appears, the composite manager. An X extension, the XComposite extension, sends window contents to the composite manager. The composite manager converts those contents to textures and uses 3d hardware via OpenGL to compose them in many ways (alpha blending, 3d projections, ...).

Not so long ago, the X server talked directly to hardware devices. A significant portion of this device handling has been moving to the OS kernel: DRI (permitting access to 3d hardware by X and direct rendering clients), evdev (unified interface for input device handling), KMS (moving graphics mode setting to the kernel), GEM/TTM (texture memory management).

So, with the complexity of device handling now mostly outside of X, it became easier to experiment with simplified window systems. Wayland is a window system based on the composite manager concept, i.e. the window system is the composite manager. Wayland makes use of the device handling that has moved out of X and renders using OpenGL.

As for Unity, it's a desktop environment designed to have a user interface suitable for netbooks.

Solution 2

The traditional stack is built upon 3 main components:

• X server which handles displaying
• Window manager that put windows into frames, handles minimizing windows etc. That's part of separation of mechanism from policy in Unix
• Clients that perform useful task as displaying stackexchange website. They may use X protocol directly (suicide), use xlib or xcb (slightly easier) or use toolkit such as GTK+ or QT.

The X architecture was made network hence allowing the clients to be on separate host then server.

So far so good. However that was the image from way back. Nowadays it is not the CPU that handles the graphics but the GPU. There have been various attempts to incorporate it in the model - and place when kernel come into place to greater extend.

Firstly there have been made some assumptions regarding the usage of graphic card. For example only the on-screen rendering was assumed. I cannot find this information on wikipedia right now but DRI 1 also assumed that only one application will use OpenGL at the same time (I cannot quote right away but I can dig the bug up where it was close as WONTFIX with note to wait to DRI 2).

A few temporary solution have been proposed for indirect rendering (needed for composite WM):

• XGL - early proposition supporting applications talking directly to card
• AIGLX - accepted proposition that uses network properties of OpenGL protocol
• Proprietary solution of NVidia

The works on newer architecture (DRI 2) was started. That included:

• In-kernel support for memory handling (GEM/TTM)
• Kernel modesetting (KMS) allowing to change resolution in-kernel hence avoiding delays when switching between X and console & few other features (like displaying message on panic even if X are running - which IIRC is planned to be implemented).

Somehow orthogonal to move to kernel the work on Gallium drivers have been started. Mesa library started as implementation of OpenGL on CPU and then it started using GPU acceleration. It always have been tighten to OpenGL. In OpenGL 3.0 the model have changed significantly and the rewrite of library was inevitable. However they are taking the opportunity to split the code into several layer extracting common code and providing low-level API allowing to implement various 3D API on top of it - allowing for example Wine to provide DirectX talking directly to Gallium instead of going through the OpenGL API layer (which may not have direct 1-1 calls).

---

Wayland is a projects which considers the above a bit complicated and with too "history". The design from 1984 (although highly modified and adapted) does not corresponds to beginning of 21 c. according to proponents.

It is suppose to offer greater flexibility and better performance although it is still missing some important features like full OpenGL support (and important for some - network support).

---

A bit more about desktop environments and window managers. The window manager is an application responsible for how window will behave - for example it is responsible for managing workspaces, drawing titlebar(the thing on top of screen with windo title and minimise/maximise/close buttons) etc.

Firstly only minimal WM was used but afterwards the user started wanting desktop environments - i.e. more featured version, which included menu start, desktop background etc. However most parts of the desktop environment is not a window manager although they are often integrated.

After some time the composite WM were introduced which uses OpenGL and indirect rendering to do their work. They provides not only graphical effects but also allow easier implementation of some accessibility features (like magnifier).

Solution 3

First of all, there is really no Linux graphics stack. Linux has no graphical display capabilities.

However, Linux applications can use graphical displays and there are a number of different systems for doing that. The most common ones are all built on top of X windows.

X is a network protocol because in the middle of an X protocol stack you can have a network as a standard component. Let's look at a specific use case. A physicist in Berlin wants to run an experiment at the CERN in Switzerland on one of the nuclear particle colliders. He logs in remotely and runs a data analysis program on one of CERN's supercomputer arrays and graphs the results on his screen.

In Berlin, the physicist has an X-terminal device running some X-server software that is providing a graphical display capability to remote applications. The X-server software has a framebuffer that talks to the specific device driver for the specific hardware. And the X-server software speaks the X protocol. So the layers might be graphical device->device driver->frame buffer->X server->X protocol.

Then, in Switzerland, the application connects to a display using the X protocol and sends graphic display commands like "draw rectangle" or "alpha blend". The application probably uses a high level graphical library and that library will likely, in turn, be based on a lower level library. For instance the application may be written in Python using the WxWidget toolkit which is built on top of GTK which uses a library called Cairo for core graphical drawing commands. There may also be OPENGL also on top of Cairo. The layers might be like this: WxWidgets->GTK->Cairo->X Toolkit->X protocol. Clearly it is the protocol in the middle that connects things, and since Linux also supports UNIX sockets, a completely internal transport for data, these two types of things can run on one machine if you want to.

X refers to the protocol and anything fundamental to the architecture such as the X-server that runs the graphical display, pointing device and keyboard.

GTK and QT are two general purpose GUI libraries that support windows, dialogs, buttons, etc.

GNOME and KDE are two desktop environments that manage the windows on the graphical desktop, provide useful applets and thingies like button bars. They also allow multiple applications to communicate through the X-server (X-terminal device) even if the apps are running on different remote computers. For instance copy and paste is a form of interapplication communication. GNOME is built on top of GTK. KDE is build on top of QT. And it is possible to run GNOME apps on a KDE desktop or KDE apps on a GNOME desktop because they all work with the same underlying X-protocol.

Solution 4

This is its organization, you'll learn more from this picture that from several pages of text:

Solution 5

Linux on desktop and some servers is still all X and frame buffer graphics. Under X window -It comes GTK+ and Qt, YES BOTH uses X system, again there are lot of desktop environments - Gnome, KDE uses X display and their shells, etc.

Btw, There was a recent video from linux conf (http://blip.tv/file/4693305/). Keith Packard from Intel spoken about X and GL* It was interesting.

View more solutions

Comments

[mattdm]

Video of Xorg developer Keith Packard on the future of Linux graphics at linux.conf.au in Jan. 2011: linuxconfau.blip.tv/file/4693305 This covers both the current model and the plans for the near and medium future.

[kijin]

arstechnica.com/open-source/guides/2011/03/… is also a recent article going over the stack, and praising Ubuntu's commitment to Wyland.

[mattdm]

— yes, although that article is full of missing pieces and even inaccuracies and isn't, in my judgment, terribly coherent.

[kijin]

@mattdm - Even if it is incoherent etc., it goes into the bigger picture of the topic, which is not touched upon directly in the answers below.

[mattdm]

This answer is long out of date. The kernel has been involved in graphics for a long time now.

[dmckee --- ex-moderator kitten]

To expand on mattdm's comment, even when graphics are being driven by drivers from outside the kernel tree, they still use kernel services to control access to the graphical resources. The kernel always sits at the bottom of the stack.

[Michael Dillon]

I would not agree that the kernel is at the bottom of the stack. Of course, device drivers do use kernel services in order to get privileged access to the hardware, but an X application is talking out the network so there are more layers beyond even the network card.

[kijin]

So a framework like QT allows one application to draw itself, and desktop environments like Gnome and KDE decide how to draw multiple windows on the same desktop?

[Louis Gerbarg]

Not quite. QT allows the application to draw itself (i.e. it allows application to specify how it behaves). WM like metacity (for Gnome) or kwin (for KDE) specifies how window behaves in the environment. Desktop environment is package which contains WM, panels and other applications like PIM providing overlall experience for user.

[Louis Gerbarg]

X being built upon network while important to some setups on many is implementation detail (especially for desktops) and there are such extentions as MIT-SHM. The kernel plays important role in current stack both providing DRM drivers, KMS as well as handling textures.

[Michael Dillon]

DRM and KMS are more about device drivers that now have to communicate over a dedicated internal network connection to a graphical rendering CPU on the graphics card. This may be part of the graphical stack, and it may not (i.e. Amazon EC2 Linux).

[mattdm]

@Michael Dillon: kernelnewbies.org/…

[missingfaktor]

Disagree about the last sentence, but a very information-rich answer. +1.

[muru]

Where is it from? There are a few circled numbers, what do they mean? And this seems specific to Wayland, whereas I'd think X alone or Mir would have different organisations.

[jmunsch]

@muru doing a reverse search i came up with this.... en.wikipedia.org/wiki/EGL_%28API%29 ... although its currently hosted on imgur, since it appears to be an upload? But I agree linking the source image and where its displayed is the right way to do it?

[jmunsch]

This image doesn't really explain anything beyond the xserver though? Looking at you X11-client its just a blob, but there is a lot going on in that blob. As explained by the other really awesome answers.

[Adam D. Ruppe]

" (nowadays most drawing is done on the client and sent as an image to the server)" That's not really true, quite a few of them will do opengl rendering through the xgl extension, even without a compositor.