Compiz and other compositors have been living in the x86 world already for an while but these compositors are also making their way to mobile devices. To understand a bit better how these compositing managers work, I decided to take an bit closer look at the de facto Linux composite window manager - Compiz. I know that there are several good articles about compositing in general (like composite_howto) and Compiz as it is, is not aimed for mobile devices but the idea with this was to understand in higher level how this compositing in general is working in Linux platform.
For this I set up an separate Xgl session and used the good old ltrace to see what happens under the hood. So, off we go...
When launching Compiz the first thing we do is query whether the needed extensions are in place. This by calling functions like XCompositeQueryExtension, XDamageQueryExtension, etc. This to ensure that the infra is there.
The actual magic starts to happen when we call the XCompositeRedirectSubwindows, which requests the XServer to redirect the entire window hierarchy to off-screen buffer. This including current and future windows. This redirection is done at XServer level and is completely transparent to the applications, and therefore does not require any changes to them.
In Compiz case the XCompositeRedirectSubwindows call requests for manual redirection (CompositeRedirectManual). This will mean that the server does not internally track damage for them but instead registers damage listeners for each window. Updating of the screen in this case is done by Compiz itself - allowing it to do the final presentation to the screen.
Next step is to notify the X Server that Compiz wishes to draw to an window of the XServer itself - window called the Composite Overlay. This is done by calling XCompositeGetOverlayWindow. This overlay window exists above all other windows and provides an surface for the compositing manager to draw to. The challenge with this overlay window comes when the user generates input events (key- / mouse presses, etc) and these would need to be passed trough this overlay window to the actual X Window for interaction. Compiz use XFixes shape to enable this to happen. This with an set of function calls - XFixesCreateRegion, XFixesSetWindowShapeRegion, XFixesSetWindowShapeRegion, XFixesDestroyRegion.
So now we would have the basics up and running and we are ready for the applications.
First thing with the applications executed through the compositing manager is to create an damage listener for the window. This is done by calling XDamageCreate. With this call we want to be notified through an event whenever the damage state changes.
For the applications we need to create the off-screen storage, as we have told the XServer that we would like to render to off-screen buffer. This is done by calling XCompositeNameWindowPixmap. With this we are creating an window sized pixmap and also getting as return value a handle to the pixmap as an reference to the off-screen storage. When rendering the screen to the display Compiz uses this handle to bind this pixmap as an texture with an GLX (not in EGL) extension texture_from_pixmap - glXBindTexImageEXT. Compiz uses these GLXPixmaps to do the final rendering to the screen.