Playing with JavaScript - Three.js & Physijs

Web technologies are hot. Everyone is speaking about HTML5 even without a clear definition what HTML5 is. This especially after the latest announcement from WHATWG  and W3C and them driving to different directions.
So lately I have been also playing around with couple of JavaScript libraries, especially with the ones touching the areas of Graphics and Physics.

Getting started dead easy. This especially if selecting Three.js as your 3D library and Physijs as your physics lib. Three.js is an 3D library for dummies, abstracting things like camera, scene, materials, etc. and also includes features like shadow mapping through an simple definition. It also allows you to select which technology you want to use for rendering your scene. But most importantly it includes some really nice examples and its example folder is really a treasury.
Physijs is an plugin to this library to provide some basic physics - built on top of ammo.js which is a direct port of Bullet physics engine.


As starers, I was using these two libraries to build an simple application where you have cubes dropping to an heightmap and bouncing from there. Problem with this was the physics which didn't react correctly to my heightmap. Nicely enough Physijs it self just fixed this problem by introducing HeightfieldMesh which allowed physics to be interpreted correctly with  my simple heightmap (vs. PlaneMesh that I was originally using).

Anyway nothing special, just testing what you can do with these libraries. At this time attaching only an image of this demo appl as the performance with my example is _really_ bad (when calculating the physics for cubes hitting the heightmap, its only ~2fps).  Live demo, including sources from here.

Mesa - from GLSL to drivers

Mesa drivers just reached an important milestone regarding to its ability to support OpenGL 3.0. With the latest patch the version is officially pumped to GLSL 1.30 for Gen4+ drivers. What is interesting is the path towards this milestone and everything happening around Mesa shader compiler on generating machine code from your Shading Language.

What the Mesa shader compiler in high level does is that it takes your GLSL shader as an input, tokenizes the input string and parses this to an form of an Abstract Syntax Tree (AST). From the AST the compiler gets its first IR, an High level Intermediate Representation (HIR), optimized for compilers usage. From this HIR, GLSL IR through optimization and linking the code is generated to an form of Mesa IR, an IR made to be similar with GPU assembly. This is kind of an classic compiler design with your front and backend and also what Mesa has adopted from the beginning.


As Tungsten Graphics started to sketch out a new driver model for Mesa - Mesa Over Gallium, they also wanted to questions this Mesa Representation model. With Mesa Over Gallium the people from Tungsten wanted to split the DRI driver and isolate the OS dependent and independent parts. Along the new driver model Gallium also introduced its own syntax for shader called Tokenized Gallium Shader Instructions (TGSI). The driver for TGSI where coming interestingly from DirectX and one of the goals was to have it close as possible to what the proprietary Direct3D assembly looked like. The model for this was coming from AMD's Intermediate Language spec which essentially is Direct3D assembly. With this goal in mind it was hoped that if a given GPU would be well optimized for Windows, it would also perform well with TGSI.

With TGSI model the Mesa vertex/fragment shaders are translated into TGSI representation passed to Gallium Mesa state tracker from where the driver code will eventually convert them to GPU specific instructions. Still the TGSI model was far from perfect especially as the TGSI IR was achieved by first parsing applications shader programs and generating IR from that and covenanting eventually the result as TGSI.


Towars GLSL 1.30 and new model for IR's

The optimization with the two models continue and Intel for ex did a lot of optimization for the IR by rewriting the ARB program parser part. Still the model for layering multiple IRs and the next GLSL specification 1.30 required a bit of an bigger changes to the current Mesa drivers.

With Gallium/TGSI path there was an extra step on generating TGSI from Mesa IR and not doing this directly from GLSL IR. For this what started as an work to optimize and make Mesa GLSL 1.30 compliant was an effort of skipping completely the Mesa IR and to generate TGSI directly from GLSL IR. This work also introduced the first GLSL 1.30 support for Mesa/Gallium drivers as the translator added an native integer support for Gallium drivers.
On the Gallium side there was also an another effort on refactoring the IR model by placing Mesa on top of of LLVM and as an ultimate goal of having GLSL generating directly LLVM IR. The first ideas of this work was introduced already a while ago and is something that the people at LunarG have adopted wih their goal on having an HW independent IR and dependent parts of the IR separated. Layering the shader compiler called LunarGlass similarly to the Gallium architecture.

Later on the Classic Mesa driver side an new GLSL compiler was introduced and developed from scratch. The new GLSL version 2 (or at least take 2) aims to generate GLSL IR directly from assembly shaders and from fixed functions, and most importantly using GLSL IR directly with the drivers rather than stepping through Mesa IR - the original Mesa Intermediate Representation. This mainly as the Mesa IR was designed and written in the times when ARB - OpenGL Assembly Language was still the way to control the graphics pipeline and the and having 1-to-1 mapping all to way to HW instructions was seen as an benefit. This was not the case with modern GPU's and having higher level shader language existing.
Along the IR changes this work also brought the GLSL 1.30 support to Classic Mesa drivers for ex by not anymore assuming that the only HW type is vec4.


Were we are now is that that the pixel shader units themselves are still faster than what we need. Still the longer and more complicated shader programs will become more common even beyond games and also the usage of shaders is becoming more diverse. This meaning that the performance will definitely be welcomed also to our Linux drivers.

Mastering Memory

Linux Kernel has two different existing memory management solutions - one based on GEM and the other one for TTM. Both of them are utilized through DRM and designed to fulfil different DRM operations. Still besides these two existing solution especially the SoC world is full of different out of the Kernel tree memory allocators. During the past couple of months the Linaro people have been taken some initiative to fix this problem but how - well the idea is to introduce a new memory manager fitting the ARM needs.

The problem setting with DRM based and SoC solution have kind of a different approach. Where DRM based memory manager is designed to provide applications a synchronous access to the HW, the SoC problem setting in the discussions has been kind of a turned upside down. In SoC case the main point of the memory manager is about how the different HW IPs/device drivers interconnect and share data. Allowing applications to share data is kind of an second priority.

A good question in this case is that why not to stick with the existing Linux kernel memory manager and making the existing upstream project to work. Making DRM to support sharing of data between the device drivers and narrowing it down by making some of its more legacy parts optional. And even if not taking DRM as the base of the solution, why every SoC have ended up to developed their own solutions - UMP, PMEM, KGSL and now ION from Google. One problem and challenges with this setup is the API. The evolvement of the graphics chipset is so fast that freezing an API cannot be taken into account. So especially in the case of SoC vendors the controlling of the API is highle beneficial and definitely a desired way of doing things. Being able to control not only the drivers but also the API will allow SoC vendors to ensure that their solution will perform the best and won't break anything around it.

So it will be definitely interesting to see whether Linaro is able to pull out an common solution that everyone agrees to use. Also interesting to see wheter Google with its power is able to driver this towards a common soluton with their ION based solution. The questions is also does the SoC vendors eventually really want to have an common Memory Management solution.

Competition or cooperation: Gnome-Shell and Unity

The new Gnome-Shell has started to land and to take its final development steps. This landing has also raised some intensive discussions around Gnome - what is Gnome, whom are the competitors and what should get into Gnome.

From the major distros the Fedora camp have indicated themselves to utilize Gnome-Shell/Gnome3 with their upcoming Fedora 15 release, where as the Canonical/Ubuntu guys will be using their own homemade Unity Dekstop and Gnome3 applications. This division was initially fired already at October 2010 @Ubuntu Developer summit where Canonical stated that all the future releases would be based on Unity.

Currently ongoing discussions have been especially intensive around libappindicator - as it has been taken as _the_ item to discuss in various of different blogs [1]. Shortly the history of indicators have evolved from KNotificationItem to Canonicals StatusNotifier proposal and now to the rejection of libappindicator as part of Gnome 3.0 module set - "there is nothing in GNOME needing it". For Canonical this statement is quite clearly saying that Unity is not part of Gnome. Naturally this has raised some discussions.


This definitely becomes interesting when we are discussing about the future of Gnome. What we have here is freedesktop.org project aiming for "interoperability and shared technology for X Window System desktops" and two most popular Linux distros differentiating themselves. What we are also discussing about is free software. Surely in both efforts it is just about that but Canonical copyright assignments contribution agreements are something that Gnome does not want to depend on (this in a scheme where libappindicator would be part of GTK+).

The two projects do have quite different directions, making them competing against each other rather than cooperating. I personally do agree with the statement that healthy competition is always welcomed. The danger is that too much competition can also ruin the possibilities to compete the outside world - competing with the closed source solutions whom are running fast. Getting some new innovation emerging from these Linux based desktops is something that we all would want to see. This would require some big time cooperation instead of competition.


[1]
* collaboration's demise
* Application Indicators: A Case of Canonical Upstream Involvement and its Problems
* Has GNOME rejected Canonical help?

Way to Wayland - take 1

There is an interesting discussion ongoing regarding to Waylands current memory allocation model and how this could be scoped better to be more suitable for different graphics architectures.

The rendering model with Wayland is simple and therefor powerful. The key part is the possibility to share a video memory buffer between the client and compositor. This all is currently done through DRM which also makes DRM an essential part of the whole Wayland concept.

So what has now been under the discussion in the Wayland forum is how the current model could,
a) provide interoperability between client and server. Different Wayland clients could more easily discuss with different Wayland Server solutions
b) isolate the important memory handling part away from the core protocol. Allowing Wayland to be used more easily with other memory architectures.

The proposal is to split the DRM specific part to an separate file which would solve both of these problems and would once again put Wayland closer to take the place as the future Linux graphics solution.

Some interesting statistics for the beginning of new year

Year 2010 has ended so its nice to check the couple of interesting OSS Graphics components have evolved during 2010. With the interesting ones meaning Mesa, XServer and the one that everyone is discussing about, Wayland. Surely along these components there has been a lot of activity on the Kernel side from graphics point of view by moving the logic to the Kernel side (DRI drivers, the kernel side GEM scheduler and kernel mode setting) plus there has also been a lot of activity around the toolkits but as said now sticking with these three.
So what has happened,

With Mesa we have:
157 people committing
and those people pushing in 12816 commits (of different sizes)
(where as in 2009 the figures were, 118 people pushing in 9719 commits)

With XServer we have:
99 people committing
and those people pushing in 1305 commits (of different sizes)
(where as in 2009 the figures were, 109 people pushing in 2057 commits)

With Wayland the figures are especially interesting as it has received a _lot_ of publicity lately (Unity, Fedora). Now during the year 2010 there were already close to 20 contributors for the project where as a year ago there were only two.


So a lot of things happening in the head of mainline. If we check these commits on daily basis - Mesa having average of 35 commits happening every single day and XServer having 3.5 commits on daily basis. And not to even mention the Kernel which has over 100 commits per day.

Starting to deviate from this with an custom build behind the closed doors will mean an big maintenance and development effort for the instances deciding to do so. On the other hand this is sometimes mandatory. When discussing about an commercial company which is still aiming at to make profit you are bound to keep something behind the closed doors.
So balancing between Open Sourcing code and potentially exposing the bits and pieces from the latest and greatest can get tricky and really needs to be thought carefully.


PS. with MeeGo the problem is tackled by doing the XServer development with the git head - see.

SceneGraph in toolkits

SceneGraph architectures can be found from quite many of the new architectures introduced. Clutter with its latest rework on the SceneGraph to make it as retained as possible, Qt with its new SceneGraph architecture and also Mozilla's Layers architecture is getting its idea from SceneGraph. Earlier this has been the way the games render them selves but now this model is starting to define how toolkits render our 2D content.
First idea with the scene graph is that you will start to take the benefit from GPU rendering. Instead of having the applications doing the rendering in SceneGraph the compositing is done in retained mode, constructing an tree structure of the scene to be visualized. The SceneGraph paradigm and problem setting with the toolkit/widget and games is still quite different cmp'd to games. When we are discussing about 2D UI's we do not have complex geometry, fancy viewport transformations or dynamic lightnings. Instead we are mainly talking about getting our textures displayed on the screen and about blending them. This difference puts also requirements to the actual traverse algorithm and renderer to ensure that we can take everything out from the GPU from 2D toolkits point of view. Also the problem statement is rather aiming for optimizing of the performance than managing the complexity of the view.

With the GPU there is one particular thing that it cannot execute efficiently - changing states. What the GPU's are designed to do is to load in the shader program and do their stuff. Simple. With the imperative way of doing the rendering where the view is constructed by hierarchically rendering your scene by the application you are bound to do a lot of state changes. This as every single application is in charge of rendering itself - widget by widget and line-by-line, constantly loading content to the GPU. So what we want to do is to feed our GPU as much data as possible and letting it do its job. This by grouping our geometry, shaders and textures data together.
The most important data (or node) from toolkits point of view would be our appearance or geometry node defining our graphics primitives (vertices data and texture info. Data required for your widget rendering). This data is something that we want to store not only per-frame but per lifetime of an application to minimize the state changes.

Second important item is the drawing it self. Although GPU's are designed for drawing, the unnecessary drawing is naturally something that we don't want to do. An easy way to improve in this area is to draw less, or draw only what is visible on the screen. This is exactly what our SceneGraph architectures want to do. With its tree structure the SceneGraph model can smartly use the z buffer to avoid the overdraw. And by this not meaning the computation time clipping and depth buffering that the GPU's are able to do, but rather limiting the objects actually sent to the GPU. This is done by selecting the rendering order smartly (front-to-back when possible) or for ex in Mozilla's Layer case placing the transparent areas into their own layer. The challenge is of course that with the modern UI's we are discussing about a lot of transparency which will force us to do back-to-front rendering.

There are of course also challenges. The SceneGraphs are balancing between the memory consumption (caching too much) and performance. Other challenge is the different nature of different GPUs and drivers as they sometimes behave a little differently and do different things well.
With the toolkit SceneGraphs the challenge is also the target. Are they only aiming for optimized 2D rendering or would they also like to serve the games.

SW that matters

During the past few weeks/months we have been seeing a lot of news about the future HW of different vendors. Qualcomm has been pushing out news about their 1.5GHz dual-core Snapdragon (ARM cortex-A8 based, Qualcomm modified), Samsung about their Orion a 1GHz dual-core ARM Cortex A9 and Marvell its tri-core Armada 628 (once again ARM based, ARM v7 cores) which can have the main two cores running up to 1.5GHz and the third at 624 MHz. Quite impressive, right?

Lets first think about the figures that the different vendors are providing. What do we do with all of that power? at least form my point of a view this is an excellent questions.
Take for ex the first generation iPhone - ARM11 downclocked to run at 412Mhz, PowerVR MBX light (optimized for low power consumption instead of speed), 128MB of RAM. Still with these figures in the paper even the first iPhones can still beat many of the devices in the markets today. And bare in mind that iPhone 2G was announced already at at the beginning of 2007.
Why is that? Well the SW and especially the framework has been build in a respect of the underlaying HW. And because of the fact that Apple was ready to make compromises with the device - no multitasking which makes things a lot simpler (limited support existing nowadays), simple set of features which really work, extremely simple and limited set of widgets without theming or the 480x320 pixel resolution combined with a color depth of 18 bits (helping a lot with memory bandwidth usage [1]) - focusing on simplicity and usability.

And its not only about the framework from the SW side that matters - badly implemented drivers, SW which is not making the use of the architecture and features a chip offers (separate cores for video, image processing, blitting, etc.). This will effect to the overall performance.


The other thing with these nice MHz figures is the battery consumption. My self I would at least like to get my hands on one touch screen based device which would really last longer than my working day. Of course this is also how the power management of the device is done, which comes once again to well designed and implemented SW...

Instead of getting the MHz's up, how about as an ultimate goal to get the CPU to run as slow as possible and really shifting the operations to GPU. The GPU's can in many cases achieve far better efficiency than the CPU. This would of course still require something from the SW - framework build in such away that it can really utilize the GPU and not just making it to do some fancy effects when switching between applications.


[1] 320x480pixels, 18 bits for a pixel and rendering that 60 times per sec vs. that a frame would hold 800 x 480 pixels (as the Nexus has).

Standards, agreenments and fragmentation

The graphics area can be seen through your OpenSource "standards" and industry _standards_. Even when discussion only about Linux and not even going to the Microsoft land the field is starting to be a quite messy.

The fragmentation in open source side is in many areas caused by the fact that the communities are competing with each other. They want to ensure that they can bring out as good as possible SW as soon as possibly - naturally. What this causes is that instead of making an effort and developing something together we fragment. Good examples of this is your TTM vs. GEM, classic Mesa vs. Mesa Gallium state tracker or EXA/XAA/UXA.

Then we have our industry standards by Khronos. What Khronos defines is an open standard mainly for Graphics and Media. We have the defacto OpenGL/ES and the counterparts GLX and EGL and then we have something else. We have things like OpenWF (display and Composition) which in OSS world is know as KMS (with limited composition support), OpenKODE with the OSS counterpart being SDL and OpenMAX which in OSS world is tackled with VAAPI or XvMC or VDPAU. Well you get the point.

So why should we care about this? What this all means is that the set of selection when developing an Linux based system - following the industry standards or selection one of the Open Source paths. Often the proprietary drivers do follow the industry standards but they have not adopted any of the OSS ways of providing the functionality and why would they as the OSS "standards" are often vendor specific. This will mean that reasonable way to build our platform would be the industry standards but then you would loose the possibility to try out the new nice things which are not executable in these platforms. Difficult choice and ideally a choice that we wouldn't need to do.

UI done right

I have always been a fan of what the guys at TAT are doing. These guys have really understood the whole thing of making great looking UI's in away that it is respecting the underlaying HW and taking every last bit out of it.

Their latest creation Velvet is just that - innovative, simple, great looking UI. Nice job!


PS. Would be nice to understand how these guys are working. You have your UI designer, framework developer and lower level graphics guy all working seamlessly together...

New Gnome-Shell release ... reminds me of something

Gnome-Shell had an new release published at mid last month. The new release is now dependent from GTK3.0 and at least according to different sources should be more stable and performing - ready for a test drive.

So what is this Gnome-Shell.
Well Gnome-Shell is your integrated window manager, compositor, and application launcher. It uses Clutter canvas as an scene where your GtkWindows are reparented as actors. The main logic behind this is under Mutter, an fork from Metacity.

The architecture is done in such away that the window manager, compositor, and application launcher are tightly coupled together. With this approach there is no need to provide any additional abstraction layers to different WM/CMs or any scene/graph abstraction. Also with this the GNOME guys have the full control of the stack. Because of the architecture Gnome-Shell has been receiving quite a bit of comments especially from the Compiz people. This as the approach will make other WM/CMs live quite difficult.

With the architecture, it is actually funny how it reminds the Hildon 5 architecture - composition with clutter canvas as scene, Hildon desktop integrated appl launcher, desktop integrated Window Manager, etc. and everything nicely coupled together.


As from the user experience,
launching the latest and greatest Gnome-Shell with my Intel 855GM based based device was an extremely slow experience. Gnome-Shell was really unusable. Bit of an Googling revealed that syncing to VBlank might be the root cause for this problem as it is enabled by default. "export CLUTTER_VBLANK=none" seemed to be the cure for this problem. Wonder why this problem... (bit of a debugging might be in place). This trick makes it at least testable.
The main thing that I noted with the Gnome-Shell is that you really want to learn to use the basics - your ALT+F2 key combo. The new Application Area did not at least convince me. Maybe as the usage is going towards more basics it would be time to put the good old Awesome back in use - going really back to basics...

When accleration came to browser

HW accelerating in website rendering has become an hot topic along the HTML5. Mozillas answer to this demand is their new Layers architecture.
Mozilla's Layers allows browser to take the suitable parts of the rendering to GPU and of course along that to get some eye candy to the end users. Idea is to divide the logically different parts of the rendered content to different layers allowing better division between the CPU and GPU.

The Layers will structure a layer tree which constructs the final webpage. To simplify, they have a container (representing the surface) and its leafs (including the actual graphical data). The final render is constructed from the textures generated from the leafs and composited to the screen.


The basic layer infra will include ThebesLayers, ContainerLayers and a LayerManager.

The Manager is responsible of constructing the layer tree and rendering that to the destination context. The rendering is always happening as an result of an transaction to the tree and only happening to the visible region parts of the texture.

The Container layers task is to group together the child layers beneath it. It represents the surface into which its children are composited and handles the Z order of the its children.

Thebes in general is nothing new. It has been part of the Mozilla gfx architecture already quite awhile as an C++ wrapper for Cairo API's and as an API for text handling.
The ThebesLayer abstracts the Thebes surface which is used to display items that will be rendered (surprise, surpise) using Thebes. So Cairo Image surface drawn to by Cairo. The question is of course - why not doing this directly with Cairo? Why the wrapper? - Some discussions about this from Bas's blog comments.

There is bit of an older bug in Mozilla bugzilla explaining this basic functionality in detailed. In principle what you get with the basic functionality is what you already have - Cairo content shown to the screen. Of course besides the basic functionality you will also need some additional layers for video, something for webgl plus you would also like to have optimized way of doing the rendering. So not all about just basics.


What from my point of view makes this layers especially in interesting are the plans to bring layers to also to Fennec 2.0 along the other goodies like Electrolysis. It will be most definitely interesting to see how it performs and behaves in a more limited environment.

BTW the first bits of layers are already available through the latest nightly builds of Firefox4. The usage of the layers is still limited to Windows/GL, full-screen video only but the sources are available for seeing and testing.

Difficulty of developing Open Source Linux drivers for GPU

During the last few days there as been an active discussion ongoing at dri-devel list about Open Source Graphics drivers. The discussion was initiated by Qualcomm's RFC message - [RFC] Qualcomm 2D/3D graphics driver.

Qualcomm's Linux team has been working on Open Sourced Kernel drivers for their chipset. What they have done so far is that they have build DRM driver with a limited set of supported functionality to enable DRI2. Their own userland parts of the drivers do still utilize the KGSL (Kernel Graphics Support Layer) interface to allocate memory making DRM just as an API to be more closer with the mainstream Linux Graphics stack. Qualcomm does still have some plans in the future to move to an standard design - possibly TTM.


The dri-devel list the opinions about drivers with an userland binary blobs was made quite clear - if you aren't going to open the whole drivers don't try to push them to mainline. Basically the door was slammed from the Qualcomm efforts for pushing the drivers in.

In between the lines I think there were some excellent point made in this discussion.
The bottom line is that the community cannot work with the drivers if there exists and dependency to closed sourced userland - this is definitely clear. With this kind of an setup it is just impossible to work with the drivers. On the otherhand if the community is going to bombard down all the drivers with the closed userland parts (even without making sure what are the attemts to open source them) how are we ever going to see open sourced mobile GPU drivers? From my point of view the community also has the means to guide this work to right direction - giving comments to the request. It really seems that the Qualcomm guys do have the enthusiasm to develop the open source drivers. How about making sure that these guys are taking the correct steps also with the userland and making sure that the Kernel bits are as they should?

MeeGo for handsets getting its UI

MeeGo for handsets has taken its next step and is now capable to provide MeeGo User Experience also for mobile users - see the blog entry.
There is some nice screenshots and video showing its functionality. Unfortunately the downloadable image is only available for Moorestown based Aava device. The N900 image still needs to be compiled by hand - instructions from here and repo from here.

GLES1.1/2.0 on Mesa DRI and a bit about Gallium

Mesa has had GLES 1.1/2.0 already since since last year through Gallium state trackers. What this means is that Gallium based drivers are capable of providing support GLES support for end users. Now there is also an activity ongoing to get the Mesa GLES1/2 support to work with DRI drivers and along that to expose multiple GL APIs without separate state trackers on Gallium side.

This work on GLES 1.1/2.0 on DRI made by Kristian received good feedback also Gallium driver developer and is now merged to Mesa.


The discussion about classic Mesa vs. Mesa Gallium State Tracker in general has not lately been this productive (see the Mesa thread). Intel guys are clearly wanting to stay with the classic drivers, and whom could actually blame after all the work that Intel has place to get their drivers to perform. Currently Nouveau camp is the only one that is really utilising Gallium3D and the RedHat guys are going to ship the latest free software NVIDIA bits with the upcoming Fedora 13.

Multitouch support for X

This is not directly grapihcs related but about interesting enablers for modern UI's anyway...

XInput2/MPX was introduced already in XServer1.7 and the Linux Kernel has had the multitouch support since 2.6.30. So we should have case closed and a working multitouch solution existing for Lixux environment. Right?
Well, what we currently have are couple of reference implementation done to show of the Linux multitouch capabilities, but the full pipeline coming directly from upstream is still missing. When we are discussing about X we should be capable to support all the different use cases from simple gestures with pinch, all the way to cases where we have multiple user interacting simultaneously with a big multitouch device. This makes things complicated.

From the current reference implementations people at ENAC have been working quite actively around the multitouch solutions - directly on top of Linux + they have also patched the existing evdev to propagate multiple pointers through MPX. Another reference implementation of multitouch X driver is done by Henrik Rydberg - found from here. Still either of these solutions have not made it to official XServer.

Now there has been an discussion ongoing in the xorg-devel mailinglist that there would be now an clear first step to include support multitouch features for X. This first step would be limited to have a single focus / input point with auxiliary information attach to the event. Most importantly X would propagate these events to the right clients.
The solution would be mainly for indirect touch devices (like touchpads), but would also cover single-application input cases. From mobile devices point of view, where we would have a simple window manager with top most application fullscreen, this solution would already be able to cover all your favourite two finger pinch gestures - making this basically a full fledged solution for these kinds of devices.

DRI2 driver for Mesa EGL

Couple of weeks ago there was an announcement that Wayland would be moving to Mesa EGL and drop its dependencies to its own Wayland specific Eagle EGL stack. The Eagle is a non-conformant EGL loader for the DRI drivers originally develped by Kristian Høgsberg as an "side product" of his DRI2 work. Eagle as an lightweight and with its way of integrating EGL directly to KMS/DRM, was the solution for Wayland - this at least until recently.

Major reason to allow Wayland to jump to Mesa EGL is Kristian's work for the work behind the DRI2 driver for EGL and to get the Eagle EGL parts as officially to Mesa. This work - EGL on DRI2 is firstly aiming to implement this support through X, but the longer term goal seems to be that the implementation would be standalone - working directly with KMS/DRM and making it suitable for Wayland purposes.

Current plan is that the DRI2 parts for the Mesa EGL would be integrated to the upcoming Mesa 7.8 - planned to be release during this month. This is good news especially for Wayland. Along the Mesa integration the Wayland enablers are now even more mainlined. This move will most definitely make Wayland it self more attractive to outside world.

Challenges in 2D acceleration - how about the future?

To achieve great 2D graphics performance from Linux Graphics stack has most definitely been an challenge for years. Trying to get the XRender extension more usable with XAA/EXA/UXA (see my previous blog post), utilising all-software image backends, utilise GL directly to accelerate the 2D and so on. The best solution really has been the CPU rasterization backend based on the fact that the CPU simply runs with higher clock speed and quite many of these 2D operations require raw processing power.
So are we stuck just to continue the everlasting optimisation of the existing solutions? Maybe not.

Now it seems that this there might be some light in the tunnel. Cairo has been getting some quite impressive performance figures with its cairo-drm backend. Instead of using XServer to do the rendering or build an abstraction of XRender with GL backend (what glitz was doing) it has the capabilities to use directly the GEM buffers and issues the rendering commands directly to HW. This will allow direct access to GPU and will give quite impressive performance figures.
The thing is still that cairo-drm is an experimental backend and that the backend is limited to Intel HW (i915 to be specific).

With the performance figures - what the cairo-drm can really do, the goal for the guys is now to get this same performance out from the GL interface directly. This would 1st of all remove the limitations mentioned above and 2nd, it would also allow all the rendering to happen under one driver - OpenGL. This of course would mean that there would be only one graphics stack to tune and maintain and would allow CPU to do other stuff. The cairo-gl project aims to do exactly this.
The cairo-gl backend would also make the solution usable with other than Intel specific HW. The questions of course is that how well the solution performs in non-Intel HW.


Time will show how the cairo-gl backend will perform and whether it is capable to beat CPU rasterization backends.
More info about the progress from:

1. http://anholt.livejournal.com/
2. http://ickle.wordpress.com/

N900/Maemo5 and different texture formats

Garage.maemo.org is providing an nice tool for benchmarking the different texture formats that the meamo5, SGX based device supports. The tool GLMemPerf is meant for measuring the texture memory bandwidth performance of an OpenGL ES 2.0 implementation with a set of different kinds of precompressed textures. As an output it blit the different textures from its /data folder to the screen and provides an nice set of information - how the different formats and texture processing methods effect to the overall performance.


From the results the compressed textures formats (IMG specific PVRTC and ETC1) are clearly giving an performance benefit - as one could expect. The IMG PVR Texture Compression format has the 2-bits-per-pixel and 4-bits-per-pixel options create PVRTC data that encodes source pixels into 2 or 4 bits per pixel. This meaning that each texel is encoded into fewer bits than the equivalent uncompressed texture, which of course is meaning less load for memory bandwidth.
The downside with the compression is that the overall quality of the render will suffer. The quality hit is still depending from the compressed content so might be worth of checking how 2 vs 4 bits per pixel will effect.

The other thing to note is how the textures are stored and accessed from the memory.
SGX as many other GPUs benefit greatly from the usage of using power-of-two sized textures - which improves the implementation of the texture mapping when converting the texture coordinates to texel coordinates. It might be worth of rounding up the textures to the nearest power-of-two to get the benefit. Note also that the PVRTC formats require that the textures are in power-of-two size.
The other thing related accessing the memory is the texture twiddling. The twiddling is by default done for all the SGX uploaded textures. What the twiddling does is that it re-arranges the texture samples to optimize the memory access for different operations. One thing to note is that with pixmaps you will not get this benefit as in a case of pixmaps you don't upload the textures to the video memory but instead source the pixels directly.


How about generating and processing your textures? For this IMGTec is providing an nice tool called PVRTexTool. The tool has an version for command line usage and also an GUI based tool for seeing in more detailed the output of your changes - for ex when wanting to do comparison between the different compression methods. The PVRTexTool supports your normal RGB/ARGB formats + the texture compression format PVRTC and ETC.

Challenges in 2D acceleration

GPU based 2D acceleration has been an hot topic in Linux already for years. With hot topic meaning should or shouldn't it be utilizing GPU for rendering and if so - what should be accelerated. The first attempts with XAA didn't give basically any speed advantage and accelerated only some core rendering elements to the graphics HW. Along the X Render Extension providing some nice functionality for modern applications like alpha blending, drop shadows and translucency the questions of whether GPU should be used needed to be revisited. EXA enabled this opportunity to HW accelerate render extension requests and therefor improving the X.Org Server 2D performance.

The latest twist in this field is UXA which is based on the EXA code base and utilizing GEM Memory Manager. With this latest combination by moving the memory manager to the Kernel side (with GEM) it seems that the equation of accelerating X is finally getting in shape but there still seems to be challenges especially with text/glyph rendering - more from this in Carl Worth's web pages.


But how about these acceleration architectures in action? As I was anyway about to reinstall my Intel 82852/855GM Graphics Device accelerated laptop (old and integrated, I know...), I decided to put these architectures in test.
As an distro I selected the latest Ubuntu 9.04. The actual benchmarking was done with mx11perf - this mainly as it had nice benchmark script already existing which covered quite nice variety of X11 operations. Some of the most interesting results from this run can be found from below.

Rendering of basic rect:

	EXA	UXA	XAA	NoAccel
Rect 8x8 Src	10410/sec (0.67 Mpix/s)	7598/sec (0.49 Mpix/s)	5527/sec (0.35 Mpix/s)	263154/sec (16.84 Mpix/s)
Rect 32x32 Src	9316/sec (9.54 Mpix/s)	6701/sec (6.86 Mpix/s)	10050/sec (10.29 Mpix/s)	33070/sec (33.86 Mpix/s)
Rect 512x256 Src	3950/sec (517.77 Mpix/s)	3862/sec (506.19 Mpix/s)	6245/sec (818.65 Mpix/s)	299/sec (39.25 Mpix/s)

Then a Copy:

	EXA	UXA	XAA	NoAccel
Copy (Render) 32x32	8884/sec (9.10 Mpix/s)	6362/sec (6.51 Mpix/s)	6320/sec (6.47 Mpix/s)	7467/sec (7.65 Mpix/s)
Copy (Render) 128x128	10909/sec (178.74 Mpix/s)	10845/sec (177.69 Mpix/s)	12141/sec (198.92 Mpix/s)	530/sec (8.69 Mpix/s)

Composite:

	EXA	UXA	XAA	NoAccel
Composite (Src 16) 32x32	5612/sec (5.75 Mpix/s)	1254/sec (1.28 Mpix/s)	5165/sec (5.29 Mpix/s)	14802/sec (15.16 Mpix/s)
Composite (Src 16) 512x256	147/sec (19.32 Mpix/s)	35/sec (4.70 Mpix/s)	307/sec (40.30 Mpix/s)	127/sec (16.76 Mpix/s)
Composite (Src 32) 32x32	7087/sec (7.26 Mpix/s)	6382/sec (6.54 Mpix/s)	6115/sec (6.26 Mpix/s)	30312/sec (31.04 Mpix/s)
Composite (Src 32) 512x256	1845/sec (241.89 Mpix/s)	3538/sec (463.78 Mpix/s)	1731/sec (226.92 Mpix/s)	286/sec (37.60 Mpix/s)

(with this one it is interesting to see UXA with 16-bit depth. Why so? Also with 24- and 32-bit UXA seemed to perform really nicely with special sized 512x256 rect's)

and finally text:

	EXA	UXA	XAA	NoAccel
Text 8px	1726/sec (1.49 Mpix/s)	2295/sec (1.98 Mpix/s)	2490/sec (2.14 Mpix/s)	6830/sec (5.87 Mpix/s)
Text 12px	1658/sec (2.39 Mpix/s)	2265/sec (3.26 Mpix/s)	2274/sec (3.28 Mpix/s)	4550/sec (6.54 Mpix/s)
Text 24px	2099/sec (9.89 Mpix/s)	2383/sec (11.24 Mpix/s)	1978/sec (9.32 Mpix/s)	1846/sec (8.69 Mpix/s)

(Showing that rendering small areas and text is something where the CPU can still outperform GPU)


At the end the mx11mark gives also an total score. From these figures EXA and actually XAA gets the highets scores which is basically the same ~80 total score. XAA mainly getting these scores as it gets really high figures with pure rect rendering. UXA and NoAccel are also even with ~20 total score - NoAccel performing with small are renders and UXA with 512x256 rect's.


OK, my selection from these was the default one - EXA. Seems that at least with out_of_the_box UXA there is still quite much work to be done and I would definitely be interested to see how UXA for ex performs in tuned Intel 965 graphics device.