Image space algorythms, polycount and fps

Image-space algorythms have one huge benefit: their costs are independent from the overall complexity of the scene. Proof is that putting a ~35k faces model in the middle of the scene does not affect the frames per seconds count.


For my experience what seem to drain GPU power are (in ascending order of heaviness):

• multisampling - useful for sorting-independent transparency, it's not for free and there's no better choice.
  
  
• accumulation buffers - I use them extensively for assembling lighted parts, shadows, and translucency maps: it probably could be done faster with a blend, or using shaders. Can work on it.
  
  
• state changes - the most stunning one: an hidden performance bloodsucker, the only one that isn't proportional to the viewport resolution; requires wise programming.
  
  
• convolution filters - god, despite the fact they're GPU-bound being programmed via shaders, they slow as hell. Ok, maybe to convolve a multisampled 800x450 window with a 5x5 kernel on a two years old video card isn't a great idea, but there must be a way to improve this performances.
  
  

Expensive state changes

Everytime you want a primitive to be multi-sampled - in order to convert its alpha values into coverage patterns - you are supposed to explicitly ask for it by enabling GL_SAMPLE_ALPHA_TO_COVERAGE.

I thought that keeping this feature enabled for non-translucent primitives was a waste. I was wrong! It's continuosly changing this state that cuts down performances!

Therefore, you're going to prefer to let alpha2coverage enabled whereas supported, and enjoy the terrific boost of performances.

Unnatural convolution

By using a simple convolution filter, with a small 5x5 kernel running on a fragment shader, one can obtain a nicer and softer shadows.

This trick was invented by Anirudh Shastry; it's not physically correct, it's prone to artifacts (and that's why I'm gonna use stencils) but it's good to see and pretty fast. Moreover, I didn't have to move the whole rendering pipeline in one shader, which is good.

Don't you know, talking 'bout the convolution it sounds like a whisper

Hard shadows destroy realism. It's a matter of fact, they look too unnatural, expecially if the rest of the scene appear softly shaded and we overlap a simulated cone of light.

Plenty of solutions have been proposed. Furthermore, Nvidia cards are so smart that they automatically apply the percentage closer filtering on shadow maps. ATI cards don't, but proposed a pretty way to implement it (by using their proprietary multiple fetching primitives).

I don't want to use proprietary features, which could lead to unexpected situations, so I took the decision to try something new: by using stencil buffers, I'm gonna separate "potentially shadowed" texels and blur them, with some kind of convolution filter:


Pros: it's REALLY fast (dirt work is done via fragment shader), it's lightweight (just an additional RGB texture is required), and it's image-space based, so I don't need to program lighting equations for each texture unit in the shader!

Cons: additional rendering steps are needed, one for each light; self-shadowing artifacts are supposed to emerge and there's no difference between shadows near and far away from light (which is best aspect of smarter algorythms like PCSS or VSM).

We'll see. In the meanwhile, I express my disappointment for ATI's engineers who didn't really implement glConvolution*D() in their drivers :| .

Causticity

A preliminary shot of my "translucency shadow mapping" engine, showing just the "filtrating light" layer. Light illuminates the glass, which projects its texture on the transparent radiation box, which is supposed to project on the wall as well.

Seems working, but actually it doesn't.

The reason is subtle: the more translucent a surface is, the more it shouldn't stop the light; this means that the projected shadow should appear brighter and colorful, but it does not.

This is due to the fact that I create a sort of translucency map by rendering the translucent objects from light's pov, and the resulting image gets darker as the alpha decreases! Some kind of "alpha inversion" is needed, maybe pre-calculated. But this means much more memory, linearly dependent from the number of lights and the resolution of viewport!! Not suitable.

A better solution should be caustic mapping, which is definitively one of the coolest algorythms I've ever seen. The results are impressive, but there are (as usual) some drawbacks: additional geometries (for refractive vertex grid and the projected points), no way to sum up many translucent contributions and so on. It would be interesting to implement it some day, but there's no time to play with it right now.

Covert alpha operations

The screenshot on the left shows my shadow mapping engine rendering two translucent textured surfaces. The first one occludes the spotlight and casts a shadow on the second one which, in turn, casts on the floor and the wall.

Through the eye of the needle

The thumbnail looks fine, but a closer look (click it!) reveils many subtle aliasing problems. Fact is that the "alpha to coverage" is just a workaround to achieve a nice effect (transparency) without adding too much complexity (sorting/splicing/etc). Its trick consists in converting the alpha informations into coverage micropatterns. When the surface is analyzed badly - eg.: shadow map generation from big distance - "non-covert" subfragments reveal themselves. This issue can be partially solved by linear filtering the shadow map, but could also create an awful moiré effect.


Images from the other side

Finally, there remains the open issue of correctly projecting the colors when lights runs through translucent surfaces. The solution I'm trying to develop is the following:

• render the scene without translucent objects
• take the depth map of this partial scene (this will block projection beyond dull surfaces)
  
• do render anything stays above (using the depth map for depth comparison)
  
• take the color map of this scene and blend it over the shadow map

Thoretically it's a working algorythm, but there are surely plenty of complex cases that I'm ignoring.

Projective light cone

Due to its intrinsic nature of texture, the shadow map looks squared.

A nice way to improve its appearance is to use the very same coordinates that project the shadow to map an additional bitmap, which simulates the bottom of the light cone produced by the "spotlight".

Now it would be cool to simulate the whole light cone by using some kind of volumetric escamotage.

Aliasing galore and FBOs

When you're going to write a 3D shadows engine, your choice is basically restricted to two possibilities: shadow volumes or shadow mapping in one of its flavours.

There's no reason to choose shadow mapping but the performances: shadow volumes are more accurate, because a shadow map is a simple texture spread on a scene and used for depth comparisons. It's obvious that texture's detail affects render realism, and all sort of aliasing will happen.

The quickest way to improve an image-space based algorythm is to improve the image.. space! You should render the scene with higher resolution. But you can't. Reason is that even if you raise the viewport size, anything greater than the actual container window will be dropped.

There are also restrictions for the texture shape and dimensions, but you could get around them using some of the latest extensions (ARB_texture_rectangle, ARB_texture_non_power_of_two)... but anyway, the problem remains: we are in need for detail. There are sophisticated geometric solutions, such as trapezoidal/perspective shadow maps and cascading shadow maps, but we could simply... render elsewhere!

Here come frame buffer objects. They are basically "containers" that you can use as targets for your renders. You can also use them to directly render on a texture, and this is our case: we render there our huge shadow map.



The difference is clear and performance hit is not dramatic. The coolest part is that using FBOs does not take modifications to previous routines (just bind the directly generated texture as usual!), and can be turned on in every moment. Approved!

Pseudo gamma correction

As several lights add up in the scene, several accumulations happen in the background, flattening average contrast and luminance. In order to have the colorful scene back, we can tweak the accumulation buffer a bit using the GL_MULT function of glAccum API, and passing as multiplication factor the result of 1+γ*numberOfLightSources.

Here's a screenshot:


compared to the previous images, also with γ=.2 the improvement is pretty evident.

Put on the red light

Who said lights are just white? Introducing an almost coherent red light.

More then ever, hard shadows and light's cone edge appear too unnatural. Some kind of linear filtering is required here, maybe a PCF for shadows and a bit of blur for cone's boundaries.

N lights are better than one

Here we go! Accumulation buffers allowed an easy drawing of multiple light sources and relative shadows.

The trickiest aspect was the balance of exposition: every light features its accumulation factor and the light contributions are supposed to be under 1, in order to avoid overexposition.

It could be a cool effect, anyway. =)

Accumulation buffers

Shadow mapping is working correctly, but only one (spot!) light at a time.

Nature is very different, so engine has to be changed a bit. How can one sum the contributions of several different lights, placed around the scene? Accumulation buffers can do the trick! These supplementary buffers will sum the RGB values of our different takes from any single light and return the final composite bitmap.

This screenshot shows a typical application of accumulation buffers: motion blur. With a very minimal modification of original dev-cpp's GLUT example we can achieve this interesting effect.

In pseudocode:

for (;;) {
  glClear (GL_COLOR_BUFFER_BIT | GL_ACCUM_BUFFER_BIT)
  const range = 10
  for i=1 to range do {
    render(i);  
    glAccum(GL_ACCUM, 1.0f/range)
  }
  glAccum(GL_RETURN, 1)
  swapBuffers
}

Three brief comments.

First, the range means "how many different takes we wanna use for motion blur overlaps". This example is not optimized in any way and renders 10 times the scene for each pass.

Second, the render is bound to the i variable. The meaning is that we want to simulate movement, so it makes no sense to render 10 times the same static scene. In the example, I did the objects rotate "i degrees".

Third, the accumulation command takes a floating point value, which tells how much of the RGBA values we want to sum up in the buffer. Given that these values are clamped in the [0,1] range, and we're going to sum 10 shots, we can set these values in a way that the final count is 1: 10*(1/10) = 1.

The same philosophy shall work for multiple lights: we sum up the n contributions of n lights.

Young 3D engines suffer acne

As the engine grows up, I refactor some routine. Cleaner code allows me to use some higher level function, like a 3Ds objects loader. And here appears a well-know issue: z-fighting.

Even on a surface closest to the light source, you will always discover minor differences in the values associated with the R texture coordinate(...). This can result in "surface acne"(...). You can mitigate this problem by applying a depth offset when rendering in the shadow map:

...
glEnable (GL_POLYGON_OFFSET_FILL);
glPolygonOffset (factor, 0.0f);
...

(...)A balance need to be struck when it comes to polygon offset usage.

from OpenGL superbible 4th edition by Richard Wright (Wesley).

That balance is not so easy to achieve; as Nvidia says in its papers, it's really matter of art.

When shadows and light combine

Now it's not just primitives and smooth shading! Multitexturing is part of OpenGL since 1.2.1 specification and allows direct textures combining. Putting the shadow map into a separate texture unit (with glActiveTexture) permit us to spread it on our already textured polygons.

I don't know if this is the most performant way to achieve a correct result, but it works fine! =)

I can't see the light!

Now we're talking.

And guess what? Nothing was wrong. I tried running the code on my work's ATI x1300 and shadows correctly appeared.

I suppose my SIS 650 messes up with alpha testing. Disappointing.

And there's another useful information: OpenGL do not work through remote desktop. If you try to connect via terminal services, you'll see just flat shaded polygons. What I can't see is the logic behind this. Everything is computed on "server" side and I don't think there's a particular boost of network performances not shading and texturing. I'm pretty sure RDP protocol does a run length encoding optimization, but a spinning cube requires more updates on client's side than a static decaled teapot. Mysteries.

Neverending story

Something weird is going on here..! Shadows are supposed to be dark! Oo"

..at least they appear now!

Reinventing the wheel

Matrix multiplication is fundamental in computer graphics, expecially when you're dealing with shadow maps and texture projections.

Unfortunately, there are no standard functions for matrix multiplication, so it was time for DIY. Here comes my snippet:

float* matrixProduct(float* A, float* B) {

// we have to allocate space for a 4x4 floats matrix
float* pointer= (float *) calloc(sizeof(float), 16);

// in order to compute the resulting matrix, we have to use two nested fors
for (int r=0; r<4; r++) {
for (int c=0; c<4; c++) {

    // actual multiplications
    for (int i=0; i<4; i++) {
      pointer[r*4+c] = pointer[r*4+c] + A[r*4+i]*B[c+i*4];
    }
}
}    

return pointer;

}

This simple function returns a pointer to an handful 4x4 array of floats, without memory flaws. Maybe this snippet will be helpful for someone in the future.

Work in progress

The answer to the previous question was: "it does work". I don't know how the framework I'm working on exactly works, but it does strange things with depth buffer.

It seems I'm at a good point. Shadow mapping consists in three steps (two, if the video cards supports the GL_ARB_shadow_ambient extension):

1. depth buffer acquisition from light's point of view
2. a "dim light" rendering from camera's point of view
3. full light rendering w/ shadow map depths comparison

Screenshots prove that everything seems to works correctly.

New question is: why doesn't the shadow map show up..? ^^"
Damn!

Uncomfortable

Summer and CG don't meet: my warfare computer resists a couple of minutes and then hangs up.

I'm working on my trusty notebook now, but the SiS graphics card is resulting a major pain in the ass:

• warning: OpenGL VBOs not supported
• warning: OpenGL shading language not supported

God! Any facility that should help me out to work on shadow mapping is not present on this machine. It's like programming in late 90s. By the way, it could tactical for me: willing or not I'm forced to develop an engine whose performances could be tweaked later, by using advanced functionalities.

There's, anyway, a couple (of thousands) of opened issues to fix before "popping champagne". For instance, why the hell reading depth buffer with glReadPixels() and putting it into the frame buffer (as luminance) using glDrawPixels() does not work? I'm surely missing something... but what?!

And the winner is: shadow mapping

Shadows projection can be tricky. I knew it before beginnig my stage at CNR, and had only confirmations so far: there's a lot of complications you've got to consider before developing. The greatest limit remains, of course, the cost of computations.

My first choice and proposal were shadow volumes a là Carmack: definitively intriguing and kind of ideal trait d'union of my universitary life: they were perfect.

Analyzing the technology, it turned out that they were not. The software module I'm asked to develop will be placed in high-polygon contexts; shadow volumes require a lot of time for each object: time to place the supplementary geometries in the pipeline, time to compute the silhouettes (checking every triangle of the mesh!!), time to extrude the cones that will be depth-checked to fill the stencil buffers. Many polygons kill performances; many polygons and many lights means killing real time rendering.

So, after a brief talk with chief, we took the decision to simulate shadows by shadow mapping: not so accurate like shadow volumes, but really faster! In addition, they have less issues, like volumes bounding and silhouette recognition.

I'm gonna try to develop them both using and not using shaders, because of accuracy and scalability.

My module is supposed to be arbitrarily activated, and selective shadowing must be guaranteed. Again, both things are easier to be achieved with shadow mapping:

• I don't have to do any invasive manipulation to the existing geometry loader classes (because of additional squares for shadow volumes), so one could turn off shadows just by not calling my procedures;
• when I render "from the light view", I can just don't place objects in the scene if I don't want them to project a shadow. Again, this could be obtained without in-depth modifications of existing routines and placing my functions in a strategic way.

Everything's nice and exciting... I hope I'll be able to do it! ^^