Shadows are a part of nearly every game created nowadays, and I've never quite been happy with how they appear. They often appear pixelated, have no color, or require expensive methods of rendering.
For this reason, I decided to see if I could implement a method of shadow rendering which could make shadows more appealing.
Originally I only intended to make colored shadows. However, after managing to implement them in a single day, I decided to also make soft shadows. The image shown is the final result of both colored and soft shadows together. The colored shadows are not softened, only regular ones are.
Colored shadows were quite easy to implement. I found an article by turanszkij which explains how transparent shadowmaps are rendered. The method is simple: render your meshes from the light's perspective just like with regular shadows, then render your pixel shader, and lastly use a blend state which multiplies the source and target color.
Blending of colors with adjustable alpha values.
CMYK color blending.
The main issue with this approach is that it doesn't blend correctly with adjustable alpha values, as one can't both multiply with target and alpha in a blend state. I decided to go with a different approach: the shader would calculate the proper blending with alpha, and the blend state would multiply source and target.
In order to avoid creating two separate shaders, one for normal and one for shadow rendering, a simple function is used to convert normal output into a shadow color, and the shader outputs the shadow color onto SRC1. The blend state accesses this color and blends appropriately. This also allows for blending of textures that have varying alpha values.
Additionally I created a stained glass model in Blender to show what the effect can look like when used effectively.
A church-like environment indended to showcase the effect of colored shadows.
The performance of colored shadows seems adequate. In this scene, without any of the windows, I get about 770 FPS. With all of them on, it is about 90 FPS. This seems like a very big performance hit, but in this scenario every window has 42 separate meshes, and although there are 4 identical windows, the individual elements almost always have different materials. When combined, the performance goes up to 720 FPS, as the engine can do automatic instancing.
First time I made soft shadows was while I was still in gymnasium at LBS. I made a soft shadow method what I now know is called Contact Hardening Soft Shadows (CHSS). The specific method I made did a circular sample with a size dependent on the depth of the shadow caster. It wasn't great or terrible.
The main issue it had was the fact that I couldn't handle blending between shadows of two objects that had wildly different depths, and for this reason I gave up with the attempt.
First test of CHSS. 96 shadow samples. Cast by a building.
Bad shadows due to shadow casters of different depths.
I thought it was fitting to try and replicate and improve this effect, now that I have 2 more years of experience and can more effectively search for information online.
Attempting again only proved the limitations of CHSS. It cannot handle shadow casters of different depths. I was able to both reduce the amount of samples and improve the quality drastically using a Vogel Disk to sample the shadowmap, but could never get around the issue of shadows blending poorly.
Shadow sampling with Vogel Disk.
Fun bug caused by incorrect shadow sampling.
Shadow sampling based on distance from caster. Looks bad when blended together.
Instead of trying to make CHSS work, I decided to find a different method of generating soft shadows. One promising example I found uses something called Smoothies. The general idea is simple: Render shadows normally, then extrude an edge along the outline of the mesh which would get rendered as the soft shadow, going from full shadow intensity at the root to completely transparent at the tip. These mesh outlines are called Smoothies. Another advantage of this method is that since it was first invented in 2003, it should be quite performant.
The method generated Smoothies by using triangle adjacency data in a geometry shader to determine if one face is facing towards the light while the other is facing away. This would mean the edge is an outline, and one should generate smoothies.
The meshes I would load in as .fbx files would come in without adjacency data, so therefore the first step was to generate this data.
After implementing an algorithm which generates adjacency data, I discovered it does not take into account flat shading. Flat shading causes the modelling program to separate all triangles into having their own vertices, meaning there can be multiple vertices in the same place.
I solved this by going through all edges and seeing if there are any duplicates(i.e. edges that have identical vertex positions), and then adding them to an edge-twins map.
Incorrect vertex adjacency calculation.
The original paper describes the process as extending the outline outwards, writing it in a smoothie buffer, and then during the shading process uses the smoothie depth to determine how dark the soft shadow should be. However, this approach has the downside of not working well with overlapping shadows; one smoothie would overwrite another smoothie's information.
I decided to go with a slightly different approach. The new approach involves shifting the smoothies' tips downwards in order to create a sort of frustum; the distance to the smoothie depth is still used to calculate how shaded the shadow should be, but overlapping shadows should no longer have issues overlapping.
(Whilst writing this I discovered my approach is almost identical to that of Penumbra Maps, which was mentioned in the original smoothie article but since the link was not working I didn't investigate it)
Shadows from the cube(lower down) getting overwritten by an arch (further up)
After implementing I discovered an issue; since the method required the depth at the root of the outline, it would get overwritten when a smoothie cast a shadow over another smoothie, causing partically shaded shadows to be overwritten with the lighter values and blend incorrectly. This meant I had to change my approach.
Instead of doing the calculation of how shaded the smoothie shadows should be whilst doing the shading, I decided to calculate it in a pixel shader right after generating the smoothie outlines, by sampling the normal shadowmap to determine the depth, which would then be written in the smoothie buffer.
The approach worked but created some visual artifacts. The smoothie shadows would be separated from the normal shadowmap by a pixel. I tried fixing it with conservative rasterization, but that simply caused the gap to move outwards by a pixel. The issue was fixed by creating a small inwards extension to the smoothie, marked with a bool, and sampling in a 3x3 square around the pixel to get the highest depth.
The issue still slightly persists in the form of slight flickering of single pixels on corners where the outline is concave, but I think the effect isn't too distracting.
Another minor issue this caused was that it would cause black outlines around the shadow caster, but this was easily solved by ignoring the shadow if the smoothie depth was below that of the pixel being rendered.
A small outline of light pixels between regular and smoothie shadows.
A black outline caused by inwards smoothie extensions.
Lastly, in order to completely hide all pixelated artefacts from the normal shadowmap, I do 12 vogel disk samples around the pixel in order to smooth out the shadows, similar to percentage-closer filtering.
The final result looks quite nice. The smoothie shadows cause negligible performance impact. The biggest performance hit is due to the vogel disk sampling, which drops FPS from 390 to 320, although this could probably be improved.
An example of smoothie shadows becoming smoother with distance.
Soft shadows blending correctly with shadow casters of largely different heights.
The final result. Both soft shadows and colored shadows together.