[Unity3D] Intro to Geometry Shader

This is a simple geometry shader written in CG language in Unity3D editor. Geometry shader, in ShaderLab term, is actually a method in Unity shader program. Even though in other game engines, geometry shader might itself serve as a small program, Unity3D conveniently combines vertex, geometry and fragment shaders into a hodgepodge which maintains its structure by ShaderLab syntax.

First, geometry shader is in a form of a function in Unity, we need to tell the compiler that we want to program a geometry shader. To do that, simply add the following after vertex and fragment shader declaration:

#pragma geometry geom

Second, we need to know geometry shader processes data passed from vertex shader and past them to the next stage, fragment shader.

So as we know, we used to use a struct called v2f to carry data we need from vertex to fragment shader. Now since we have an extra middle stage, we need to rename this struct and add another struct as well:

// self-defined struct that stores data retrieved from GPU<span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>
struct appdata{
	float4 vertex : POSITION;
	float3 normal : NORMAL;
	float2 uv : TEXCOORD0;

struct v2g{
	float4 objPos : SV_POSITION;
	float2 uv : TEXCOORD0;
	float3 normal : NORMAL;

struct g2f{
	float4 worldPos : SV_POSITION;
	float2 uv : TEXCOORD0;
	fixed4 col : COLOR;

(Note that in this specific shader, we want to directly pass object space coordinate to geometry shader because it is easier for later calculation. But it is not definite. )

After these, we are ready to write the geometry shader. We need to consider the structure a little bit:

void geom (triangle v2g input[3], inout TriangleStream<g2f> tristream){
       // here goes the real logic.

There are three important things we need to look at:
a) maxvertexcount indicates the max number of vertices that might come out of this geometry shader, including new vertices you created during this phase or deleted vertices.
b) This method accepts an input of type v2g, regards it as a triangle (since we know that models are made of triangle primitives), and names it as input whose [] operator would give access to the three vertices of the triangle.
c) TriangleStream is a little bit tricky. Note that the return type of this function is void, so that we need another way to pass our modified data to the next stage. inout keyword serves that functionality: since we are processing on triangle basis, we need a structure to output them; Therefore we need this stream object.

Now we need to think about how we can construct the pyramid.


Take a detailed look, and you will find that this pyramid actually consists of two separate triangles. So as we process each triangle, we need to first find the longer side of the triangle and consider this as the central side because in this way we can extrude our pyramid tip from the middle point of this lateral side:


Now we just need to construct three triangles, namely ABC, ACD, and ABD. We first need to find the direction by which we extrude the point A. And that should be the normal of this triangle, which can be calculated using a cross product of two sides. Then we declare a variable called _Factor that describes the intensity of the stretching:

float3 faceNormal = normalize(cross(input[1].objPos - input[0].objPos, input[2].objPos - input[0].objPos));

// determine which lateral side is the longest
float edge0 = distance(input[1].objPos, input[2].objPos);
float edge1 = distance(input[2].objPos, input[0].objPos);
float edge2 = distance(input[1].objPos, input[0].objPos);

float4 CentralPos = (input[1].objPos + input[2].objPos)/2;
float2 CentralTex = (input[1].uv + input[2].uv)/2;

// step(a,x) if x > a return 1 otherwise 0
// so if the multiplication is equal to 1, both steps return 1
// which means that x is longer than either of them
if(step(edge1, edge2) * step(edge0, edge2) == 1){
        CentralPos = (input[1].objPos + input[0].objPos)/2;
        CentralTex = (input[1].uv + input[0].uv)/2;
}else if(step(edge0, edge1) * step(edge2, edge1) == 1){
	CentralPos = (input[2].objPos + input[0].objPos)/2;
	CentralTex = (input[2].uv + input[0].uv)/2;

CentralPos += float4(faceNormal,0) * _Factor;

Now we start to construct triangles using left-handed coordinates (clockwise). First we construct three side triangles we just talked about. After we finish one point, we append it to the triangleStream. After every triangle finishes, we call tristream.RestartStrip(); to indicate that we now start a new triangle.

for(int i = 0; i < 3; i++){
o.worldPos = UnityObjectToClipPos(input[i].objPos);
	o.uv = input[i].uv;
	o.col = fixed4(0,0,0,1);

	o.worldPos = UnityObjectToClipPos(CentralPos);
	o.uv = CentralTex;
	o.col = fixed4(1,1,1,1);

	int nexti = (i+1)%3;
	o.worldPos = UnityObjectToClipPos(input[nexti].objPos);
	o.uv = input[nexti].uv;
	o.col = fixed4(0,0,0,1);


After this, we basically finished the shader, but we can also create a button as well.

        o.worldPos = UnityObjectToClipPos(input[2].objPos);
	o.uv = input[2].uv;
	o.col = fixed4(0,0,0,1);

	o.worldPos = UnityObjectToClipPos(input[1].objPos);
	o.uv = input[1].uv;
	o.col = fixed4(0,0,0,1);

	o.worldPos = UnityObjectToClipPos(input[0].objPos);
	o.uv = input[0].uv;
	o.col = fixed4(0,0,0,1);


Finally, in the fragment shader, we just use the texture coordinates passed from geometry shader to sample our main texture and output its color.

Now we have successfully created a pyramid-like extrusion shaders. But we can still play around with the geometry shader to create other interesting effects like rectangle extrusions, etc.


Thanks for reading. Please leave any comments!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a free website or blog at WordPress.com.

Up ↑

%d bloggers like this: