Reflecting SPIR-V to generate C# code
This is a post about my experience reflecting SPIR-V shader code to generate C# code!
Problem:
anyone who has worked with shader programming probably knows about the headache of memory layout, and how updating a datatype in the shader doesnt update the corresponding datatype in your engine code. this usually doesnt throw an error, so you can forget about it, continue working, and everything seems fine until something unrelated breaks and you have no idea why, only to find out the memory layout was wrong the whole time.
for example, i was working on a noita like game engine, but gpu accelerated, and i had a struct in my shader code that looked like this:
1struct Particle {
2 int type;
3 vec2 vel;
4};
and i had a corresponding struct in my c# code that looked like this:
1struct Particle {
2 public int type;
3 public Vector2 vel;
4}
except this did not align properly, as i had a 1d array of particles where the gpu expected a different packed memory layout, causing weird artifacting:
the above should both be fully white, but because of the wrong memory layout it has weird aliasing which admittedly looks kinda cool but was not what i needed.
SPIR-V Reflection:
i started by looking at existing tools for reflecting SPIR-V, but they didnt do everything id need. so i created my own as a C# library, which i called ShaderReflect.
it provides a simple API that takes in a SPIR-V binary and gives you a result!
1var reflector = new Reflector(File.ReadAllBytes(path));
2
3if (reflector.Result.Code != ReflectorResultCode.Success)
4{
5 Console.Error.WriteLine($"Failed to parse SPV binary: {reflector.Result}");
6 return 1;
7}
8
9foreach (var layout in reflector.Result.AnalysisResult!.StructLayouts)
10{
11 // Its not guaranteed for a struct definition to have a layout
12 Console.WriteLine($"Struct Layout for {layout.StructDefinition.Name} (Size: {layout.SizeInBytes} bytes):");
13 for (int i = 0; i < layout.Members.Count; i++)
14 {
15 var member = layout.Members[i];
16 var memberLayout = layout.MemberLayouts.ElementAtOrDefault(i);
17 if (memberLayout != null)
18 {
19 Console.WriteLine(
20 $"\tMember: {member.Name}, Type: {member.Type}, Offset: {memberLayout.OffsetInBytes} bytes, Size: {memberLayout.SizeInBytes} bytes, Stride: {memberLayout.StrideInBytes} bytes");
21 }
22 else
23 {
24 Console.WriteLine($"\tMember: {member.Name}, Type: {member.Type}, Layout: (unknown)");
25 }
26 }
27}
28
29foreach (var resource in reflector.Result.AnalysisResult!.BoundResources)
30{
31 Console.WriteLine(
32 $"Resource: {resource.Name}, Type: {resource.ResourceType}, Set: {resource.Set}, Binding: {resource.Binding}" +
33 (resource.InputAttachmentIndex < uint.MaxValue
34 ? $", Input Attachment Index: {resource.InputAttachmentIndex}"
35 : ""));
36}
this exposes struct definitions and their layouts, and bound resources (it also handles entry points and extensions, but those are not relevant here). existing tools like SPIR-V Reflect are more feature complete and also expose layout data for bound resources, but not for arbitrary struct definitions, which this one does.
Code Generation:
the code generation consists of a shader compiler and a code generator.
the shader compiler gets called by msbuild, gathers all shader attributes, compiles them accordingly, and outputs all information to a json file. the code generator then runs with that json file as an argument, so it knows which classes to generate definitions for.
1[GenerateShader("Shaders/SampleShader.vert", "main", ShaderKind.VertexShader)]
2public partial class SampleVertexShader;
3
4[GenerateShader("Shaders/SampleShader.frag", "main", ShaderKind.FragmentShader)]
5public partial class SampleFragmentShader;
definitions.glsl:
1#ifndef DEFINITIONS_GLSL
2#define DEFINITIONS_GLSL
3
4struct Material {
5 vec4 albedo;
6 float metallic;
7 float roughness;
8};
9
10struct Light {
11 vec3 position;
12 vec3 color;
13 float intensity;
14};
15
16layout(buffer_reference, std430) buffer LightBuffer {
17 Light lights[];
18};
19
20layout(buffer_reference) buffer Scene {
21 Material material;
22 Material materials[4]; // testing stuff not actually used.
23 LightBuffer lights;
24 int lightCount;
25};
26
27layout(push_constant) uniform PushConstants {
28 mat4 model; // 64
29 mat4 view; // 64
30 mat4 projection; // 64
31 vec3 cameraPosition; // 12
32 Scene scene; // 32
33} pc;
34
35#endif // DEFINITIONS_GLSL
vertex shader:
1#version 460 core
2#extension GL_EXT_buffer_reference : require
3
4layout(location = 0) in vec3 inPosition;
5layout(location = 1) in vec3 inNormal;
6layout(location = 2) in vec2 inTexCoord;
7
8#include "definitions.glsl"
9
10layout(location = 0) out vec3 fragPosition;
11layout(location = 1) out vec3 fragNormal;
12layout(location = 2) out vec2 fragTexCoord;
13
14void main() {
15 fragPosition = vec3(pc.model * vec4(inPosition, 1.0));
16 fragNormal = mat3(transpose(inverse(pc.model))) * inNormal;
17 fragTexCoord = inTexCoord;
18 gl_Position = pc.projection * pc.view * vec4(fragPosition, 1.0);
19}
fragment shader:
1#version 460 core
2#extension GL_EXT_buffer_reference : require
3
4layout(location = 0) in vec3 fragPosition;
5layout(location = 1) in vec3 fragNormal;
6layout(location = 2) in vec2 fragTexCoord;
7layout(location = 0) out vec4 outColor;
8
9#include "definitions.glsl"
10
11void main() {
12 vec3 normal = normalize(fragNormal);
13 vec3 viewDir = normalize(pc.cameraPosition - fragPosition);
14 vec3 color = vec3(0.0);
15
16 for (int i = 0; i < pc.scene.lightCount; ++i) {
17 Light light = pc.scene.lights.lights[i];
18 vec3 lightDir = normalize(light.position - fragPosition);
19 float diff = max(dot(normal, lightDir), 0.0);
20 vec3 diffuse = diff * light.color * light.intensity;
21
22 color += diffuse * pc.scene.material.albedo.rgb;
23 }
24 outColor = vec4(color, 1.0);
25}
no real c# code is written apart from the shader class definition and the attribute telling the generator which shader to reflect. then you can use it like:
1using System;
2using System.Numerics;
3using ShaderLace;
4using Vortice.ShaderCompiler;
5
6// Simple example of using generated shader classes and structs
7
8// This matches 1:1 with the Material struct in GLSL
9var material = new SampleVertexShader.Material()
10{
11 albedo = new Vector4(1.0f, 0.0f, 0.0f, 1.0f), // red
12 roughness = 0.5f,
13 metallic = 0.0f
14};
15
16ulong someGpuAddressPointingToSceneData = 0x1234567890ABCDEF;
17
18// This also matches 1:1 with the PushConstants struct in GLSL, so you can just do vkCmdPushConstants with a pointer to this struct
19var pushConstants = new SampleVertexShader.PushConstants()
20{
21 model = Matrix4x4.Identity,
22 view = Matrix4x4.Identity,
23 projection = Matrix4x4.Identity,
24 cameraPosition = new Vector3(0.0f, 0.0f, 5.0f),
25 scene = new GpuBufferReference<SampleVertexShader.Scene>(someGpuAddressPointingToSceneData)
26};
27
28Console.WriteLine($"{material.albedo}, {material.roughness}, {material.metallic}");
29Console.WriteLine($"{pushConstants.model},\n{pushConstants.view},\n{pushConstants.projection},\n{pushConstants.cameraPosition}, {((ulong)pushConstants.scene):X16}");
30Console.WriteLine($"{SampleVertexShader.FilePath} - {SampleVertexShader.EntryPoint} - {SampleVertexShader.ShaderKind} - {SampleVertexShader.SpirVBinary.Length} SpirV bytes");
31Console.WriteLine($"{SampleFragmentShader.FilePath} - {SampleFragmentShader.EntryPoint} - {SampleFragmentShader.ShaderKind} - {SampleFragmentShader.SpirVBinary.Length} SpirV bytes");
32Console.WriteLine("Shader generation complete.");
internally SampleVertexShader.Material and PushConstants are structs marked with [StructLayout(LayoutKind.Explicit)], with each field offset matching the corresponding SPIR-V offset.
GpuBufferReference is an internal 64-bit handle with a type parameter for clarity. it cant verify your gpu address actually points to that type, but it makes intent obvious at a glance.
here is what the SampleVertexShader class gets generated into:
1/// Generated code do not edit.
2
3public partial class SampleFragmentShader
4{
5 public static string? FilePath { get; } = "Shaders/SampleShader.frag";
6 public static string? EntryPoint { get; } = "main";
7 public static Vortice.ShaderCompiler.ShaderKind ShaderKind { get; } = Vortice.ShaderCompiler.ShaderKind.FragmentShader;
8 public static byte[] SpirVBinary { get; } = new byte[] {...}
9
10 public string? GetFilePath() => FilePath;
11 public string? GetEntryPoint() => EntryPoint;
12 public Vortice.ShaderCompiler.ShaderKind GetShaderKind() => ShaderKind;
13 public byte[] GetSpirVBinary() => SpirVBinary;
14
15 [System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Explicit, Size = 216)]
16 public unsafe struct PushConstants
17 {
18 [System.Runtime.InteropServices.FieldOffset(0)]
19 public System.Numerics.Matrix4x4 model;
20 [System.Runtime.InteropServices.FieldOffset(64)]
21 public System.Numerics.Matrix4x4 view;
22 [System.Runtime.InteropServices.FieldOffset(128)]
23 public System.Numerics.Matrix4x4 projection;
24 [System.Runtime.InteropServices.FieldOffset(192)]
25 public System.Numerics.Vector3 cameraPosition;
26 [System.Runtime.InteropServices.FieldOffset(208)]
27 public ShaderLace.GpuBufferReference<Scene> scene;
28 }
29
30 [System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Explicit, Size = 32)]
31 public unsafe struct Material
32 {
33 [System.Runtime.InteropServices.FieldOffset(0)]
34 public System.Numerics.Vector4 albedo;
35 [System.Runtime.InteropServices.FieldOffset(16)]
36 public float metallic;
37 [System.Runtime.InteropServices.FieldOffset(20)]
38 public float roughness;
39 }
40
41 [System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Explicit, Size = 172)]
42 public unsafe struct Scene
43 {
44 [System.Runtime.InteropServices.FieldOffset(0)]
45 public Material material;
46 [System.Runtime.InteropServices.FieldOffset(32)]
47 public InlineArray_Material_4 materials;
48 [System.Runtime.InteropServices.FieldOffset(160)]
49 public ShaderLace.GpuBufferReference<LightBuffer> lights;
50 [System.Runtime.InteropServices.FieldOffset(168)]
51 public int lightCount;
52 }
53
54 [System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Explicit, Size = 32)]
55 public unsafe struct Light
56 {
57 [System.Runtime.InteropServices.FieldOffset(0)]
58 public System.Numerics.Vector3 position;
59 [System.Runtime.InteropServices.FieldOffset(16)]
60 public System.Numerics.Vector3 color;
61 [System.Runtime.InteropServices.FieldOffset(28)]
62 public float intensity;
63 }
64
65 [System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Explicit)]
66 public unsafe struct LightBuffer
67 {
68 /// <summary>
69 /// Runtime array of Light
70 /// Note: Runtime arrays cannot be directly represented in C# structs.
71 /// You will need to handle this manually when working with buffers containing this struct.
72 /// </summary>
73 public static readonly System.Type lightsType = typeof(Light);
74 }
75
76
77 [System.Runtime.CompilerServices.InlineArrayAttribute(4)]
78 public unsafe struct InlineArray_Material_4
79 {
80 public Material Element0;
81 }
82}
Difficulties
most issues i ran into were related to msbuild and source generators.
c# source generators need to be written in netstandard2.0 and dont support external libraries without some awkward workarounds. i originally started with just the code generator, but had to split the project up since the source generator couldnt use the SPIR-V compiler library.
getting msbuild to actually execute the shader compiler was also a pain, and i eventually just made the shader compiler a dotnet tool so msbuild could call it cleanly.
Conclusion
having compile time type and field access checking for shader development is really nice. apart from modifying the .csproj for initial setup, it requires no manual intervention. it increased my productivity a good bit, since i just no longer have to worry about something that always managed to bite me.
i might release it publicly, but it still has a long way to go. im also considering porting it to c, c++, or other languages, but for now c# will do.