SFU CS Architecture Group
Software Engineer (Systems), Research
Reproduced CUDA GEMM kernels to 93.7% of cuBLAS throughput and authored a GPU kernel guide for undergraduates.
Overview
Chasing NVIDIA's own math library on its own hardware, then writing down how, so the next person does not have to start from zero.
What happened
The group's question was why large matrix multiplications dominate the cost of training and running LLMs, and how close you can get to peak hardware by hand. That meant getting underneath transformer attention to the linear algebra that eats the compute.
I reproduced CUDA GEMM kernels and tuned them against cuBLAS, NVIDIA's own library, landing at 93.7% of its throughput. Getting there meant living in Nsight Compute and reading the profiler instead of guessing, then going down to the PTX assembly to see what the compiler was really doing to my code.
The kernels were only half of it. I wrote a guide to GPU matrix-multiply optimization that builds the mental model from the execution hierarchy up: thread blocks, warps, SM scheduling. Someone new to the GPU can follow the reasoning behind each optimization instead of copying it.