Optimizing matrix multiplication on NVIDIA GPU.

Introduction.

Memory bound vs Compute bound for Matrix Multiplication.

Kernel 1: Naive implementation.

Kernel 2: .

Kernel 3: Shared Memory Cache-Blocking.

Kernel 4: .

Kernel 5: .

Kernel 6: .

Conclusion

This technical blog discussed step by step on how to optimize kernel optimization for matrix multiplication of NVIDIA GPU.

References

• CUDA C++ Best Practices Guide
• SOL Analysis with NVIDIA Nsight Compute
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
• Fast N-Body Simulation with CUDA