Mean & Variance

one block per row → strided loop → block_reduce_sum
Launch Grid
Strided Loop
block_reduce_sum
Thread 0 Writes
All Blocks
Global Memory · input[N][C]
N = 6 · C = 8 · one block per row
Global Memory · mean[] · from the previous kernel
computed by row_mean<<<N, threads>>> before this launch
MEAN
Registers · block 0 · one sum per thread
4 threads share row 0 · thread t reads col t, t + 4, …
Global Memory · mean[N]
mean[row] = sum / C
OUT
Kernel Sourcelayer_norm.cu

    
block_reduce_sum is the warp-shuffle + shared-memory reduction from the Block Reduce animation.
Press RUN to launch the kernel
waiting for kernel launch
CLK