Scott Gray

2 posts

Generative Modeling with Sparse Transformers
Generative Modeling with Sparse Transformers

We've developed the Sparse Transformer, a deep neural network which sets new records at predicting what comes next in a sequence—whether text, images, or sound.


Block-Sparse GPU Kernels

We’re releasing highly-optimized GPU kernels for an underexplored class of neural network architectures: networks with block-sparse weights. Depending on the chosen sparsity, these kernels can run orders of magnitude faster than cuBLAS or cuSPARSE.


5 minute read