Close Menu
techwavedispatch.com

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Exact Bayesian layers via characteristic functions

    November 21, 2026

    Few-Shot Cross-City Traffic Prediction with Conditional Diffusion

    November 21, 2026

    Simulation-pretrained autoencoders versus PCA for compressing multimode fibre transmission matrices

    November 21, 2026
    Facebook X (Twitter) Instagram
    Trending
    • Exact Bayesian layers via characteristic functions
    • Few-Shot Cross-City Traffic Prediction with Conditional Diffusion
    • Simulation-pretrained autoencoders versus PCA for compressing multimode fibre transmission matrices
    • Design and Evaluation of Epira: An Automated Interview System for Case Investigation and Contact Tracing
    • [2611.26291] Who Moved My $Z$ Boson? Revealing Resonance Features Inside a Neural Network
    • [2611.25570] Reinforcement-Aligned Probabilistic Weather Forecasting with Discrete Tokens
    • Backbone Matters: Benchmarking Encoder-Decoder Networks for Semantic Segmentation of Driving Scenes on BDD100K
    • Is a Description Worth a Thousand Demonstrations? Specificity and Reliability of Task Descriptions in In-Context Learning
    Facebook X (Twitter) Instagram
    techwavedispatch.com
    Subscribe
    Saturday, November 21
    • Home
    • AI Tools & Software
    • Machine Learning
    • Generative AI
    • Robotics & Automation
    • Future Tech & Hardware
    • AI in Business
    • Ethics & Regulations
    • More
      • Gadgets & Smart Devices
      • Cybersecurity & AI
      • Tech Breakthroughs
    techwavedispatch.com
    Home»AI Tools & Software»Block Sparse Matrices for Smaller and Faster Language Models
    AI Tools & Software

    Block Sparse Matrices for Smaller and Faster Language Models

    adminBy adminNovember 16, 2026No Comments3 Mins Read
    Facebook Twitter Pinterest LinkedIn Tumblr Email
    Block Sparse Matrices for Smaller and Faster Language Models
    Share
    Facebook Twitter LinkedIn Pinterest Email



    Saving space and time, one zero at a time

    In previous blog
    posts
    we introduced sparse matrices and what they could do to improve neural networks.

    The basic assumption is that full dense layers are often overkill and can be pruned without a significant loss in precision.
    In some cases sparse linear layers can even improve precision or/and generalization.

    The main issue is that currently available code that supports sparse algebra computation is severely lacking efficiency.
    We are also still waiting for official PyTorch support.

    That’s why we ran out of patience and took some time this summer to address this “lacuna”.
    Today, we are excited to release the extension pytorch_block_sparse.

    By itself, or even better combined with other methods like
    distillation
    and quantization,
    this library enables networks which are both smaller and faster,
    something Hugging Face considers crucial to let anybody use
    neural networks in production at low cost, and to improve the experience for the end user.



    Usage

    The provided BlockSparseLinear module is a drop in replacement for torch.nn.Linear, and it is trivial to use
    it in your models:

    
    from pytorch_block_sparse import BlockSparseLinear
    
    ...
    
    
    self.fc = BlockSparseLinear(1024, 256, density=0.1)
    

    The extension also provides a BlockSparseModelPatcher that allows to modify an existing model “on the fly”,
    which is shown in this example notebook.
    Such a model can then be trained as usual, without any change in your model source code.



    NVIDIA CUTLASS

    This extension is based on the cutlass tilesparse proof of concept by Yulhwa Kim.

    It is using C++ CUDA templates for block-sparse matrix multiplication
    based on CUTLASS.

    CUTLASS is a collection of CUDA C++ templates for implementing high-performance CUDA kernels.
    With CUTLASS, approching cuBLAS performance on custom kernels is possible without resorting to assembly language code.

    The latest versions include all the Ampere Tensor Core primitives, providing x10 or more speedups with a limited loss of precision.
    Next versions of pytorch_block_sparse will make use of these primitives,
    as block sparsity is 100% compatible with Tensor Cores requirements.



    Performance

    At the current stage of the library, the performances for sparse matrices are roughly
    two times slower than their cuBLAS optimized dense counterpart, and we are confident
    that we can improve this in the future.

    This is a huge improvement on PyTorch sparse matrices: their current implementation is an order of magnitude slower
    than the dense one.

    But the more important point is that the performance gain of using sparse matrices grows with the sparsity,
    so a 75% sparse matrix is roughly 2x faster than the dense equivalent.

    The memory savings are even more significant: for 75% sparsity, memory consumption is reduced by 4x
    as you would expect.



    Future work

    Being able to efficiently train block-sparse linear layers was just the first step.
    The sparsity pattern is currenly fixed at initialization, and of course optimizing it during learning will yield large
    improvements.

    So in future versions, you can expect tools to measure the “usefulness” of parameters to be able to optimize the sparsity pattern.
    NVIDIA Ampere 50% sparse pattern within blocks will probably yield another significant performance gain, just as upgrading
    to more recent versions of CUTLASS does.

    So, stay tuned for more sparsity goodness in a near future!

    Block Faster Language Matrices Models Smaller Sparse
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    admin
    • Website

    Related Posts

    Simulation-pretrained autoencoders versus PCA for compressing multimode fibre transmission matrices

    November 21, 2026

    A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training

    November 21, 2026

    Interference Bounds and Merge-Aware Training Dynamics for Composable Fine-Tuned Models

    November 20, 2026
    Leave A Reply Cancel Reply

    Top Posts

    NVIDIA Is Investigating Reports That CEO Jensen Huang Was The Target Of Chinese Spy Efforts

    November 8, 202619 Views

    ADIndex: Layer-Adaptive Sparse Attention Indexing via Group-Level Distillation

    November 11, 20265 Views

    Anthropic releases new Mythos and Fable models amid RSI concerns

    November 9, 20265 Views

    If AGI is so close, why is everyone leaving? The neolabs and their bets

    November 6, 20265 Views
    Don't Miss
    Machine Learning

    Exact Bayesian layers via characteristic functions

    By adminNovember 21, 20260

    arXiv:2611.26426v1 Announce Type: cross Abstract: Bayesian neural networks provide principled uncertainty estimates, but training them…

    Few-Shot Cross-City Traffic Prediction with Conditional Diffusion

    November 21, 2026

    Simulation-pretrained autoencoders versus PCA for compressing multimode fibre transmission matrices

    November 21, 2026

    Design and Evaluation of Epira: An Automated Interview System for Case Investigation and Contact Tracing

    November 21, 2026
    About Us

    Welcome to TechWave Dispatch

    At TechWave Dispatch, we bring you the latest news, insights, and developments from the ever-evolving world of technology. Our mission is to keep readers informed about the innovations shaping our digital future.

    From artificial intelligence and machine learning to cybersecurity, robotics, smart devices, and groundbreaking technologies, we cover the topics that matter in today's fast-moving tech landscape.

    Our Picks

    Exact Bayesian layers via characteristic functions

    November 21, 2026

    Few-Shot Cross-City Traffic Prediction with Conditional Diffusion

    November 21, 2026

    Simulation-pretrained autoencoders versus PCA for compressing multimode fibre transmission matrices

    November 21, 2026
    Most Popular

    After blocking Muse, Amazon now lets users text Alexa to help them shop

    November 6, 20260 Views

    A controlled world for robot learning and evaluation

    November 6, 20260 Views

    Egyptian and Nigerian English join the Open ASR Leaderboard

    November 6, 20260 Views
    Facebook X (Twitter) Instagram Pinterest
    • Terms and Conditions
    • Privacy Policy
    • Contact Us
    • About Us
    © 2026 TechWave Dispatch. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.