Close Menu
techwavedispatch.com

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

    November 22, 2026

    Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

    November 22, 2026

    Learning from Context: Leave-One-Out Prediction in Jump Diffusions

    November 22, 2026
    Facebook X (Twitter) Instagram
    Trending
    • HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
    • Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
    • Learning from Context: Leave-One-Out Prediction in Jump Diffusions
    • Represented but misread: Post-training changes how language models read out what they represent
    • Mamba Flow Matching Neural Processes: Linear-Time Inference for Irregularly Observed Spatial Fields
    • Cost-Aware Agentic Feature Optimization System for Fleet-Scale Ranking Models
    • TRCF: Temporal-Reference Complementary Forecasting for Irregular Multivariate Time Series
    • [2611.26515] Simple Private Noisy SGD is Minimax-Optimal under Population Strong Convexity
    Facebook X (Twitter) Instagram
    techwavedispatch.com
    Subscribe
    Sunday, November 22
    • Home
    • AI Tools & Software
    • Machine Learning
    • Generative AI
    • Robotics & Automation
    • Future Tech & Hardware
    • AI in Business
    • Ethics & Regulations
    • More
      • Gadgets & Smart Devices
      • Cybersecurity & AI
      • Tech Breakthroughs
    techwavedispatch.com
    Home»Machine Learning»HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
    Machine Learning

    HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

    adminBy adminNovember 22, 2026No Comments1 Min Read
    Facebook Twitter Pinterest LinkedIn Tumblr Email
    HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
    Share
    Facebook Twitter LinkedIn Pinterest Email

    arXiv:2611.26363v1 Announce Type: new
    Abstract: Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective. We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.

    Attention Dimension hidden HiSA LLMs RoPE Sparsifying
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    admin
    • Website

    Related Posts

    Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

    November 22, 2026

    Learning from Context: Leave-One-Out Prediction in Jump Diffusions

    November 22, 2026

    Represented but misread: Post-training changes how language models read out what they represent

    November 22, 2026
    Leave A Reply Cancel Reply

    Top Posts

    NVIDIA Is Investigating Reports That CEO Jensen Huang Was The Target Of Chinese Spy Efforts

    November 8, 202619 Views

    ADIndex: Layer-Adaptive Sparse Attention Indexing via Group-Level Distillation

    November 11, 20266 Views

    Anthropic releases new Mythos and Fable models amid RSI concerns

    November 9, 20266 Views

    One Policy for 100 Household Tasks: Our Solution for the 2026 BEHAVIOR Challenge

    November 11, 20265 Views
    Don't Miss
    Machine Learning

    HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

    By adminNovember 22, 20260

    arXiv:2611.26363v1 Announce Type: new Abstract: Attention in large language models (LLMs) scores every query against…

    Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

    November 22, 2026

    Learning from Context: Leave-One-Out Prediction in Jump Diffusions

    November 22, 2026

    Represented but misread: Post-training changes how language models read out what they represent

    November 22, 2026
    About Us

    Welcome to TechWave Dispatch

    At TechWave Dispatch, we bring you the latest news, insights, and developments from the ever-evolving world of technology. Our mission is to keep readers informed about the innovations shaping our digital future.

    From artificial intelligence and machine learning to cybersecurity, robotics, smart devices, and groundbreaking technologies, we cover the topics that matter in today's fast-moving tech landscape.

    Our Picks

    HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

    November 22, 2026

    Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

    November 22, 2026

    Learning from Context: Leave-One-Out Prediction in Jump Diffusions

    November 22, 2026
    Most Popular

    After blocking Muse, Amazon now lets users text Alexa to help them shop

    November 6, 20260 Views

    A controlled world for robot learning and evaluation

    November 6, 20260 Views

    Egyptian and Nigerian English join the Open ASR Leaderboard

    November 6, 20260 Views
    Facebook X (Twitter) Instagram Pinterest
    • Terms and Conditions
    • Privacy Policy
    • Contact Us
    • About Us
    © 2026 TechWave Dispatch. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.