Findings

DeepSeek's Engram introduces conditional memory as a new dimesion of sparsity for LLMs, complementing Mixture-of-Experts by replacing expensive neural computation for static knowledge retrieval with O(1) hash-based lookups.

• Language models without Engram waste compute just for retrieval of static multi-token phrases through layers of attention – work that is structurally just a dictionary lookup. Engram addresses this with a modernised N-gram embedding table, gated by context to remain semantically relevant.
• DeepSeek discovers U-shaped sparsity allocation law, identifying ~20-25% of sparse capacity allocated to Engram as the optimal memory budget, with ~80-85% allocated to MoE.

DeepSeek’s mHC is an architectural upgrade over the original Hyper-Connections (HC) paper, which replaces single residual paths with multiple parallel streams to make them stable enough for scaling. mHC addresses hyper-connections' sensitivity to signal explosions during training by introducing mathematical guardrails, making it viable for large-scale use.

• mHC forces the mixing of parallel streams to follow a "doubly stochastic" pattern. This prevents the signal from exploding or collapsing.
• It uses the Sinkhorn-Knopp algorithm and custom GPU kernels to restrict the hyper-connection signals without slowing down the model’s training speed.

The seminal paper introducing the Transformer architecture.