06:00
From GPT2 to Kimi3: A 22,580x Scale-Up with Architectural Evolution
从GPT-2到KimiK3：七年规模增长22,580倍的架构演进全解析
This worklog traces the architectural evolution from GPT-2 (2019) to KimiK3 (2026) with code snippets and diagrams. The central claim: progress is not just scale but innovations in state management and retrieval. It covers KV cache bottlenecks, linear attention's fixed-state trade-off, DeltaNet's precise overwriting via delta rule, Gated DeltaNet adding forgetting, KDA/Kimi Linear with per-channel gating, and finally KimiK3's hybrid of KDA and MLA layers, MoE, SiTU activation, and blockwise AttentionRes (AttnRes) for selective depth-wise residual access. Suitable for engineers interested in LLM internals.