When RETRO was released, i thought retrieval is a big deal for LLMs, to reduce the param number while increasing the knowledge depth for models. It was a surprise to me that companies avoided that idea. Whale brings Retrieval to the table: > Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains (HumanEval +3.0; MATH +2.4). Mechanistic analyses reveal that Engram relieves the backbone’s early layers from static reconstruction, effectively deepening the network for complex reasoning.