Late Interaction Hopfield-Fenchel-Young Network for Sparse Retrieval of Language Model Hidden States
This work investigates the integration of Late Interaction and Hopfield-Fenchel-Young (HFY) Networks for associative retrieval and direct incorporation of external memories into large language models (LLMs). Unlike conventional retrieval approaches that compress sequences into single embeddings or provide retrieved documents as textual context, the proposed architecture stores complete sequences of contextualized token-level hidden states extracted from intermediate Transformer layers and allows retrieval. The proposed HFY network exposes how MaxSim can be incorporated into a HFY energy for matrix-valued memories and how hidden states from decoder models can be used as memories to provide neuronal information retrieval. Specifically, the experiments were done with hidden states extracted from Qwen3.5-4B and investigate whether retrieved associate memories can be ingested directly into the model’s internal representations to support context-grounded generation in one full language model inference. In particular, HFY-based retrieval improves the identification of relevant factual information compared with the unaugmented model and dense softmax-based memory integration in the evaluated example. ...