1EIT-NLP Lab, Eastern Institute of Technology, Ningbo
2The Hong Kong Polytechnic University 3Tencent Youtu Lab
ATTUNER adapts online queries while preserving independently constructed artifact KV, learning to read the cache without rewriting it.
Queries are computed for online tokens, while cached keys and values remain available for direct reuse. ATTUNER augments the frozen query projection in every Transformer layer with a low-rank update. Stored artifact KV remains unchanged; online states and newly produced KV evolve under the adapted computation.
We distill Full-context answering behavior into the PIC student. Both branches receive the same input and condition on the same response prefix, but the teacher uses full prefill while the student reuses independently cached artifact KV. Only query adapters are updated; the backbone and cached artifact KV remain frozen.
A single adapter improves the use of skills, documents, and memories while preserving their original cached KV. Average in-domain scores improve over PIC by 5.13 and 7.77 points on Qwen3-4B and Qwen3-8B, reducing the gaps to Full to 1.61 and 0.51 points. The benefits also extend to TheoremQA and CHAMP, neither of which is included in adapter training.
Scores in %. TTFT is median warm-cache latency in ms, measured at batch size 1 without concurrent GPU workloads. Bold and underlined values retain the paper’s rankings. Select a table to view it at full size. Read the paper ↗
With query updates merged into the projection weights, ATTUNER’s geometric-mean in-domain TTFT is only 1.8% and 0.5% higher than PIC on Qwen3-4B and Qwen3-8B, preserving 1.79× and 2.45× speedups over Full. ATTUNER requires neither online token selection nor artifact-token recomputation.
These results highlight query-side adaptation as an effective path toward high-quality, efficient cache reuse: learning to read the cache rather than rewriting it.
Understanding quality loss, recovery, and learned cache reading.
PIC is more sensitive to distractors than Full. Removing irrelevant candidates still leaves substantial gaps on multi-artifact tasks, and position alignment does not consistently close them. These observations motivate improving how the model reads independently cached context.
The routing oracle replaces online attention weights with a Full reference while retaining PIC’s artifact KV. In this MuSiQue example, output cosine similarity to Full increases from 0.898 to 0.984. Paired task evaluations also recover approximately Full performance. This diagnostic uses Full-reference attention; ATTUNER learns cache reading without requiring that reference at deployment.
On 128 examples per dataset, learned queries bring attention outputs closer to Full across many intermediate layers, at both prompt and answer positions. Mean answer-position cosine similarity rises from 0.903 to 0.927 on MuSiQue and from 0.932 to 0.949 on BigCodeBench, despite training only on output distributions. Recovery is not uniform across layers.
Limited training data already enables substantial recovery. Additional examples bring performance closer to the paired Full references.
Two 18-layer windows both recover quality on 128 MuSiQue test examples; their difference is inconclusive. The same full-layer adapter improves PIC on every task below, while its effect under Full is mixed.
If Attuner is useful for your research, please cite our paper.
Paper
Hugging Face
Code
Demo@misc{chen2026attunerrecomputationfreekvcache,
title={ATTUNER: Recomputation-Free Agentic Context KV Reuse},
author={Xinghao Chen and Junnan Dong and Cai Ke
and Chak Tou Leong and Haocheng Sun and Keyu Chen
and Siyu An and Ruizhi Qiao and Xing Sun
and Wenjie Li and Xiaoyu Shen},
year={2026},
eprint={2609.36722},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2609.36722}
}