Tair Kvcache

Tair KVCache

---

|中文版本|

---

[Tair KVCache](https://www.aliyun.com/product/kvcache) is Alibaba Cloud's high-performance KVCache system designed for Large Language Model (LLM) inference scenarios. Through technologies such as distributed memory pooling and dynamic multi-level caching, it achieves acceleration and efficiency improvement while reducing resource costs. Currently, the global KVCache management system Tair KVCache Manager and the LLM inference simulation system Tair KVCache HiSim have been open-sourced.

Tair KVCache Manager

Tair KVCache Manager is one of the core components of Tair KVCache, designed to provide unified KVCache metadata management services for Large Language Model (LLM) inference scenarios.

System Architecture


• Tair KVCache Manager is deployed in a centralized mode, responsible for global metadata management of KVCache, providing services such as KVCache queries and storage capacity management.
• Tair KVCache Manager Client/Connector is responsible for interfacing with inference engines, implementing metadata queries and KVCache data transmission.


**Tair KVCache Manager** mainly consists of the following components:

• Access Layer (Server): Provides HTTP and gRPC services
• Cache Logic (CacheManager): Responsible for implementing external interfaces and core business logic
  • Provides multiple matching logics: prefix matching, sliding window matching, KV matching, etc.
  • Implements two-phase write mechanism: obtaining write addresses + notifying after write completion. Ensures data reliability.
  • Storage backend selection: Dynamically selec