LLM Inference Lab
A traceable learning record for LLM inference, KV cache, and evaluation methods.
Problem and goal
An API caller can observe a long wait for the first token or pauses during generation without knowing where the time went. This project follows those symptoms into the serving system: how requests are scheduled, how much memory KV cache needs, and which part of latency changes when cached state is reused or moved.
The current work focuses on understanding the mechanisms and measurement methods before running experiments. Performance improvements still need to be established through those experiments.
My contribution
I start with a question, read about the relevant inference stage, cache mechanism, or metric, and use AI assistance to fill background gaps and develop notes. Each record should explain how far the question has been answered and what still needs an experiment.
Existing material
The project reading list connects the Chinese inference overview with notes on Prefill/Decode, KV cache memory, prefix reuse, scheduling, and benchmarking. S00 AI-assisted study was completed on 11 October 2026; independent experiments remain planned. These articles retain derivations and open questions without claiming measured inference speedups.
Planned experiments
| Direction | Metrics | Status |
|---|---|---|
| Prefill and Decode latency | TTFT / TPOT | Planned |
| KV cache memory estimates | GPU memory | Planned |
| Prefix cache reuse | TTFT / hit proportion | Planned |
Method and limitations
For example, comparing TTFT with prefix caching on and off requires a fixed model, hardware, and software version, plus records of concurrency, input lengths, and prefix distributions. A repeatable baseline makes the comparison interpretable. The listed tools and experiments remain study plans until they have been run and evaluated.