LLM Inference Lab
A traceable learning record for LLM inference, KV cache, and evaluation methods.
My work： Framing study questions, understanding inference mechanisms, and planning reproducible experiments.
Problem and goal
Understanding inference systems goes beyond calling an API. Cache capacity, reuse, scheduling, and memory transfers affect latency and throughput. Experimental conditions determine whether results can be compared.
This is primarily a learning project connecting concepts, methods, and future experiments. It is not a completed inference optimization case study.
My contribution
I organize study questions and work through inference stages, cache mechanisms, and measurement concepts, using AI assistance to fill background gaps and develop notes. The record should connect back to the original question and retain unresolved issues.
Existing material
The site contains three foundational notes on Prefill/Decode, KV cache memory, and Prefix Cache/TTFT. They explain mechanisms and planned experiments, without claiming measured performance gains.
Planned experiments
| Direction | Metrics | Status |
|---|---|---|
| Prefill and Decode latency | TTFT / TPOT | Planned |
| KV cache memory estimates | GPU memory | Planned |
| Prefix cache reuse | TTFT / hit proportion | Planned |
Method and limitations
Future experiments need to record models, hardware, software versions, concurrency, input lengths, and prefix distributions. A reproducible baseline comes before conclusions about cache benefits. Tools listed for future study are not claims of validated engineering proficiency.