Large-Scale System Platform Lab

Efficient LLM Serving Systems: We design inference and serving systems that meet strict latency and cost targets at production scale. Continual and On-Device Learning: We build methods and system support for continual adaptation on resource-constrained devices and edge environments. Large-Scale Distributed Training: We develop techniques to train models efficiently across heterogeneous and geo-distributed clusters. Fast and Scalable Big Data Analytics: We build data systems that accelerate iterative analytics and scalable ML data pipelines. SLO-aware, adaptive KV cache management is a system-level optimization for Large Language Model (LLM) serving that dynamically balances memory usage and performance to meet specific Service Level Objectives (SLOs), such as Time-To-First-Token (TTFT) or Time-Between-Tokens (TBT). Generalized KV Cache Reuse is an advanced optimization for LLM serving that allows the system to reuse KV cache segments from any part of a prompt, moving beyond the limitations of traditional prefix caching. Accuracy-preserving long-context pruning is a technique used to compress the KV cache of Large Language Models (LLMs) by removing less important tokens without degrading the model's performance on long-context tasks. SLO-aware failure recovery for LLMs is a resilience strategy designed to recover from hardware or software failures while strictly minimizing the impact on Service Level Objectives (SLOs) like latency and throughput. Systems for Agentic AI are specialized software architectures designed to support AI agents that don't just "chat," but independently plan, use tools, and execute multi-step tasks to achieve a goal. Low-latency, high-accuracy on-device learning refers to the ability of a local device (like a smartphone, IoT sensor, or medical de