Running LLMs on a company’s own GPUs
They can’t send prompts to a hosted API, so everything runs in their cluster. I set up vLLM on Kubernetes, sorted out the GPU scheduling, and got model rollouts working without dropping requests that were already in flight. Prometheus, Grafana and Langfuse sit on top so that when latency moves we can say why.
- vLLM
- Kubernetes
- Prometheus
- Grafana
- Langfuse