Inference Infrastructure Architect (Remote)

What we look for

• You have owned production LLM serving under meaningful traffic and latency constraints, and can explain the architecture, the bottlenecks you diagnosed, the changes you made, and the resulting improvements in latency, reliability or cost. Scale on the order of thousands of GPUs or millions of requests a day is a strong signal, not a hard requirement.
• Kubernetes on GPU fleets, operated end to end: GPU Operator, device plugins, node pools and topology-aware placement, gang scheduling (LeaderWorkerSet, Kueue), GitOps rollouts, and the ability to debug why a pod landed where it did.
• Deep operational command of vLLM or SGLang: deploying, tuning and upgrading it (parallelism with TP / EP, quantization, batch and KV-cache settings, prefix caching, disaggregation), and knowing which knob moves which metric.
• Performance engineering at the system level: you read engine and DCGM metrics, reason from roofline, and size deployments with numbers.
• Python and Go for automation; real Linux, networking and storage depth. You don't need to write CUDA; you need to know when a problem is one, and route it upstream.
• You work with the community in English and Chinese, and write runbooks and design docs people actually read.
• Experience we especially value
• Operated inference platforms at the scale of Ant Group, Alibaba Cloud / PAI, Tencent, ByteDance, Baidu, Huawei, DaoCloud, Moonshot, DeepSeek, or comparable teams.
• Contributor to, or heavy production user of, any of: vLLM, SGLang, llm-d, NVIDIA Dynamo, Ray / KubeRay, Kata Containers, Volcano, Kueue, HAMi, Dragonfly, Mooncake, Envoy.
• Presence in the CNCF / OpenInfra community: talks at KubeCon China, OpenInfra Days, vLLM or SGLang meetups.
• Nice to have
• Fine-tuning / RL infrastructure: LoRA pipelines, evaluation harnesses, champion/challenger rollout.
• Real-time voice latency work: sub-second time-to-first-token budgets on live calls.
• Multi-region deployments and data-residency requirements.
• What we offer
• The newest hardware, ours from the metal up. A globally expanding B300 fleet that Telnyx builds and operates end to end. Bring your experience optimizing demanding production workloads to a fleet where you shape the architecture from bare metal to customer endpoint.