Senior Software Engineer, Inference Platform

Aion is the enterprise AI platform, a full-stack solution for building, fine-tuning, and deploying AI at scale. Whether an organization is modernizing internal operations, launching AI-powered products, or transforming customer experiences, Aion takes them from concept to production on a single, unified platform.

We work differently than most AI companies: our teams deploy alongside our customers, turning production-ready AI into real business outcomes in weeks, not quarters.

We’re a fast-growing, VC-backed startup led by founders with a track record of successful exits. With teams across the US, UK, and India, we’re building the next generation of enterprise AI and we’re looking for exceptional people to help us scale.

You're a seasoned engineer who has built and scaled high-performance inference systems for AI/ML workloads. . You understand the complexities of serving models at scale latency optimization, resource orchestration, autoscaling dynamics, and production reliability. You've designed distributed systems that handle thousands of requests per second while maintaining sub-second response times and cost efficiency.

Experience with Golang is strongly preferred, and exposure to inference engines (vLLM, TGI, TensorRT), containerization, and distributed systems is an added bonus. You take ownership of platform-level decisions, think strategically about performance vs. cost trade-offs, and want your work to power AI inference for thousands of developers globally.

You're product-minded, you understand how your technical decisions impact developers using aion's platform and think about the end-to-end user experience. You're a team player comfortable wearing multiple hats one day you're optimizing inference latency, the next you're joining customer calls to understand their deployment challenges, and the day after you're helping with UI/UX, customer success, documentation and product ops.

Design and build aion's inference service platform the backbone for serving AI models at scale across diverse workloads

Own and architect core platform components: AI Gateway, Resource Orchestrator, Runtime Engines, and Autoscaler

Design highly modular, scalable, and extensible low-level designs (LLDs) for inference infrastructure components

Lead high-level design discussions, establish architectural patterns, and drive technical decision-making for the inference stack

Understand and optimize the dynamics of model deployment, version upgrades, and rollback strategies

Build robust deployment pipelines for seamless model updates with zero-downtime deployments

Design intelligent routing systems for multi-model serving, A/B testing, and canary deployments

Implement strategies for efficient GPU utilization and model cold-start optimization

Implement highly performant and optimized software for low-latency, high-throughput inference serving

Build and debug production-grade code in distributed systems handling real-time AI workloads

Optimize inference pipelines for latency, throughput, batching efficiency, and resource utilization

Design fault-tolerant systems with graceful degradation and automatic recovery mechanisms

Build high-performance telemetry and observability stack for inference metrics, performance tracking, and debugging

Implement comprehensive monitoring for model latency, throughput, error rates, GPU utilization, and cost per inference

Conduct thorough code reviews to maintain code quality, performance standards, and architectural consistency

Establish engineering best practices for testing, documentation, and production readiness.

HPC & Cluster Management: Experience handling large-scale HPC clusters using Kubernetes and Slurm for job scheduling, resource allocation, and workload orchestration

Data Engineering: Expertise with data pipelines, ETL systems, and large-scale data processing frameworks

Systems-Level Programming: Experience with low-level systems programming such as storage systems, Kubernetes operators, OS-level software development, or daemon services (llm-d, system agents)

ML Platform Engineering: Experience productionizing ML pipelines, batch job orchestration, model fine-tuning workflows, and Jupyter notebook orchestration systems

Enterprise Deployment: Experience platformizing and packaging software for on-premises deployments or customer VPC installatiaons with emphasis on security, compliance, and operational simplicity

High ownership, self driven and biased for action.

Strong strategic thinking and ability to connect technical decisions to business impact.

Excellent communication and mentoring skills.

Thrives in ambiguity, fast-paced environments, and early-stage startup culture.

Work directly with high-pedigree founders shaping technical and product strategy.

Build infrastructure powering the future of AI computers globally.

Significant ownership and impact with equity reflective of your contributions.

Competitive compensation, flexible work options, and wellness benefits