Senior Software Engineer, Google Distributed Cloud AI

Responsibilities

• Lead the technical design, development, and optimization of software components critical for Large Language Model (LLM) inference serving on GDC. This includes areas such as model life-cycle management, efficient data loading, dynamic request routing, and intelligent load balancing.
• Drive horizontal integration across core platform services, including billing, logging, observability, security, and quota management.
• Implement and enhance serving capabilities to support advanced LLM techniques like disaggregated serving, speculative decoding, quantization, and efficient model sharding across distributed hardware.
• Collaborate closely with internal teams developing core LLM frameworks, container orchestration (Kubernetes, Google Kubernetes Engine (GKE)), networking infrastructure, and hardware acceleration to build a cohesive and high-performance serving platform.
• - Lead the technical design, development, and optimization of software components critical for Large Language Model (LLM) inference serving on GDC. This includes areas such as model life-cycle management, efficient data loading, dynamic request routing, and intelligent load balancing. - Drive horizontal integration across core platform services, including billing, logging, observability, security, and quota management. - Implement and enhance serving capabilities to support advanced LLM techniques like disaggregated serving, speculative decoding, quantization, and efficient model sharding across distributed hardware. - Collaborate closely with internal teams developing core LLM frameworks, container orchestration (Kubernetes, Google Kubernetes Engine (GKE)), networking infrastructure, and hardware acceleration to build a cohesive and high-performance serving platform.

Minimum qualifications: