Software Engineer II - Backend & Data Platform (Python, Kafka, PySpark)
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
Role Overview
Build scalable, reliable, production-grade backend services for the core data platform.
Work across Python backend engineering, PySpark distributed processing, Kafka and Redis event-driven systems, ML inference, and AWS security.
Collaborate with Data Science, Data Engineering, DevOps, SecOps, and Product teams.
Key Skills for This Role
Full Job Posting
About HiLabs
HiLabs builds AI-driven data solutions for US healthcare payers across provider, claims, and clinical datasets.
Engineering teams in Pune build the platform that supports production-scale data quality solutions.
About the Role
Build scalable, reliable, production-grade backend services for the core data platform.
Work across Python backend engineering, PySpark distributed processing, Kafka and Redis event-driven systems, ML inference, and AWS security.
Collaborate with Data Science, Data Engineering, DevOps, SecOps, and Product teams.
What You Will Do
- Design, build, and maintain Python backend services and FastAPI or Flask microservices.
- Build PySpark pipelines for ingestion, classification, linkage, feature engineering, and schema generation.
- Develop asynchronous Kafka and Redis Streams services, including producers, consumers, topics, partitions, offsets, and retries.
- Use Redis for caching, fast lookups, TTL-based state, and distributed coordination.
- Deploy containerized services on AWS EKS and Kubernetes.
- Integrate AWS S3, EMR, EventBridge, PostgreSQL, Snowflake, Kafka, and Redis.
- Deploy batch scoring and ML inference services using versioned model artifacts.
- Implement reliability patterns including idempotency, retries, timeouts, dead-letter handling, and failure recovery.
- Design database schemas, indexes, queries, and data-access layers.
- Implement observability, write unit and integration tests, participate in code reviews, and troubleshoot across technical layers.
Must Have
- 3–5 years of hands-on backend software engineering experience.
- B.E., B.Tech, M.Tech, or MCA in Computer Science or a related field from a Tier-1 institute or equivalent; this is mandatory.
- Strong Python skills covering object-oriented programming, data structures, and design principles.
- Experience with PySpark or Apache Spark, distributed data processing, Apache Kafka, Redis, FastAPI or Flask, REST APIs, and microservices.
- Experience with AWS S3, IAM, CloudWatch, EKS or EMR, Docker, Kubernetes, PostgreSQL or SQL, Git, CI/CD, and testing.
Good to Have
- Experience with Amazon MSK, Kafka on Kubernetes, Redis Cluster, ElastiCache, AWS EMR, Spark at scale, Snowflake, Parquet, or schema management.
- Experience with ML inference, model serving, embeddings, risk scoring, MLflow, GPU workloads, healthcare data, claims or clinical data, PHI, or HIPAA awareness.
Location
- The position is based onsite in Pune, Kharadi.
- Candidates open to relocating to Pune may apply.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career