zoho_recruit
Sr Data Engineer- Databricks
Aptus Data Labs
Remote, IND
Full-time
Senior · 5+ years experience
Remote
Discovered 3 weeks ago
DatabricksDelta LakeApache SparkPySparkUnity CatalogSQL
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
DatabricksDelta LakeApache Spark
Key Skills for This Role
DatabricksDelta LakeApache SparkPySparkUnity CatalogSQL
Full Job Posting
Location
- Remote (Preferred candidates to be in bangalore)
- Design, implement, and optimize scalable data pipelines using Databricks and Apache Spark.
- Architect data lakes using Delta Lake, ensuring reliable and efficient data storage.
- Manage metadata, security, and lineage through Unity Catalog for governance and compliance.
- Ingest and process streaming data using Apache Kafka and real-time frameworks.
- Collaborate with ML engineers and data scientists on LLM-based AI/GenAI project pipelines.
- Apply CI/CD and DevOps practices to automate data workflows and deployments (e.g., with GitHub Actions, Jenkins, Terraform).
- Optimize query performance and data transformations using advanced SQL.
- Implement and uphold data governance, quality, and access control policies.
- Support production data pipelines and respond to issues and performance bottlenecks.
- Contribute to architectural decisions around data strategy and platform scalability.
Requirements
- 5+ years of experience in data engineering roles.
- Proven expertise in Databricks, Delta Lake, and Apache Spark (PySpark preferred).
- Deep understanding of Unity Catalog for fine-grained data governance and lineage tracking.
- Proficiency in SQL for large-scale data manipulation and analysis .
- Hands-on experience with Kafka for real-time data streaming.
- Solid understanding of CI/CD, infrastructure automation, and DevOps principles .
- Experience contributing to or supporting Generative AI / LLM projects with structured or unstructured data.
- Familiarity with cloud platforms (AWS, Azure, or GCP) and d ata services.
- Strong problem-solving, debugging, and system design skills.
- Excellent communication and collaboration abilities in cross-functional teams.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Aptus Data Labs
Sr Data Engineer-AWS
, IND
Discovered 3 weeks agoFull-time
Sr /lead Data Scientist
, IND
Discovered 3 weeks agoFull-time
AI Lead
Bengaluru, IND
Discovered 3 weeks agoFull-time
Join our Team - Current Openings
Bengaluru, IND
Discovered 3 weeks agoFull-time
Enterprise Data Architect
Bengaluru, IND
Discovered 3 weeks agoFull-time
Boomi - Application Integration Specialist
, IND
Discovered 3 weeks agoFull-time
AI Software Engineer
Bengaluru, IND
Discovered 3 weeks agoFull-time
Data Architect
Bengaluru, IND
Discovered 3 weeks agoFull-time