DATA Architect - Lead Systems Engineer

Qualifications

• 10+ years of experience in data engineering, data architecture, or related roles, with a track record of designing and deploying data pipelines and platforms at scale.
• Significant hands-on experience with GCP Data Lake and the Apache Spark ecosystem. Proficient in building data pipelines using PySpark/Scala and managing data in Delta Lake format.
• Strong experience working with cloud data platforms (GCP preferred, or AWS/Azure). Familiarity with GCP Storage principles.
• Strong skills in vector databases and embedding models to support scalable RAG systems. Proficient in optimizing retrieval and indexing for LLM integration.
• Strong experience in managing structured, semi structured and unstructured data in GCP Data Lake.
• Ability to inspect existing data pipelines, discern their purpose and functionality, and re-implement them efficiently in GCP Data Lake.
• Advanced SQL skills with the ability to write and optimize complex queries. Solid understanding of data warehousing concepts and performance tuning for SQL engines.
• Proven ability to optimize ETL jobs for performance and cost efficiency. Experience tuning cluster configurations, parallelism, and caching to improve job runtimes and resource utilization.
• Demonstrated experience implementing data security and governance measures. Comfortable configuring Unity Catalog or similar data catalog tools to manage schemas, tables, and fine-grained access controls. Able to ensure compliance with data security standards and manage user/group access to data assets.
• Experience leading and mentoring engineering teams. Excellent project leadership abilities to coordinate multiple projects and priorities. Strong communication skills to effectively collaborate with cross-functional teams and present architectural plans or results to stakeholders.
• Experience working in an Agile environment. Tools & Technologies
• GCP Data Lake Lakehouse Platform: GCP Data Lake Workspace, Apache Spark, Delta Lake, GCP Data Lake SQL, MLflow (for model tracking), Postgres Database.
• Data Governance: GCP Data Lake Unity Catalog for data catalog and access control.
• Programming & Data Processing: PySpark and Python for building data pipelines and Spark Jobs; SQL for querying .
• Cloud Services: GCP Cloud Storage, GCP Pub/Sub technologies and Vector Databases.
• DevOps & CI/CD: Git for version control (GitLab), Jenkins and experience with Terraform for infrastructure-as-code is a plus. • Other Tools: Project and workflow management tools JIRA and confluence. Looker Studio and PowerBI Preferred Certificates, Licenses, and Registrations
• GCP Data Lake Certified Data Engineer Professional or GCP Data Lake Certified Data Engineer Associate.
• Exposure to related big data and streaming tools such as Apache Kafka, GCP Pub/Sub services, Apache Airflow and BI/analytics tools (e.g., Power BI, Looker Studio) is advantageous.