Senior Data Engineer

Requirements

• Eight or more years in data engineering overall. That includes hands-on work building data for AI or ML systems such as retrieval, embeddings, or feature data, which can be a more recent part of your background.
• Strong SQL and strong Python, including PySpark or similar distributed processing.
• Production experience across the AWS data stack: S3 for the lake, Glue for ETL and the Data Catalog, Athena for serverless query, and Redshift as the warehouse.
• Hands-on experience with a layered data architecture, whether you call it medallion (bronze, silver, gold), a data lake feeding a warehouse, or a lakehouse, including building the transformation stages that move data from raw to curated.
• Experience with an ELT or integration tool such as Airbyte, Fivetran, or Meltano, including building or maintaining connectors.
• Experience with event-driven pipelines using SQS and SNS, and with at least one streaming or change-data-capture technology such as Kinesis, Amazon MSK, or Debezium.
• Hands-on experience with a semantic or metrics layer over the warehouse, such as Cube or the dbt Semantic Layer.
• Hands-on experience with at least one vector store and embedding workflow: pgvector, Amazon OpenSearch, Pinecone, Weaviate, or Milvus.
• Comfort with columnar and open table formats: Parquet together with Apache Iceberg, Delta Lake, or Hudi.
• Working knowledge of an orchestrator such as Amazon MWAA, Step Functions, Dagster, or Prefect, and enough infrastructure as code to work closely with DevOps.