Data Engineer - Generative AI & Vector Systems

Design and manage Vector Database architectures.

Optimize indexing, storage, metadata management, and retrieval performance.

Improve similarity search performance and retrieval latency.

Work with embedding models and vector search optimization techniques.

Develop production-grade ETL/ELT pipelines.

Build batch and near real-time ingestion pipelines.

Automate workflow orchestration using Apache Airflow.

Monitor pipeline performance and ensure high availability.

Develop cloud-native solutions on AWS.

Work with services such as:

Amazon S3

AWS Glue

EMR

Lambda

Athena

IAM

CloudWatch

Optimize compute resources for large-scale AI workloads.

Write complex SQL queries across distributed data sources.

Use Starburst/Trino to federate and query multiple data platforms.

Design efficient data models for AI workloads.

Perform large-scale joins and transformations.

Work closely with AI engineers to support LLM-based applications.

Build Retrieval-Augmented Generation (RAG) pipelines.

Manage embedding generation and vector indexing.

Support prompt engineering and retrieval optimization initiatives.

Improve data pipeline performance and scalability.

Implement monitoring, logging, and alerting.

Troubleshoot production data issues.

Ensure data integrity and governance.

Python (Advanced)

SQL (Advanced)

ETL / ELT

Data Transformation

Data Cleansing

Data Chunking Strategies

Metadata Management

Data Modeling

Hands-on experience with one or more:

Pinecone

Milvus

Qdrant

Chroma

Weaviate (Good to have)

FAISS (Good to have)

Apache Airflow

Strong experience with AWS services:

Amazon EMR

Experience with:

Starburst

Trino

Presto (Good to have)

Working knowledge of:

Large Language Models (LLMs)

Text Embeddings

Semantic Search

Vector Search

Retrieval-Augmented Generation (RAG)

Prompt Engineering (basic understanding)

REST APIs

JSON

Data Connectors

4–8 years of experience in Data Engineering.

Experience building scalable cloud-native data platforms.

Hands-on experience with Vector Databases.

Experience working with enterprise-scale SQL environments.

Strong background in Python-based data engineering.

Experience building production-grade Airflow pipelines.

Familiarity with Generative AI architectures.

Experience supporting Machine Learning pipelines.

Flexible working hours (self-managed)

Annual bonus, subject to company performance

Access to Udemy online training and opportunities to learn and grow within the role

At Mindera we use technology to build products we are proud of, with people we love.

Software Engineering Applications, including Web and Mobile, are at the core of what we do at Mindera.

We partner with our clients, to understand their products and deliver high-performance, resilient and scalable software systems that create an impact on their users and businesses across the world.

You get to work with a bunch of great people, and the whole team owns the project together.

Our culture reflects our lean and self-organisation attitude.

We encourage our colleagues to take risks, make decisions, work in a collaborative way and talk to everyone to enhance communication. We are proud of our work and we love to learn all and everything while navigating through an Agile, Lean and collaborative environment.

Our offices are located: Porto, Portugal | Aveiro, Portugal | Coimbra, Portugal | Leicester, UK | San Diego, USA | Chennai, India | Bengaluru, India