zoho_recruit
Data Engineer — ML Training Data Pipeline
DATAECONOMY
Hyderabad, IND
Full-time
Senior · 6+ years experience
Hybrid
Discovered 1 weeks ago
PythonpandaspyarrowHuggingFace DatasetsAWSS3
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
Pythonpandaspyarrow
Key Skills for This Role
PythonpandaspyarrowHuggingFace DatasetsAWSS3
Full Job Posting
What We Expect
Build end-to-end data pipelines: raw trace ingestion → dedup → format conversion → quality gating → training-ready datasets
Process large-scale JSONL data on AWS S3 (tens of thousands of traces per batch)
Convert between chat-completion formats (e.g., OpenAI → Llama 3.1 tool-calling format)
Implement smart deduplication and sampling to balance training distribution
Design identity-aware train/test splits that measure true generalization
Build data validation gates to detect schema drift and format anomalies
Create a continuous pipeline that auto-processes new production traces for retraining
Requirements
- Experience: 6+ years data engineering focused on ML data pipelines
- Python: Strong — pandas, pyarrow, JSONL processing at scale
- ML Data Libraries: HuggingFace Datasets, Arrow-based storage
- Data Formats: Multi-turn conversation/chat data structures and tokenizer-specific formatting
- Deduplication: Content hashing, identity-based grouping strategies
- AWS: S3, EC2, batch processing workflows
- Preferred (Not Required): LLM training data prep (chat templates, tool-calling schemas); Axolotl or similar dataset formats; data versioning (DVC, LakeFS); browser-automation trace data or Playwright.
Benefits
- Comprehensive Medical Coverage: Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
- Robust Protection Plans: Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
- Retirement Benefits: PF and Gratuity provided as per standard government regulations.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at DATAECONOMY
AI/ML MLOps Engineer
Hyderabad, IND
MidFull-time
DATAECONOMY is seeking an AI/ML MLOps Engineer to fine-tune, deploy, evaluate, and monitor self-hosted large language models across the full machine learning lifecycle. The role requires 5–8 years of experience, strong P
Discovered 1 weeks agoView →
Java Full Stack Developer
Hyderabad, IND
SeniorFull-time
DATAECONOMY is seeking an experienced Java Full Stack Developer to build and maintain enterprise applications across backend, frontend, cloud, and deployment layers. The role requires strong Java, Spring Boot, Spring Sec
Discovered 1 weeks agoView →
ML / Fine-Tuning Engineer
Hyderabad, IND
Discovered 1 weeks agoFull-time
AI/ML MLOps Engineer
Hyderabad, IND
Discovered 1 weeks agoFull-time
Java Full Stack Developer
Hyderabad, IND
Discovered 1 weeks agoFull-time
Java Full Stack Developer
Hyderabad, IND
Discovered 2 weeks agoFull-time
Databricks Engineer
Hyderabad, IND
Discovered 2 weeks agoFull-time
AI/ML MLOps Engineer
Hyderabad, IND
Discovered 2 weeks agoFull-time