Data Engineer / Databricks Engineer-Senior at Nationwide IT Services — Maryland, USA | Base Career

Full Job Posting

Design, build, test, deploy, and maintain automated and manual ETL/ELT pipelines supporting approximately 15–20 DAF, ANG, and DoD source systems.
Implement ingestion patterns for structured, unstructured, API, batch, and streaming data; support near-real-time data currency requirements where applicable.
Develop SQL and Python transformation logic to cleanse, standardize, deduplicate, validate, and enrich enterprise data in accordance with CDAO data standards and business rules.
Build and operate Azure Databricks jobs, orchestration sequences, triggers, and workload schedules while optimizing execution performance and cloud-compute utilization.
Implement automated data-quality checks, reconciliation controls, invalid-record quarantine, monitoring, logging, alerting, and recoverable error-handling patterns.
Triage critical pipeline failures, perform root-cause analysis, and restore service within required response windows.
Maintain pipeline code, configuration, infrastructure/deployment scripts, and test assets in the official Git repository and execute automated CI/CD build and security testing before release.
Create and maintain data lineage, source-to-target mappings, architecture documentation, operational runbooks, and quarterly technical-documentation updates.
Support data warehouse/lake administration, query optimization, Power BI dataflows, and secure cross-platform data movement as needed.

Demonstrated hands-on experience developing production-grade ETL/ELT pipelines using Azure Databricks or comparable Spark-based data platforms.
Advanced SQL and Python skills for data transformation, validation, automation, troubleshooting, and performance optimization.
Experience with batch and streaming ingestion, APIs, schema management, orchestration, monitoring, data quality, and failure recovery.
Experience with Git-based version control and CI/CD practices for data-engineering solutions.
Knowledge of data warehousing/lakehouse patterns, indexing/query optimization, access control, encryption, and secure handling of sensitive/CUI data.

Data Engineer / Databricks Engineer-Senior