Senior Databricks Migration Engineer at MDW Associates — Arlington, USA | Base Career

MDW Associates is seeking a Senior Databricks Migration Engineer to support the US Postal Services' Inspector General's Office (OIG).

This role leads technical data migration efforts from legacy SQL systems, warehouses and servers to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.

This also includes building large data structures and dashboards and reporting on data migration projects.

This position is on-site 4 days a week at our USPS customer site in Arlington, VA.

The Senior Databricks Migration Engineer may perform a combination of some or all of the following duties:

Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
Establish best practices for partition strategies and file size management within Delta Lake.
Design and implement a robust data security model using Unity Catalog for centralized governance.
Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.

5+ years' experience in data engineering, data system development or related roles.
5+ years' experience with cloud platforms (e.g. Azure, AWS, GCP).
1+ year leading complex, cross-functional data projects and technical teams.
Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.

Mastery of data engineering principles, including data modeling, ETL (Extract, Transform, Load) processes, and data pipelines.
Proficiency with Azure Data Lake data storage and processing services.
Skilled at designing, building, and optimizing data pipelines for ingesting, transforming, and loading data.
Proficiency in languages such as SQL and Python/PySpark for data manipulation and pipeline development.
Skilled at identifying and resolving data-related challenges.
Skilled at creating efficient data models that meet business requirements.
Skilled at optimizing query performance and system scalability.

Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field.

Public Trust - Candidates for this position must be submitted for a background investigation in order to satisfy government requirements.

Incorporated in 2012, MDW is a small business headquartered in McLean, VA with offices in Huntsville, AL.

We provide Federal government clients with consulting, management services, advisory and assistance services, decision support, strategic planning, process planning and improvement, task execution and metrics analysis.

Our mission is to make our clients more successful by providing honest assessments, exceptional analysis, prudent advice, direct communication, and hard work.

Our core capabilities include:

Strategic Planning
Cost Estimating
Budget Programming and Justification
Budget Execution
Acquisition Management

MDW Associates, LLC ("MDW") is an equal opportunity employer and does not discriminate in hiring or employment on the basis of race, religion, color, sex, national origin, age, marital status, covered veteran status, mental or physical disability, pregnancy or any other unlawful basis unrelated to the ability to do the job. MDW seeks to employ talented, ethical, and entrepreneurial-minded persons. Our applicants and employees are protected from discrimination as required by Federal law.

We offer a competitive and comprehensive benefits and compensation package for our employees, including but not limited to:

Health
Dental
Vision
Group Life Insurance/Long Term Disability/Short Term Disability
Retirement/401(k)

All your information will be kept confidential according to EEO guidelines.

Senior Databricks Migration Engineer

Job Fit Check

Full Job Posting

Apply for this job in 1 click

More jobs at MDW Associates

Mid-level Financial Analyst

Senior Strategic Planning and PPBE Policy Analyst

Full Stack Developer

Mid-Level Congressional Budget Analyst

Budget Analyst III

Mid-level Financial Analyst

Senior Strategic Planning and PPBE Policy Analyst

Full Stack Developer

Mid-Level Congressional Budget Analyst

Budget Analyst III

Required Experience And Skills

Desired Experience And Skills

Requirement(S)

Security Requirement(S)

Company Description

• Financial Systems Support & Integration

Additional Information

• Flexible Leave Policy

About MDW Associates

Autofill Plugin