Data Engineering Manager

This is a deep stack, so we do not expect someone to master everything immediately. A rough first-year path would look like this:

In the first 90 days, you understand the main parts of the data stack, the people who depend on it, and the biggest pain points. You have shipped improvements to at least one real pipeline, not just reviewed documents or attended meetings.

By six months, you are helping steer the roadmap for market data infrastructure. You have improved reliability, performance, observability, or recoverability in a way we can measure. The team is relying on you in code reviews, design reviews, and production decisions.

By the end of the first year, you own the platform end to end, from ingestion through delivery. People across trading, research, and technology know to come to you for market data platform questions. You also have a clear view of how the platform needs to evolve as AI becomes a larger data consumer, and you have started moving it in that direction.

Own the market data pipeline from ingestion through normalization and near-real-time delivery. The data has to be correct first, and the system has to recover cleanly when something breaks.

Responsibilities include:

Integrating direct exchange feed capture alongside third-party vendor data

Building and improving replay, recovery, and gap-detection capabilities

Keeping market data correctly sequenced, validated, and available fast enough for downstream users

Understanding when latency matters, when durability matters more, and how to make the right tradeoff

Design, maintain, and improve the KDB+/Q platforms that hold our real-time and historical market data.

Schema design, partitioning, and query-performance tuning

Supporting real-time and historical analytics use cases

Managing retention and data lifecycle policies

Keeping the platform maintainable as data volumes and usage grow

Debugging production HDB and tickerplant issues directly

Deliver data reliably to downstream consumers through streaming, messaging, and platform integrations.

Defining data contracts and schemas that other teams can depend on

Supporting replayable and durable data flows where needed

Working with downstream teams to understand how they actually consume the data

Balancing real-time delivery needs with reliability and operational simplicity

Build the internal tooling and shared libraries that make the data platform easier to operate and easier to use.

Building validation, monitoring, replay, and analytics tools

Owning supporting systems for reference data, configuration, and metadata

Improving developer workflows around market data testing and troubleshooting

Reducing repeated manual work through better tools and automation

Lead the team by staying close to the work.

Writing production code

Reviewing pull requests and technical designs

Working directly with trading and research teams to understand their needs

Debugging production issues during market hours when needed

Setting expectations for quality, reliability, and maintainability

Improving monitoring, alerting, and data-quality checks so problems are caught before the desk finds them

KDB+ / Q

Python

C / C++

Linux

Docker

Git / CI-CD

Binary market data protocols

Streaming / message bus platforms

Kernel-bypass / high-performance networking

Industry-standard messaging protocols (FIX, SBE)

Reports to the CTO

Based in Chicago, IL; hybrid, with three days per week in the office

No direct on-call rotation; the platform has 24-hour coverage from dedicated staff and support teams