Duckdb Engineer Jobs in Chicago, IL (NOW HIRING)

As an engineer for the Data Ingest team, you will build and maintain the core, top-level data assets that power our research platforms. These are used by every desk at CTC, from junior traders to executives. Because of this broad user base, the product's surface area is extensive. By ensuring high quality data in a timely fashion, you help support data-driven decision-making on daily basis.

WHAT YOU'LL DO

• Own the quality, reliability, and overall health of our source data pipelines.
• Work closely with Traders and Quantitative Researchers to understand their needs, and how they use our final data products.
• Act as the data-delivery expert for your stakeholders, and design the most reliable and efficient way to source the data they need.
• Optimize existing pipelines to improve SLA for users and lower the compute and storage footprint across the firm.
• Migrate existing pipelines onto modern data platforms and hardware, and improve the process for future migrations.
• Collaborate with the broader data organization to improve our overall platform offering and functionality.
• Detect source data quality issues that lead to incorrect downstream metrics, and partner with data owners to build tools that surface and resolve these issues before users see them.

WHAT WE'RE LOOKING FOR

• Hands-on experience building and operating large-scale data pipelines, ideally in the hundreds of terabytes to petabytes range.
• Strong software architecture and large-scale system design skills.
• Strong proficiency in SQL and in object-oriented languages such as Python.
• Hands-on experience with at least one major cloud provider (AWS, Azure, GCP) and/or HPC workloads.
• Strong interpersonal skills and the ability to work in a team environment.

NICE TO HAVE

• Experience with tick capture, market data, or trading industry data (options, equities, futures).
• C++.
• Experience with data observability and data quality tooling, including diagnosing source data problems.
• Experience with open table formats such as Iceberg or Delta, and with query engines such as Trino, Spark, or DuckDB.