The ChallengeAs AI and data products become business-critical, enterprises face a metadata crisis:
- No unified way to track the complex data supply chain feeding AI systems
- Engineering teams struggling with data discovery, lineage, and governance
- Organizations needing machine-scale metadata management, not just human-browsable catalogs
Why This MattersThis is where infrastructure meets impact. The metadata layer you'll build will directly power the next generation of AI systems at massive scale. Your code will determine how safely and effectively thousands of organizations deploy AI, affecting millions of users worldwide.
The RoleWe're looking for an exceptional Python engineer to lead development of DataHub's ingestion framework - the core that connects diverse data systems and powers our metadata collection capabilities.
You'll Build- Scalable, fault-tolerant ingestion systems for enterprise-scale metadata
- Clean, intuitive APIs for our connector ecosystem
- Event-driven architectures for real-time metadata processing
- Schema mapping between diverse systems and DataHub's unified model
- Versioning systems for AI assets (training data, model weights, embeddings)
You Have- 4+ years building production-grade distributed systems
- Advanced Python expertise with a focus on API design
- Experience with high-scale data processing or integration frameworks
- Strong systems knowledge and distributed architecture experience
- A track record of solving complex technical challenges
Bonus Points- Experience with DataHub or similar metadata/ETL frameworks (Airflow, Airbyte, dbt)
- Open-source contributions
- Early-stage startup experience
Location and CompensationBay Area (hybrid, 3 days in Palo Alto office)
Salary Range: $225,000 to $300,000
DataHub is an equal opportunity employer committed to workplace diversity and inclusion. We provide equal employment opportunities to all employees and applicants without regard to race, religious creed, color, national origin, ancestry, disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, military and veteran status, or any other characteristic protected by federal, state, or local law.