Databricks Data Engineer
Location: New york
We are seeking an experienced Databricks Data Engineer to design, develop, and optimize enterprise-scale data pipelines on the Databricks platform. The ideal candidate will have strong expertise in Databricks, Delta Lake, Python, SQL, ETL/ELT, cloud platforms, and Sales Incentive Compensation (SIC) data processing.
< data-start="434" data-end="463">
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, Delta Lake, and Delta Live Tables.
- Build and optimize data models, notebooks, workflows, and enterprise data pipelines.
- Develop data solutions supporting Sales Incentive Compensation (SIC) including quota management, commission calculations, attainment tracking, and payout reporting.
- Collaborate with Sales Operations, Finance, BI, and business stakeholders to translate business requirements into scalable data solutions.
- Integrate Databricks with cloud services (Azure, AWS, or Google Cloud Platform) and data sources such as Azure Data Factory, Kafka, Event Hubs, or similar.
< data-start="1376" data-end="1400">
Required Skills
- 7+ years of Data Engineering experience.
- 3+ years of hands-on Databricks development.
- Strong expertise in:
- Databricks
- Delta Lake
- Delta Live Tables (DLT)
- Unity Catalog
- Databricks Workflows
- Python and/or Scala
- Advanced SQL
- ETL/ELT development and data modeling
- Experience with Sales Incentive Compensation (SIC) processes including commissions, quotas, payouts, and attainment calculations.
- Experience integrating with Anaplan, Oracle ICM, Varicent (preferred).
- Experience with Azure, AWS, or Google Cloud Platform cloud platforms.
- Excellent analytical, communication, and stakeholder management skills.
- Databricks Data Engineer Certification is a plus.