Databricks Software Jobs in New York (NOW HIRING)

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Data Engineer (Databricks) based in Canada.

As a Data Engineer, you will build and operate the data products that power audience solutions for leading brands and commercial partners.
The role is heavily focused on Databricks development, data pipelines, orchestration, data modeling, and scalable engineering practices.
You will transform survey, behavioral, and machine-learning signals into reliable syndicated and custom audience segments.
Working closely with Data Engineering, Data Science, and commercial stakeholders, you will translate business needs into robust technical solutions.
You will also help advance engineering standards across testing, CI/CD, observability, metadata, and data privacy.
The environment encourages experimentation with emerging Databricks capabilities and AI-assisted development tools.
This is a fully remote opportunity for Canada-based professionals, with occasional travel to New York City.

• Build, maintain, and optimize data tables, views, jobs, pipelines, and orchestration workflows within the Databricks environment.
• Develop and operate pipelines that produce and deliver syndicated and custom audience segments based on survey-derived and machine-learning methodologies.
• Apply data modeling and architecture best practices while helping improve standards for testing, repositories, CI/CD, naming conventions, and code quality.
• Support the delivery and performance reporting of audience segments distributed through data marketplaces and partner platforms.
• Investigate and implement emerging Databricks capabilities for real-time processing and operational workloads, including Spark Structured Streaming and Lakeflow Declarative Pipelines.
• Maintain a high-quality engineering environment with strong data observability, metadata standards, documentation, and data-quality practices.
• Ensure consumer data is handled responsibly and in accordance with applicable data privacy and internal data governance standards.
• Use AI-assisted development tools such as Claude Code to accelerate development, testing, documentation, and engineering workflows while maintaining rigorous quality standards.
• Collaborate with Data Engineering, Data Science, and commercial stakeholders to understand business needs and translate them into functional and technical requirements.
• Contribute to architecture discussions, engineering solution design, execution planning, releases, and internal training.
• Monitor emerging technologies, trends, and capabilities across the Databricks ecosystem and evaluate opportunities for practical adoption.
• Share technical knowledge through brown-bag sessions, tech talks, documentation, and advocacy of effective engineering practices.

• Bachelor's or Master's degree in Computer Science or a related technical discipline.
• 4+ years of professional Data Engineering experience, with strong expertise in Python, SQL, Spark, and distributed data processing.
• 2+ years of hands‑on experience with Databricks, including Data Engineering and Data Warehousing capabilities such as Jobs & Workflows, Notebooks, Unity Catalog, and Delta Lake.
• Strong understanding of database design, data structures, algorithms, data modeling, and scalable data architecture.
• Demonstrated ability to translate business requirements into practical and maintainable engineering solutions.
• Experience using AI coding assistants such as Claude Code, GitHub Copilot, or comparable tools within a modern software engineering workflow.
• Strong individual‑contributor mindset, with the ability to work independently while collaborating effectively with Data Engineering, Data Science, and business teams.
• Strong focus on data quality, testing, observability, documentation, privacy, and engineering best practices.
• Curiosity and willingness to stay current with new technologies and developments in the Databricks ecosystem.
• Familiarity with emerging Databricks technologies such as Genie, Lakebase, AI/BI Dashboards, or Databricks Apps is an asset.
• Previous experience in digital marketing, ad‑tech, media, advertising, or a related industry is a plus.
• Familiarity with audience, identity, or consumer segmentation data and data marketplaces such as LiveRamp, TransUnion, or Lotame is an advantage.

• Competitive compensation: CAD $80,000–$90,000 annually, depending on factors such as location, experience, and performance.
• Fully remote work opportunity for candidates based in Canada.
• Occasional travel opportunities to New York City.
• Career and professional development opportunities.
• Health, dental, and vision insurance.
• Pre-tax savings plans and transit/parking programs.
• 401(k) plan with competitive employer matching, where applicable.
• Educational, social, and team-building events.
• Volunteer and philanthropic initiatives.
• Collaborative culture with opportunities to learn from experienced technical and business professionals.
• Exposure to innovative technologies across data engineering, AI, advertising, and audience solutions.

We appreciate your interest and wish you the best!