ZipRecruiter

Log In

1

Databricks Software Jobs in New York (NOW HIRING)

TheStaffed

Senior Software Engineer - Databricks

New York, NY · On-site

$134K - $176K/yr

... Software Engineer with 8-10 years of experience to rebuild and develop applications for the Direct Investment Unit, focusing on data pipelines and platform modernization using Databricks while ...

next page

Showing results 1-20

All JobsDatabricks Software JobsDatabricks Software Jobs in New York

Databricks Software information

What is Databricks software?

Databricks Software is a unified analytics platform built on Apache Spark that provides tools for big data processing, machine learning, and collaborative data science. It enables organizations to store, manage, and analyze large datasets efficiently, supporting both batch and streaming data workloads. Databricks also offers collaborative notebooks, automated workflows, and integrations with cloud storage and data lakes, making it a popular choice for data engineering, data science, and business analytics teams.

What are some common challenges faced by Databricks software engineers, and how can they be overcome?

Databricks Software Engineers often encounter challenges related to scaling big data pipelines, optimizing Spark workloads, and integrating diverse data sources. Navigating the complexity of distributed systems and managing cloud infrastructure can be demanding, especially when ensuring data reliability and security. To overcome these challenges, engineers typically collaborate closely with data scientists, DevOps, and platform teams, leverage Databricks' extensive documentation and community support, and adopt best practices such as version control and continuous integration. Regular knowledge sharing and staying updated with new features also help engineers succeed in this dynamic environment.

What are the key skills and qualifications needed to thrive as a Databricks software engineer, and why are they important?

To thrive as a Databricks Software Engineer, you need strong programming skills in languages like Python, Scala, or Java, as well as a solid understanding of distributed computing and data engineering concepts. Familiarity with Databricks platform, Apache Spark, cloud services (such as AWS or Azure), and relevant certifications like Databricks Certified Data Engineer are highly valued. Excellent problem-solving abilities, collaboration, and effective communication are important soft skills for this role. These skills ensure efficient development, deployment, and optimization of big data solutions that drive business insights and innovation.

What is the difference between Databricks Software vs Data Engineer?

AspectDatabricks SoftwareData Engineer
Primary RolePlatform for data analytics and machine learningBuilds, maintains data pipelines and infrastructure
Required SkillsSQL, Spark, cloud platforms, data science basicsSQL, ETL, programming (Python, Scala), database management
Work EnvironmentCloud-based, collaborative data platformData teams, cloud or on-premises environments
CertificationsDatabricks certifications, cloud certificationsNone specific, often cloud or data certifications

While Databricks Software provides a platform for data analytics and machine learning, Data Engineers focus on building and maintaining data pipelines and infrastructure. Both roles often work together but have distinct responsibilities and skill sets within the data ecosystem.

What are popular job titles related to Databricks Software jobs in New York? For Databricks Software jobs in New York, the most frequently searched job titles are:
What job categories do people searching Databricks Software jobs in New York look for? The top searched job categories for Databricks Software jobs in New York are:
What cities in New York are hiring for Databricks Software jobs? Cities in New York with the most Databricks Software job openings:
Databricks Software jobs near you

Senior Software Engineer - Databricks

TheStaffed

New York, NY • On-site

$134K - $176K/yr

Contractor

Posted 5 days ago

Job description

Our client is seeking a Senior Software Engineer with 8-10 years of experience to rebuild and develop applications for the Direct Investment Unit, focusing on data pipelines and platform modernization using Databricks while collaborating across technical and non-technical teams.
Responsibilities & Qualifications
  • Rebuild existing applications and design new solutions for the Direct Investment Unit on the Databricks platform
  • Develop and optimize data pipelines, leveraging Databricks scheduler and the internal Alpha Data Platform
  • Analyze requirements and work cross-functionally with non-technical stakeholders to define pragmatic solutions
  • Integrate with investment management systems, including CRIMS and CRIMS API, to support core business processes
  • Build and maintain full-stack applications using React, REST/GraphQL, and backend technologies (C#, MongoDB)
  • Redesign and modernize legacy processes to improve scalability and maintainability on the Databricks ecosystem
  • Contribute to platform architecture decisions and best practices across the technology organization

Requirements
  • 8-10 years of professional software engineering experience
  • Strong expertise in Databricks, SQL, and data pipeline development using Databricks scheduler
  • Hands-on experience with investment management systems, specifically CRIMS and CRIMS API integration
  • Proficiency in React, C#, MongoDB, and REST/GraphQL API design
  • Demonstrated ability to work effectively across technical teams and with non-technical stakeholders
  • Familiarity with the internal Alpha Data Platform and investment management domain knowledge

Apply