Director Databricks Data Engineer Jobs in New Jersey

Role: Senior Databricks Data Architect (AWS)

Location: Ridgefield Park, New Jersey 07660

12 Contract onsite position - This role requires 100% onsite presence Monday through Friday and will cover core business hours based on the EST timezone.

JOB DESCRIPTION

JOB TITLE: Senior Databricks Data Architect (AWS) - Hands-On Technical Lead

PURPOSE OF THE JOB:

• Lead the strategic design and architectural roadmap of the Enterprise Data Platform on Databricks/AWS while remaining actively hands-on in design, prototyping, coding, and performance tuning.
• To provide high-level technical consultancy and solution design to internal business stakeholders and external partners, acting as the bridge between complex business requirements and technical implementation.
• To govern the technical quality of the platform by establishing design standards, conducting code reviews, and guiding the engineering team on best practices for Databricks Serverless and AWS integration.
• As part of the role, you will be required to provide technical assurance for all major data initiatives, ensuring that solutions are robust, cost-effective, and compliant with enterprise security standards.

KEY ACCOUNTABILITIES:

• Lead all architecture design and review sessions for strategic Databricks initiatives.
• Actively contribute code to critical architectural components, building complex frameworks, reusable libraries, and core ETL/ELT pipelines using Python and PySpark.
• Lead by example by prototyping Proof of Concepts (PoCs) for new features (e.g., Databricks Asset Bundles, Delta Live Tables) before handing them off for wider team implementation.
• Perform hands-on performance tuning and debugging of high-latency jobs by deep-diving into Spark UI, execution plans, and spill logs.
• Defining and enforcing coding standards, data modeling best practices (Medallion Architecture), and CI/CD protocols for the development team to follow.
• conducting deep-dive technical reviews of designs and code produced by the engineering team to ensure efficiency, maintainability, and security.
• Designing the strategy for Unity Catalog governance, ensuring granular data access controls and compliance with data privacy regulations.
• Acting as the Technical Design Authority for the Databricks platform, making high-level decisions regarding the adoption of new Serverless features and AWS services.
• Mentoring and coaching Senior and Junior engineers, fostering a culture of technical excellence and continuous improvement.
• Own the technical relationship with key business stakeholders and translate business strategy into robust data architecture
• Engage and collaborate with Executive Stakeholders to present technical roadmaps and demonstrate the business value of data initiatives.
• Actively seeks opportunities to develop knowledge in emerging technologies (e.g., Generative AI on Databricks, Vector Databases) and evaluate their impact on the business.
• Provide guidance to the Change Management board regarding the risk and technical impact of proposed architectural changes.
• Develop strategies to architect resilience, high availability, and disaster recovery into the data platform.
• Develop clear communication channels with C-level executives and Product Managers to ensure technical strategies align with business goals.

KEY LIAISONS:

• Head of Data & Analytics / CIO
• Business Unit Directors & Product Managers
• Lead Data Engineers and Data Scientists
• Enterprise Security & Infrastructure Architects

DIMENSIONS:

• Ownership of the Technical Roadmap and Architecture for the Data Estate
• Maintain excellent relationships with all key liaisons
• Responsible for architectural decisions impacting critical data assets
• Primary Technical Advisor in the CST timezone

EDUCATION:

• Educated to degree level in Computer Science, Engineering, or equivalent (Master’s degree preferred)

SKILLS AND ATTRIBUTES:
Essential –

• Exceptional "Customer Facing" skills: proven ability to lead workshops, manage stakeholder expectations, and present complex technical solutions to non-technical audiences.
• Minimum 8-10 years’ experience in Data Engineering/Architecture, with at least 4 years specialized in Databricks and AWS.
• Expert-level knowledge of Data Lakehouse Architecture, Delta Lake internals, and Spark optimization strategies.
• Proven experience in Data Modeling (Dimensional, Vault, Relational) and designing complex ETL/ELT pipelines.
• Advanced hands-on coding proficiency in Python and SQL, with the ability to write clean, production-ready, and testable code.
• Strong experience with Databricks Serverless architecture, Unity Catalog implementation, and security patterns.
• Deep understanding of Cloud Architecture (AWS Well-Architected Framework), including networking, storage (S3), and IAM.
• Experience in Design Review and Code Review processes, capable of constructive critique and enforcing standards.
• Detailed knowledge of DevOps principles (IaC with Terraform, CI/CD pipelines).
• Knowledge of the MS Office suite (Word, Excel, PowerPoint, Outlook) – specifically for creating architectural diagrams and presentations.
• Very strong relationship building and influencing skills at a senior level.
• Strategic thinking with the ability to balance immediate business needs with long-term architectural vision.

EXPERIENCE:
Essential -

• 8+ years of experience in Data Architecture and Engineering.
• Extensive experience designing solutions on the Databricks platform.

Advantage -

• Strong experience working with large-scale enterprises or as a Consultant for big agencies.
• Certified Databricks Data Architect or AWS Solutions Architect (Professional).
• Excellent organizational and planning skills, with an ability to manage architectural backlogs.
• A best-in-class technical leader proactively driving business innovation and process improvements.
• Languages: Python, SQL, Scala.
• Experience designing APIs for data consumption.
• Experience with Enterprise Architecture frameworks (TOGAF).