Cloud Reboot

Duration: long term contract

Location: Philadelphia, PA (Hybrid role)

JOB DESCRIPTION:

Senior Azure Databricks Architect with experience leading tenant-to-tenant migrations and platform separations

The Azure Databricks environment currently runs inside another company's Azure tenant. As part of a broader technology separation, the platform must be migrated into its own tenant while preserving security, governance, operational continuity, and business-critical workloads.

We're seeking a hands-on Senior Azure Databricks Architect to lead this migration and separation effort end-to-end, covering platform architecture, tenant-to-tenant migration strategy, Unity Catalog and governance, security and networking, Infrastructure as Code, CI/CD, and cutover execution. The ideal candidate has led Azure Databricks migrations, cloud carve-outs, or tenant separation initiatives; data engineering experience is a plus, but deep expertise in Databricks architecture and migration is the core requirement.

Qualifications (Critical Skills)

• 8 years in cloud data platform architecture or enterprise platform operations, including 5 years hands-on Azure Databricks architecture, administration, and operations.
• Proven migration leadership on Azure Databricks tenant-to-tenant migrations, cloud carve-outs, divestitures, or comparable enterprise-scale platform migrations.
• Cutover and rollback expertise, including migration plans, runbooks, validation checklists, and hypercare — with strong risk management and stakeholder coordination for live production transitions.
• Deep governance expertise with Unity Catalog or Hive metastore — catalog/schema design, external locations, permissions, and lineage.
• Proven ability to design scalable, secure, cost-aware Databricks environments across Dev/Test/UAT/Prod with monitoring, runbooks, and support models.

Azure Infrastructure, Security & Automation

• Strong Azure networking and security experience (VNets, Private Link, NSGs, Key Vault, managed identities, Entra ID).
• Strong Terraform experience (ARM/Bicep acceptable); CI/CD experience with Azure DevOps or GitHub Actions.
• Ability to lead technical decisions across platform, security, and data teams, and communicate migration risks, tradeoffs, and readiness decisions to technical and business stakeholders.

What you’ll be doing Platform Assessment & Migration Planning

• Assess & document. Audit workspaces, clusters, jobs, secrets, identities, networking, storage, and metastore/Unity Catalog; identify dependencies, risks, and migration blockers.
• Plan the migration. Define sequencing, cutover approach, validation plan, rollback procedure, and post-migration support model.

Databricks Architecture & Target Environment Design

• Design target-state architecture. Define the workspace strategy, network topology, governance model, and Unity Catalog/metastore strategy (external locations, storage credentials, catalog/schema structure, and access control) in the tenant.
• Establish platform standards. Define cluster policies, compute patterns, naming conventions, environment promotion, monitoring, and cost management.

Security, Identity & Governance

• Rebuild identity & access. Implement Microsoft Entra ID, SSO/SCIM, RBAC, service principals, managed identities, and least-privilege access.
• Re-establish security controls & governance. Implement Key Vault, private endpoints/Private Link, VNet peering, NSGs, storage firewalls, and Unity Catalog governance policies with auditability.

Tenant-to-Tenant Migration Execution

• Lead migration execution. Move or recreate workspaces, notebooks, jobs, cluster configurations, secrets, CI/CD assets, and integrations in the target tenant.
• Manage cutover & validate readiness. Coordinate migration windows, go/no-go decisions, hypercare, and post-migration validation of access, job execution, security, and monitoring.

Infrastructure Automation & DevOps

• Automate & rebuild pipelines. Provision environments with Terraform (preferred) or Bicep/ARM; rebuild CI/CD in Azure DevOps or GitHub Actions for platform, notebook, and job deployment.
• Support pipeline migration. Partner with data engineering on ETL/ELT, Databricks Workflows, and Delta Live Tables affected by the separation; validate data parity and job reliability.

Stakeholder Coordination & Handoff

• Coordinate & document. Manage dependencies and cutover readiness across infrastructure, security, application, and business teams; produce architecture diagrams, runbooks, and handoff documentation.

Relevant Cert or Tech Experience

• Certifications: Databricks Certified Professional Data Architect / Data Engineer Professional, Azure Solutions Architect Expert, or similar.
• M&A / carve-out experience: IT carve-outs, divestitures, mergers, acquisitions, TSA environments, or complex enterprise technology separations.
• Advanced Databricks features: Asset Bundles, Lakehouse Federation, system tables, FinOps, or governance automation.
• Data engineering (plus): PySpark, SQL, Delta Lake, Delta Live Tables, Azure Data Factory, and ETL/ELT validation.
• Enterprise integration (plus): BI/reporting platforms, APIs, and ERP/EDW systems.

Work Experience

• 8 years in cloud data platform architecture or enterprise platform operations, including 5 years of hands-on Azure Databricks architecture, administration, and operations.