Sr Azure Data Engineer Jobs in Calgary, AB (NOW HIRING)

Data Elephant is a fast-growing Canadian data and AI consulting firm. We help organizations design, modernize, and scale cloud data platforms using technologies including Databricks, Microsoft Azure, AWS, Power BI, and modern AI and analytics solutions.

We are looking for a Senior Azure Databricks Architect with deep, hands-on experience designing and deploying secure enterprise Databricks environments. This role requires demonstrated experience with Terraform and Azure Private Link - not simply experience working in an environment where these technologies were already configured.

The Opportunity

You will lead the architecture and deployment of secure Azure Databricks environments for enterprise clients. You will work closely with client security, networking, cloud platform, and data engineering teams to design solutions that meet demanding security, connectivity, governance, and operational requirements.

This is a hands-on architecture role. The successful candidate must be comfortable defining the target architecture, making technical decisions, writing and reviewing Terraform, troubleshooting complex networking issues, and supporting implementation through production deployment.

KeyResponsibilities

• Designsecure,scalableAzureDatabricksplatformarchitectures.
• DeployandconfigureAzureDatabricksworkspacesusingTerraformandinfrastructure-as-codepractices.
• Design and implement Azure Private Link and private endpoint connectivity for Databricks
• Configure front-end, back-end, and browser-authentication private connectivity where required.
• Designcustomer-managedVNetandVNet-injectedDatabricksdeployments.
• ImplementSecureClusterConnectivityandno-public-IPconfigurations.
• Design private DNS zones, DNS resolution, routing, NSGs, user-defined routes, and firewall rules.
• Integrate Databricks into hub-and-spoke and other enterprise Azure network architectures.
• Work with security and networking teams to troubleshoot DNS, routing, authentication, and connectivity issues.
• Design identity and access controls using Microsoft Entra ID and Databricks Unity Catalog.
• Establisharchitecturestandardsanddeploymentpatternsacrossdevelopment,test,andproductionenvironments.
• ImplementCI/CDprocessesforDatabricksinfrastructureandplatformconfiguration.
• Producearchitecturediagrams,designdecisions,deploymentdocumentation,andoperationalrunbooks.
• Providetechnicalleadershipandguidancetodataengineers,cloudengineers,andclientstakeholders.

Required Experience

• SignificantexperiencearchitectingandimplementingAzureDatabricksenvironments.
• Hands-onexperiencedeployingAzureDatabricksusingTerraform.
• Demonstrated experience implementing Azure Private Link and private endpoints for Databricks.
• Experience with VNet injection, customer-managed VNets, Secure Cluster Connectivity, and no-public-IP deployments.
• StrongunderstandingofAzurenetworking,including:
  • PrivateDNSzonesandDNSresolution
• Experience implementing hub-and-spoke architectures, Azure firewalls, network security groups, and user-defined routes
• ExperiencewithExpressRouteorVPNconnectivity
• ExperiencewithTerraformresources
• Experience designing secure connectivity between Databricks and services such as ADLS Gen2, Key Vault, Azure SQL, Event Hubs, or other Azure data services.
• Strong knowledge of Databricks security, identity, access management, Unity Catalog, and service principals.
• Experience establishing CI/CD and infrastructure-as code practices for enterprise data platforms.
• Ability to translate security and business requirements into practical architecture decisions.
• Strong written and verbal communication skills, including experience working directly with clients and technical stakeholders.

PreferredQualifications

• Databricksarchitectureorengineeringcertification.
• RelevantMicrosoftAzurecertification.
• ExperiencewithDatabricksAssetBundles.
• ExperiencewithAzureDevOpsorGitHubActions.
• Knowledgeofdatagovernance,medallionarchitecture,observability,andcloudcostmanagement.
• Consultingorprofessionalservicesexperience.

What Success Looks Like

YoucandescribeAzureDatabricksenvironmentsyouhavepersonallydesignedanddeployedusingTerraformandPrivateLink.Youunderstandhowfront-endaccess,back-endconnectivity,browserauthentication,privateDNS,routing,andno-public-IPclusterconnectivityworktogether-andyoucandiagnoseissueswhentheydonot.

Youcombinestrongarchitecturejudgmentwiththeabilitytoworkhands-onandguideasolutionthroughimplementation.

Location

Remote within Canada - only candidates based in Canada will be considered for this role.