Senior Databricks Architect Jobs in Indiana (NOW HIRING)

Description:

At CSpring, we believe in the power of people and data to drive real-world impact. We’re a purpose-driven consulting firm that helps organizations solve complex business and technology challenges through modern data, analytics, and engineering solutions.

We are seeking a Senior Azure Data Engineer to help design, build, and support a next-generation enterprise data platform on Microsoft Azure. In this role, you will lead the development of scalable data pipelines and data products that power analytics, operational reporting, dashboards, and emerging AI/ML use cases.

You’ll work closely with data architects, analytics engineers, data scientists, platform teams, and business stakeholders to deliver secure, high-performing, and cost-effective cloud data solutions. This role is ideal for someone with deep hands-on experience in Azure data technologies, modern Lakehouse architectures, and enterprise-scale data migrations.

What You'll Do

Data Pipeline Engineering

• Design and develop reusable, parameter-driven ingestion and transformation pipelines using Azure Data Factory, Synapse Pipelines, Databricks, and/or Microsoft Fabric Data Factory
• Build and maintain medallion architecture (Bronze / Silver / Gold) solutions using Azure Data Lake Storage Gen2, Delta Lake, Parquet, and structured streaming patterns
• Develop performant ELT workflows leveraging pushdown processing to platforms such as Synapse Dedicated SQL Pool, Azure SQL, and Teradata
• Create and optimize PySpark notebooks and distributed processing jobs in Azure Databricks or Synapse Spark

Data Warehousing & Modeling

• Design dimensional data models using Kimball star and snowflake methodologies
• Implement data vault patterns, Slowly Changing Dimensions (Type 1/2/3), Change Data Capture, and late-arriving data strategies
• Optimize distributed SQL workloads in Synapse Dedicated SQL Pool and/or Fabric Warehouse environments
• Tune partitioning, indexing, and query performance for enterprise-scale datasets

Cloud Platform Engineering & DevOps

• Implement CI/CD processes for data pipelines using Azure DevOps, YAML pipelines, ARM templates, Bicep, and/or Terraform
• Build monitoring, logging, and auditing solutions using Azure Monitor, Log Analytics, and KQL
• Support code reviews, branching strategies, release management, and engineering standards across environments
• Participate in troubleshooting and production incident response for critical data pipelines

Migration & Modernization

• Lead or contribute to cloud modernization initiatives, including Informatica PowerCenter to Azure Data Factory migrations
• Support migration efforts from on-premises Teradata, Oracle, or SQL Server environments to Azure Synapse or Microsoft Fabric
• Assist with workload assessments, capacity planning, and cloud cost optimization initiatives

Requirements:

What You Bring

Required Qualifications

• Deep hands-on expertise with Azure Data Factory, including pipelines, datasets, linked services, triggers, parameterization, mapping data flows, and Integration Runtime types (Azure, Self-hosted, and SSIS)
• Strong experience with Azure Databricks and PySpark
• Production experience with one or more of the following:
  • Azure Synapse Analytics (Dedicated SQL Pools, Serverless SQL Pools, Spark Pools)
  • Azure Databricks (Delta Lake, Unity Catalog)
  • Microsoft Fabric (Warehouse, Lakehouse, OneLake)
• Strong understanding of Azure Data Lake Storage Gen2, including hierarchical namespace, RBAC/ACL security, lifecycle management, and governance
• Experience with Azure Key Vault, Azure AD / Entra ID, managed identities, service principals, and private networking concepts
• Experience monitoring and troubleshooting data solutions using Azure Monitor, Log Analytics, and KQL
• Advanced SQL skills including window functions, CTEs, query optimization, execution plan analysis, and performance tuning
• Strong Python skills for data engineering, including pandas, PySpark, REST API integration, and unit testing with pytest
• Proficiency with T-SQL and familiarity with Spark SQL, KQL, PowerShell, and Bash scripting

Preferred Qualifications

• 5+ years of enterprise data warehouse or data engineering experience
• 5+ years of data modeling experience using ERWIN or similar modeling tools
• 2+ years of experience with Azure Data Factory and Snowflake
• Experience working in healthcare or Medicaid environments

Why CSpring

At CSpring, you’ll join a collaborative, mission-driven team focused on helping organizations use technology and data to improve outcomes and drive meaningful change. We value curiosity, accountability, continuous learning, and building strong partnerships with our clients and communities.If you’re passionate about modern cloud data engineering and want to work on impactful enterprise initiatives, we’d love to hear from you.Source role provided by user: ?filecite?turn0file0?L1-L46?