Key Responsibilities
Architecture & Platform Engineering
- Define and implement end-to-end data platform architecture utilizing Databricks, Apache Spark, Delta Lake, Unity Catalog, and Azure Data Services.
- Design scalable, secure, and resilient Lakehouse architectures that support enterprise reporting, advanced analytics, machine learning, and data products.
- Establish architectural standards and best practices for data ingestion, transformation, storage, consumption, and lifecycle management.
- Design multi-tenant data platform solutions leveraging Unity Catalog while enforcing governance, security, metadata management, and data-sharing principles.
- Define logical and physical data models, integration patterns, and framework accelerators to improve platform consistency and reuse.
- Lead architecture reviews and ensure alignment with enterprise architecture, security, and compliance standards.
Technical Leadership & Delivery Governance
- Serve as the primary technical authority for the data platform program and guide engineering teams through design and implementation decisions.
- Provide technical leadership to onshore and offshore teams, ensuring solution quality, consistency, and adherence to architectural standards.
- Review solution designs, code, deployment strategies, and performance optimization approaches.
- Establish engineering best practices for scalability, observability, reliability, and operational support.
- Drive issue resolution for complex technical challenges and act as an escalation point for critical platform concerns.
- Collaborate with business stakeholders, product owners, data engineers, data scientists, and enterprise architects to translate business requirements into technical solutions.
Platform Innovation & Engineering Excellence
- Evaluate emerging Databricks capabilities, Azure services, and industry best practices to enhance platform maturity and reduce technical debt.
- Drive automation across deployment, testing, monitoring, and operational processes using Azure DevOps and modern CI/CD frameworks.
- Establish reusable frameworks, accelerators, templates, and operational playbooks to improve delivery velocity and platform consistency.
- Define and monitor platform KPIs related to performance, cost optimization, data quality, reliability, and operational efficiency.
- Collaborate with governance and security teams to implement data lineage, access controls, auditing, and compliance frameworks.
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
- 10–15+ years of experience in Data Engineering, Data Architecture, or Cloud Data Platform leadership roles.
- Hands-on expertise with Databricks, Apache Spark, Delta Lake, and Azure Data Services.
- Strong understanding of Lakehouse architecture, modern data warehousing principles, and enterprise-scale data platforms.
- Proven experience designing and implementing batch and real-time ETL/ELT data pipelines.
- Strong proficiency in SQL, data modeling, schema design, and performance tuning.
- Experience with structured and semi-structured data formats including Parquet, ORC, Avro, and JSON.
- Deep understanding of cloud security, identity management, data governance, and metadata management using Unity Catalog.
- Experience implementing CI/CD pipelines and infrastructure automation for data platforms.
- Demonstrated experience leading distributed engineering teams and managing large-scale platform implementations.
- Excellent communication, stakeholder management, analytical, and problem-solving skills.
Preferred Qualifications
- Experience implementing data governance, data lineage, metadata management, or data quality frameworks.
- Exposure to domain-driven design, data product operating models, and data mesh concepts.
- Experience supporting AI/ML, GenAI, or advanced analytics workloads on Databricks.
- Databricks, Azure, or cloud architecture certifications.
- Experience within the Property & Casualty Insurance domain or financial services industry.
- Familiarity with FinOps, cloud cost optimization, and platform observability frameworks.