Job Summary:
Ziosk is a company that empowers restaurants to enhance the guest experience through innovative solutions. They are seeking a highly skilled Senior Data Engineer with expertise in Azure Databricks to design and optimize their modern data platform, focusing on building scalable ETL pipelines and collaborating with various teams to deliver high-performance data solutions.
Responsibilities:
• Design and implement scalable ETL/ELT pipelines using Azure Databricks (PySpark, Spark SQL).
• Build robust data ingestion workflows from diverse sources (Event Hub, APIs, Azure SQL, Blob Storage, etc.) into the Bronze/Silver/Gold layers of the medallion architecture.
• Define and implement best practices for Delta Lake and Unity Catalog for schema evolution, versioning, and governance.
• Optimize performance and cost efficiency of large-scale data pipelines through intelligent partitioning, caching, clustering, and tuning Spark configurations.
• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), Azure Synapse, Event Hub, Key Vault, and Azure DevOps.
• Implement secure and governed access to data using managed identities, role-based access control (RBAC), and data masking techniques.
• Partner with data analysts, data scientists, and business users to understand data needs and deliver trusted datasets and KPIs.
• Provide technical leadership and mentorship to junior engineers, conducting code reviews and guiding architectural decisions.
• Collaborate with enterprise architects and InfoSec to ensure compliance with data security and privacy policies.
• Implement data quality checks, alerts, and lineage tracking using tools such as Great Expectations, Unity Catalog, and Data Observability platforms.
• Set up monitoring dashboards, alerts, and job recovery mechanisms for mission-critical pipelines.
Qualifications:
Required:
• Bachelor's or Master's degree in Computer Science, Engineering, or related field
• 5+ years of experience in data engineering, with 3+ years focused on Azure Databricks and Spark (PySpark)
• Expertise in Delta Lake, Unity Catalog, Databricks Notebooks, and SQL Warehousing
• Hands-on experience with Azure cloud services: ADLS Gen2, Event Hub, ADF, Key Vault, Synapse
• Strong programming skills in Python, SQL, and familiarity with Scala
• Solid understanding of data modeling (OLTP/OLAP), star/snowflake schemas, and performance tuning
• Experience with CI/CD pipelines in Databricks using Databricks CLI, Terraform, or Azure DevOps
Preferred:
• Databricks certification (e.g., Databricks Certified Data Engineer Professional)
• Experience with real-time streaming (e.g., using Structured Streaming, Event Hub, Kafka)
• Familiarity with MLflow or machine learning pipeline support in Databricks
• Experience with Power BI, Azure Purview, or Microsoft Fabric for downstream reporting
• Knowledge of data governance, lineage, and compliance standards (GDPR, HIPAA, SOC2)
Company:
Ziosk provides technology solutions for the restaurant industry Founded in 2007, the company is headquartered in Dallas, USA, with a team of 51-200 employees. The company is currently Growth Stage.