Location: Remote (Preference for candidates located in New England)
Engagement: Contract or Contract-to-Hire
Lead Data Engineer
Databricks & Microsoft Fabric
Overview
Spyglass MTG is seeking a highly skilled Lead Data Engineer with strong expertise in Databricks and Microsoft Fabric to help design, build, and scale modern data platforms in Azure.
This role is both strategic and hands-on. The ideal candidate will lead the development of enterprise data solutions while actively building scalable pipelines, lakehouse architectures, and data platforms that support analytics, reporting, and AI initiatives.
The Lead Data Engineer will work closely with data architects, analysts, and business stakeholders to deliver high-performance data solutions that enable organizations to leverage their data more effectively.
Responsibilities
โข Design and implement modern data platforms using Databricks and Microsoft Fabric
โข Build and maintain scalable data pipelines and ingestion frameworks
โข Develop and optimize Spark and PySpark workloads within Databricks
โข Implement lakehouse and data warehouse architectures
โข Design and manage data models supporting analytics and reporting
โข Collaborate with data scientists and analytics teams to support AI and advanced analytics initiatives
โข Implement data quality, governance, and security best practices
โข Provide technical leadership and mentorship to data engineering team members
โข Optimize performance, reliability, and scalability of data platforms
Requirements
โข 7+ years of experience in data engineering or data platform development
โข Strong hands-on experience with Databricks and Apache Spark
โข Experience implementing Microsoft Fabric data solutions
โข Proficiency in Python and PySpark
โข Experience working with Azure data services and cloud-based data platforms
โข Experience designing modern lakehouse or data warehouse architectures
โข Strong understanding of data pipeline design, ETL/ELT processes, and data modeling
Preferred
โข Experience supporting AI or machine learning workloads
โข Experience working in enterprise data environments
โข Familiarity with CI/CD and data platform automation