Thought Catalog Jobs (NOW HIRING)

Databricks Subject Matter Expert Permanent Full Time Location: Atlanta, Georgia, United States Description: Databricks Subject Matter Expert Position Description We are seeking a highly experienced Databricks Subject Matter Expert to join at the Lead / Principal level. In this role, you will serve as the firm's foremost technical authority on the Databricks Lakehouse Platform, guiding clients through complex data transformation initiatives—from strategic architecture design through hands-on implementation and production deployment. You will partner directly with client stakeholders (CTOs, VPs of Data, and Engineering Leads) to define data strategies, architect scalable solutions on Databricks, and mentor delivery teams. This is a high-visibility role that blends deep technical expertise with client advisory, pre-sales support, and thought leadership. This role is a hybrid model. We are seeking a highly experienced Databricks Subject Matter Expert to join at the Lead / Principal level. In this role, you will serve as the firm's foremost technical authority on the Databricks Lakehouse Platform, guiding clients through complex data transformation initiatives—from strategic architecture design through hands-on implementation and production deployment. You will partner directly with client stakeholders (CTOs, VPs of Data, and Engineering Leads) to define data strategies, architect scalable solutions on Databricks, and mentor delivery teams. This is a high-visibility role that blends deep technical expertise with client advisory, pre-sales support, and thought leadership. Your future duties and responsibilities Solution Architecture & Design . Lead end-to-end design of Databricks Lakehouse architectures, including Delta Lake, Unity Catalog, and medallion (bronze/silver/gold) data models. . Define reference architectures, best practices, and reusable accelerators for client engagements. . Evaluate and recommend deployment patterns across Azure Databricks, AWS Databricks, and GCP, aligning platform choices with client infrastructure and governance needs. Data Engineering & Pipeline Development . Architect and oversee large-scale ETL/ELT pipelines using Apache Spark, Delta Live Tables, and Databricks Workflows. . Design real-time and batch ingestion frameworks covering structured, semi-structured, and unstructured data sources. . Establish data quality, lineage, and observability standards using Unity Catalog, Delta Lake features, and complementary tooling. Machine Learning & MLOps . Guide clients on implementing end-to-end ML workflows using MLflow, Feature Store, and Model Serving on Databricks. . Advise on MLOps patterns including model versioning, A/B testing, monitoring, and automated retraining pipelines. . Collaborate with data science teams to productionize models and integrate ML outputs into business applications. Client Advisory & Engagement Leadership . Serve as the trusted advisor to senior client stakeholders on all matters related to data platform strategy and Databricks adoption. . Lead discovery workshops, architecture reviews, and technical assessments to define solution roadmaps. . Support pre-sales efforts by developing proposals, estimating level of effort, and delivering technical demonstrations. Thought Leadership & Practice Development . Contribute to the firm's intellectual property by developing frameworks, white papers, and training curricula around Databricks. . Mentor and upskill consultants across the data practice; foster a community of Databricks practitioners. . Represent the firm at industry events, Databricks partner summits, and webinars. Required qualifications to be successful in this role . 10+ years of professional experience in data engineering, analytics, or platform architecture. . 5+ years of deep, hands-on experience with Databricks, including production deployments at enterprise scale. . Expert-level proficiency in Apache Spark (PySpark and/or Scala), Delta Lake, and SQL. . Proven track record designing multi-cloud Lakehouse architectures (Azure, AWS, or GCP). . Strong experience with Unity Catalog for data governance, access control, and lineage tracking. . Demonstrated ability to lead client-facing engagements and communicate complex technical concepts to executive audiences. . Experience in a consulting or professional services environment, with a history of managing multiple concurrent engagements. . Bachelor's degree in Computer Science, Engineering, Data Science, or a related field (or equivalent practical experience). Preferred Qualifications . Databricks Certified Data Engineer Professional or Machine Learning Professional certification. . Experience with Delta Live Tables, Databricks Workflows, and Photon engine optimization. . Familiarity with complementary tools such as dbt, Terraform/Pulumi (IaC), and CI/CD pipelines for data platforms. . Hands-on experience with MLflow, Feature Store, and Model Serving in production environments. . Prior experience as a Databricks partner or in a Databricks-aligned practice within a consulting firm. . Advanced degree (M.S. or Ph.D.) in a quantitative or technical discipline. #M1 #DI-CB2 #L1 - KB1