Job Summary:
Tredence Inc. focuses on delivering insights through business analytics, data science, and software engineering. They are seeking an experienced Databricks Architect to design, build, and optimize their modern Lakehouse platform, leveraging the Databricks ecosystem to deliver scalable data solutions.
Responsibilities:
• Architect and implement end-to-end Lakehouse solutions on Databricks.
• Design and optimize Delta Lake storage, including ACID transactions, time travel, and schema evolution.
• Configure clustering, partitioning, Z-order, and vacuuming for performance tuning.
• Build data models (Kimball, Inmon, Data Vault, or medallion architecture: bronze/silver/gold).
• Implement Data Catalog (Unity Catalog) for metadata management, lineage, and access control.
• Define and enforce Data Quality rules using Great Expectations, DBT, or Databricks DLT (Delta Live Tables).
• Develop scalable ETL/ELT pipelines using PySpark, Python, and SQL.
• Optimize Spark jobs for performance, cost, and reliability.
• Automate workflows with Databricks Jobs, workflows, and orchestration tools (Airflow, Azure Data Factory, etc.).
• Partner with data scientists, analysts, and business stakeholders to understand data requirements.
• Establish CI/CD for Databricks notebooks and repositories (DBFS, Repos, Git integration).
• Monitor and troubleshoot pipeline failures, data drift, and performance bottlenecks.
Qualifications:
Required:
• Databricks Core: Lakehouse, Delta Lake, Unity Catalog, DLT, Workflows
• Languages: PySpark, Python, SQL
• Data Modeling: Star schema, slowly changing dimensions (SCD), medallion architecture
• Data Quality: Validation, anomaly detection, DQ rules implementation
• Catalog & Governance: Unity Catalog, Hive Metastore, lineage, access controls
• Experience: 5+ years in data engineering; 2+ years hands-on with Databricks
Preferred:
• Databricks Certification (e.g., Data Engineer Professional or Associate)
• Experience with cloud platforms (AWS S3/Glue, Azure Data Lake/ADF, GCP)
• Knowledge of streaming (Kafka, Kinesis, Structured Streaming)
• Familiarity with DBT, Terraform, or MLflow
• Strong analytical and problem-solving abilities
• Excellent communication for technical and non-technical audiences
• Self-starter capable of leading architectural decisions
Company:
Tredence is a global data science solutions provider focused on solving the last mile problem in AI. Founded in 2013, the company is headquartered in San Jose, USA, with a team of 1001-5000 employees. The company is currently Late Stage.