About BayRock Labs
At BayRock Labs, we pioneer innovative tech solutions that drive business transformation. As a leading product engineering firm based in Silicon Valley, we provide full-cycle product development, leveraging cutting-edge technologies in AI, ML, and data analytics. Our collaborative, inclusive culture fosters professional growth and work-life balance. Join us to work on ground-breaking projects and be part of a team that values excellence, integrity, and innovation. Together, let's redefine what's possible in technology.
We are looking for a highly skilled
Senior Data Engineer / Data Architect with deep hands-on expertise in
Azure, Databricks, PySpark, Python, SQL, and modern data warehousing. This role is ideal for someone who has built large-scale data platforms, designed enterprise-grade architectures, and delivered high-quality data products in cloud environments.
Core Technical Expertise
Azure Cloud Platform
- ADLS Gen2, ADF, Synapse, Azure SQL, Key Vault, App Services
- Cloud-native architecture design, scalability, and cost optimization
- Security & governance: RBAC, Managed Identity, Private Endpoints
Databricks Engineering
- End-to-end development on Azure Databricks
- Lakehouse architecture implementation
- Delta Live Tables (DLT), Unity Catalog
- Performance tuning, cluster optimization
- CI/CD for Databricks workloads
Big Data & Processing Frameworks
- Apache Spark with advanced PySpark transformations
- Structured Streaming & batch data pipelines
- Delta Lake architecture & optimization
Programming & Querying
- Python for automation, data engineering, and orchestration
- PySpark for distributed data processing
- SQL for complex query optimization and analytics engineering
Data Architecture & Engineering
- Enterprise Data Warehouse (EDW) architecture
- Dimensional modeling (Star/Snowflake)
- Data Vault 2.0 modeling
- Metadata-driven ingestion frameworks
- Change Data Capture (CDC) implementation
- Medallion architecture (Bronze/Silver/Gold)
- Data lineage, governance, cataloging
- Master Data Management (MDM)
Data Products & Analytics Enablement
- Design and delivery of scalable data products
- Business-aligned semantic layer design
- KPI frameworks & enterprise reporting enablement
- Integration of ERP, SaaS, and operational systems
- Hybrid Lakehouse + EDW architectures
DevOps & Engineering Practices
- CI/CD pipelines (Azure DevOps, GitHub Actions, Bitbucket)
- Infrastructure as Code (Terraform, ARM templates)
- Automated testing (unit, integration, data quality)
- Monitoring, logging, and observability
- Agile/Scrum delivery
Strategic & Leadership Impact
- Principal-level solution architecture design
- Cross-functional stakeholder collaboration
- Technical roadmap planning and execution
- Guidance on data governance frameworks (GDPR, SOX)
- Cloud cost optimization strategies
- Migration from legacy EDWs (Teradata, Oracle, SQL Server) to Lakehouse
Ideal Candidate Profile
- 5-10+ years of hands-on experience in data engineering or data architecture
- Strong expertise in Azure + Databricks ecosystem
- Proven experience building scalable, secure, high-performance data platforms
- Ability to lead design discussions, influence architecture decisions, and mentor teams
- Strong problem-solving mindset and passion for modern data engineering
The pay range for this role is:
80 - 85 USD per hour (Milpitas, CA)