Job Title: Azure Databricks Architect
Location: Remote with (50% travel)
Role SummaryWe are seeking Senior Data Platform Engineers to join our CSO team. This is a dual-domain environment spanning two fully segregated platforms built on Azure Databricks and the Medallion Architecture: the CSO Security Data Platform and the SOX Application Data Platform (fully segregated to satisfy Sarbanes-Oxley requirements). Engineers are the backbone of both platforms — ensuring data quality, enabling security consumers, onboarding new sources, maintaining pipelines, and supporting SOX audits.
Required Qualification:Experience
• 5+ years of data engineering experience with at least 2 years on Azure Databricks or equivalent cloud lakehouse platforms.
• Hands-on experience with Medallion Architecture pipelines, Delta Lake (MERGE, OPTIMIZE, VACUUM, Z-ORDER, schema enforcement), and Databricks Structured Streaming.
• Familiarity with SOX compliance, IT General Controls (ITGC), or regulated-environment data engineering; experience supporting audits or evidence collection.
Technical Skills
Area
Technologies
Languages
Python (PySpark), SQL; Scala a plus
Cloud
Azure (ADLS Gen2, Azure Monitor, Log Analytics, Event Hubs, APIM, Azure DevOps)
Data Platform
Databricks (Unity Catalog, Auto Loader, Structured Streaming, Workflows, Genie, AI/BI Dashboards, MLflow)
Governance
Unity Catalog (RBAC, column/row security, lineage), MS Purview (auto-classification, sensitivity labels)
Integration
Azure APIM, Delta Sharing, Databricks SQL API, SIEM/SOAR connector patterns
BI & Reporting
Databricks AI/BI Dashboards, Power BI
DevOps
Git, CI/CD via Azure DevOps or GitHub Actions
Security Domain Knowledge
• Familiarity with OCSF (Open Cybersecurity Schema Framework) or willingness to develop deep expertise rapidly.
• Understanding of security log source types (EDR, firewall, IAM/PAM, CASB) and SOC/SIEM/SOAR workflows; awareness of MITRE ATT&CK framework.
Preferred Qualifications
• OCSF normalization implementation across multiple security log sources.
• Experience with Azure API Management (APIM) for data platform API governance.
• UEBA, threat scoring, or ML-backed security analytics (MLflow, Databricks Feature Store, Mosaic AI).
• SOX audit participation — auditor walkthroughs, evidence packages, IT General Controls testing.
• Certifications: Databricks Certified Data Engineer Associate/Professional, Azure Data Engineer Associate (DP-203), or equivalent.
• Graph-based correlation (Databricks GraphX); Delta Sharing.
Core Competencies
Competency
Why It Matters
Ownership & Reliability
Numerous sources, high volume TB/day — you own what doesn''t run.
Audit Mindset
Audits mean evidence gaps have real regulatory consequences.
Security Domain Curiosity
The better you understand analyst needs, the better the platform serves them.
Stakeholder Communication
You bridge data engineering, SOC, compliance, and auditors.
Structured Problem-Solving
Quarantine spikes and pipeline failures require root-cause discipline.
Documentation Discipline
Runbooks, audit evidence, and onboarding guides are first-class deliverables.
Compliance Awareness
SOX platform changes carry regulatory risk; process adherence is non-negotiable.
Platform & Tooling Landscape
Area
Technologies
Cloud Infrastructure
Microsoft Azure, ADLS Gen2, Azure Monitor, Log Analytics, Event Hubs, APIM
Data Platform
Databricks (Unity Catalog, Structured Streaming, Auto Loader, Workflows, Genie, AI/BI Dashboards)
Storage / Format
Delta Lake, JSON + GZip (Landing Zone), ADLS Gen2 Hot & Archive tiers
Schema Standard
OCSF + custom.* namespace (Security); segregated schemas (SOX)
Governance
Unity Catalog, MS Purview (auto-classification, sensitivity labels, federated discovery)
Integration
Azure APIM, Delta Sharing, Event Hubs, Databricks SQL API
BI & Analytics
Databricks AI/BI Dashboards, Databricks Genie, Power BI
AI & ML
MLflow, Databricks Feature Store, Mosaic AI (roadmap)
DevOps & CI/CD
Azure DevOps / GitHub Actions, Git
Scale
Numerous log sources · High daily volume in TBs/day raw
Educational Qualifications:
· Required - Bachelor’s degree in Computer Science, Information Technology, Computer Engineering or closely related or equivalent.
· Preferred - Master’s degree in Management Information Systems (MIS), Computer Science, Big Data or Analytics or equivalent.
Travel:
· Open to travel based-up on the nature of the engagement.
Thanks & Regards
Srikanth Donkani
Resource Manager
(w):
(E):
2260 Haggerty Road, Suite 285 Northville, MI 48167
Equal Employment Opportunity
Reliable Software employment does not discriminate on the basis of race, religion, gender, sexual orientation, age or any other basis as covered by federal, state, or local law.
Employment decisions are based solely on qualifications, merit and business needs.