Databricks Delta Lake Jobs (NOW HIRING)

Job Summary:
State Street is seeking a Senior Data Lakehouse Architect to design and lead the build-out of a Legal Data Lakehouse platform on AWS and Databricks. This role will drive the architecture, engineering, and governance of scalable, secure, and compliant data capabilities supporting legal operations, contract intelligence, eDiscovery, and AI/ML use cases.
Responsibilities:
• Define and implement the end-to-end Legal Data Lakehouse architecture using Databricks (Delta Lake, Unity Catalog, Workflows) on AWS
• Design multi-layered data architecture (Bronze, Silver, Gold) to support:
• Contract metadata and document ingestion
• Legal matter management data
• eDiscovery datasets
• External regulatory and compliance feeds
• Establish scalable ingestion frameworks (batch and streaming) for structured and unstructured legal data (PDFs, contracts, emails)
• Lead development of ETL/ELT pipelines using Databricks, Spark, and Python/SQL
• Integrate with enterprise platforms, including:
• Contract lifecycle management systems
• AI platforms and LLM pipelines
• Document repositories and enterprise content systems
• Design patterns for extracting structured data from unstructured legal documents and persisting into Delta Lake
• Enable downstream integration with enterprise data platforms, analytics tools, and AI/ML pipelines
• Implement data governance frameworks using Databricks Unity Catalog and AWS-native controls (IAM, KMS)
• Establish:
• Fine-grained access controls (row/column-level security)
• Data lineage and auditability
• Ensure compliance with:
• Data privacy regulations (e.g., GDPR)
• Internal security and audit requirements
• Partner with IAM teams to integrate with enterprise identity providers (e.g., Entra ID / Azure AD)
• Architect data models supporting:
• Contract analytics, clause extraction, and obligation tracking
• Legal AI use cases (contract review, litigation insights, compliance monitoring, legal spend analytics)
• Design search and retrieval architectures (RAG) for enterprise legal knowledge bases
• Enable entity extraction and knowledge graph frameworks
• Integrate with LLM/GenAI platforms to support capabilities such as document summarization, Q&A, and workflow automation
• Establish CI/CD pipelines and infrastructure-as-code (Terraform, Git-based workflows)
• Define standards for:
• Code quality and versioning
• Environment promotion (Dev / QA / Prod)
• Implement observability and alerting for platform health and reliability
• Partner with Legal and Technology leadership to define platform roadmap and priorities
• Provide architectural governance and design oversight
• Mentor data engineers and platform teams
• Translate business and legal requirements into scalable, enterprise-grade solutions
• Operate within a federated data and platform model, collaborating across engineering, security, and domain teams
Qualifications:
Required:
• 10+ years of experience in data architecture, engineering, or analytics platforms
• 5+ years of hands-on experience with Databricks and Apache Spark
• Strong experience with AWS-based data platforms
• Expertise in data governance, security, and compliance in regulated environments
• Experience working with unstructured data and NLP/document processing pipelines
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related technical discipline
• Relevant certifications strongly preferred: Databricks Certified Data Engineer / Architect, AWS Certified Solutions Architect (Associate or Professional)
• Define and implement the end-to-end Legal Data Lakehouse architecture using Databricks (Delta Lake, Unity Catalog, Workflows) on AWS
• Design multi-layered data architecture (Bronze, Silver, Gold) to support contract metadata and document ingestion, legal matter management data, eDiscovery datasets, external regulatory and compliance feeds
• Establish scalable ingestion frameworks (batch and streaming) for structured and unstructured legal data (PDFs, contracts, emails)
• Lead development of ETL/ELT pipelines using Databricks, Spark, and Python/SQL
• Integrate with enterprise platforms, including contract lifecycle management systems, AI platforms and LLM pipelines, document repositories and enterprise content systems
• Design patterns for extracting structured data from unstructured legal documents and persisting into Delta Lake
• Enable downstream integration with enterprise data platforms, analytics tools, and AI/ML pipelines
• Implement data governance frameworks using Databricks Unity Catalog and AWS-native controls (IAM, KMS)
• Establish fine-grained access controls (row/column-level security), data lineage and auditability
• Ensure compliance with data privacy regulations (e.g., GDPR) and internal security and audit requirements
• Partner with IAM teams to integrate with enterprise identity providers (e.g., Entra ID / Azure AD)
• Architect data models supporting contract analytics, clause extraction, and obligation tracking, legal AI use cases (contract review, litigation insights, compliance monitoring, legal spend analytics)
• Design search and retrieval architectures (RAG) for enterprise legal knowledge bases
• Enable entity extraction and knowledge graph frameworks
• Integrate with LLM/GenAI platforms to support capabilities such as document summarization, Q&A, and workflow automation
• Establish CI/CD pipelines and infrastructure-as-code (Terraform, Git-based workflows)
• Define standards for code quality and versioning, environment promotion (Dev / QA / Prod)
• Implement observability and alerting for platform health and reliability
• Partner with Legal and Technology leadership to define platform roadmap and priorities
• Provide architectural governance and design oversight
• Mentor data engineers and platform teams
• Translate business and legal requirements into scalable, enterprise-grade solutions
• Operate within a federated data and platform model, collaborating across engineering, security, and domain teams
Preferred:
• Strong hands-on experience with the Databricks Lakehouse platform, including Delta Lake, Unity Catalog, Workflows, and MLflow
• Deep expertise in AWS data platform services, including S3, Glue, EMR, Lambda, Redshift, and IAM
• Proven experience architecting and delivering enterprise-scale data lakehouse solutions on AWS using Databricks
• Advanced proficiency in Apache Spark (PySpark/Scala), SQL, and Python
• Strong understanding of data governance and security, including access controls, metadata management, and encryption (KMS, CMK/BYOK)
• Experience building end-to-end data pipelines (batch and streaming) and supporting AI/ML workloads within a lakehouse architecture
• Experience in Legal, Compliance, Financial Services, or other regulated industries
• Understanding of legal data constructs, including contracts, clauses, obligations, and matters
• Experience supporting legal AI use cases (contract analytics, document summarization, compliance monitoring)
• Experience handling sensitive data in highly regulated, audit-driven environments
Company:
State Street offers a range of financial services, including investment management, research and trading, as well as asset management. Founded in 1792, the company is headquartered in Boston, USA, with a team of 10001+ employees. The company is currently Late Stage.