Databricks Engineer Jobs in Fort Mill, SC (NOW HIRING)

Position Summary:

Vallen Distribution is building a governed, production-grade AI capability on an Azure-first, Databricks-centered architecture. This is not an advisory role - it is a hands-on builder position with broad ownership across the AI layer.

The AI Platform Engineer will own the Databricks AI/ML platform layer, drive enterprise AI governance, review and remediate shadow AI solutions, and directly deliver automation use cases with business teams. You will be the internal AI expert for the organization - the first call for departments exploring automation and the technical authority on what gets built, how, and where data flows.

This is a greenfield opportunity with real scope and visibility. You'll work directly with the SVP of Data & Technology Innovation and have immediate impact on a program that is active and growing today.

Core Responsibilities

Lakehouse AI Layer (Databricks) - ~35%

• Own the AI/ML layer on Databricks: feature stores, MLflow experiment tracking and model registry, and RAG/prompt architectural standards
• Define and enforce prompt engineering standards and LLM integration patterns across internal tools
• Partner with Data Engineering to design data pipelines that feed AI/ML use cases from the Unity Catalog lakehouse
• Evaluate and implement agentic frameworks (Claude API, Databricks AI agents) for internal automation

Automation Delivery - ~25%

• Directly build and deliver 2-3 automation use cases per year with business teams (HR, Legal, customer service, operations)
• Own full delivery lifecycle: scoping, design, build, testing, and handoff to platform operations
• Produce well-documented, governed solutions - not one-off scripts

AI Governance & Shadow Solution Review - ~20%

• Serve as the first-filter reviewer for all AI tools and platforms proposed for use at Vallen
• Partner with Security and Infrastructure to assess data classification risk, vendor posture, and integration risk before production deployment
• Review user-built solutions (Claude Desktop, Copilot, Cowork, and similar tools) and determine when a productivity workflow has crossed into enterprise scope - then lead the governed rebuild
• Maintain Vallen's enterprise AI acceptable use policy, data classification guardrails, and platform-tier standards

Use Case Evangelism & Business Partnering - ~20%

• Embed with departments to surface and prioritize automation opportunities
• Maintain a scored use case backlog; facilitate structured discovery sessions with business stakeholders
• Serve as the internal AI resource teams engage before going external

Job Qualifications:

• Bachelor's degree in Computer Science, Information Technology, or a related field
• Experience in AI/ML engineering, data engineering, or a closely related technical role
• Hands-on experience with Databricks - MLflow, notebooks, Unity Catalog, and Python/SQL workflows; or equivalent lakehouse platform experience with demonstrated ability to ramp quickly
• Practical experience building with LLMs: prompt engineering, RAG pipelines, or API integration- production or project-level experience counts
• Strong Python skills; ability to own full solution delivery from prototype to production
• Solid grounding in Azure: Azure OpenAI, Azure Data Lake Storage, Azure Key Vault, and core networking/security concepts
• Demonstrated ability to work across technical and non-technical stakeholders - you can explain what you're building and why it matters
• Experience reviewing or documenting AI/ML solutions for risk, data sensitivity, or governance considerations

Preferred Qualifications:

• Experience in distribution, supply chain, or industrial B2B environments
• Familiarity with agentic frameworks: Databricks AI Agent Framework, LangChain, AutoGen
• Exposure to enterprise AI governance concepts: acceptable use policies, data classification tiers, model risk review
• Experience with Azure DevOps (ADO), CI/CD pipelines)
• Familiarity with Power BI, Databricks Genie, or other BI/AI consumption layers
• Working knowledge of MDM, ERP data structures, or multi-system data environments

Work Environment:(Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.):

• Long periods of time working on a computer and performing repetitive key-boarding activities.

Physical Demands:(Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.):

• While performing the duties of this job, the employee is regularly required to talk and hear. The employee frequently is required to sit. The employee is occasionally required to stand and walk. The employee may be required to occasionally lift and/or move up to 10 pounds. Specific vision abilities required by this job include close vision, and ability to adjust focus.

#LI-REMOTE