Manager Databricks Data Engineer Jobs in North Charleston, SC

KORE1, a nationwide provider of staffing and recruiting solutions, has an immediate opening for a Data Engineer that is based in Charleston, SC. Remote /hybrid flexibility. 

Position Overview
We are seeking a highly capable Data Engineer to design and build the data infrastructure that powers investment operations, reporting, and business intelligence capabilities. This is a foundational role within a firm actively modernizing its data architecture by replacing fragmented automations with scalable, production-grade pipelines and establishing a centralized, reliable source of truth across multiple systems.
The Data Engineer will own the end-to-end data stack, including ingestion, transformation, modeling, data quality, and governance. This role requires deep technical expertise along with the ability to navigate a complex, multi-system financial data environment.
This position will play a central role in supporting the organization's evolving AI and analytics initiatives. The ideal candidate will help structure, govern, and optimize warehouse data to support future AI-enabled workflows and advanced reporting capabilities.
The Data Engineer will work closely with cross-functional teams including Operations, IT, business stakeholders, and external technology partners to ensure the data platform evolves in alignment with operational and strategic needs.
Key Responsibilities

Architecture & Infrastructure
* Evaluate and recommend scalable data architecture solutions, including warehouse and lakehouse strategies.
* Build and maintain a centralized cloud data warehouse environment.
* Design and maintain scalable ingestion pipelines across multiple operational and financial systems.
* Develop data structures and governance models that support analytics, reporting, and future AI-driven workflows.
* Ensure the data platform is designed with long-term scalability, security, and accessibility in mind.
Data Modeling & Transformation
* Design and implement canonical data models across financial and operational datasets.
* Write and optimize complex SQL queries, transformations, and performance-tuned data workflows.
* Replace manual or lightweight automations with production-grade ETL/ELT pipelines using Python and cloud-native tooling.
Integrations & APIs
* Build and maintain REST API integrations across third-party platforms and internal systems.
* Develop Python-based ingestion and transformation processes.
* Support orchestration and storage within a cloud environment using modern data engineering best practices.
Data Quality, Governance & Security
* Implement validation frameworks, deduplication logic, monitoring, and exception handling processes.
* Configure role-based access controls and maintain auditability across the data environment.
* Document and maintain end-to-end data lineage and governance standards.
* Support compliance and security requirements for sensitive financial and operational data.
Reporting & Documentation
* Deliver reporting-ready datasets supporting portfolio analytics, operational reporting, and business intelligence initiatives.
* Support BI tool integrations and semantic layer development for technical and non-technical users.
* Create and maintain documentation for architecture decisions, pipelines, and data dictionaries.
Cross-Functional Collaboration
* Partner closely with Operations, IT, and leadership teams to align data infrastructure with business goals.
* Participate in ongoing technology and systems planning initiatives.
* Act as a key liaison between technical and business stakeholders to ensure data solutions meet organizational needs.
Technical Requirements
* Advanced Python development for data engineering and automation
* Advanced SQL skills including performance optimization and complex transformations
* Strong experience with Snowflake or similar cloud data warehouse platforms
* Experience designing scalable data models and ETL/ELT pipelines
* REST API integration experience including authentication, pagination, and error handling
* Cloud platform experience, preferably AWS
* Strong understanding of data quality, governance, and access controls
* Ability to work cross-functionally in fast-paced environments
Preferred Skills
* Experience supporting AI or LLM-enabled workflows
* Financial services, private equity, asset management, or investment operations experience
* Familiarity with CRM, fund administration, or portfolio management platforms
* Experience with BI tools such as Tableau, Looker, or Sigma
Required Qualifications
* Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical field
* 5+ years of experience in data engineering
* Strong production experience with Python and SQL
* Experience building scalable cloud-based data platforms
* Strong documentation and communication skills
Preferred
* Experience working with financial datasets and investment operations
* Exposure to AI-driven analytics or agent-based workflows
* Experience implementing enterprise reporting and business intelligence solutions

Core Competencies
* Ownership and accountability
* Strong analytical and systems thinking
* Cross-functional collaboration
* Clear communication with technical and non-technical stakeholders
* Adaptability in fast-paced environments
* High standards for governance, accuracy, and compliance
Location & Work Environment
This position is preferably based in the Southeast United States, though remote candidates may be considered for the right fit.
Compensation & Benefits
* Competitive compensation package
* Performance-based incentives
* Comprehensive health and retirement benefits
* High visibility with leadership and strategic initiatives
* Opportunity to help build foundational infrastructure within a growing organization

Compensation depends on experience but is typically 145-160k

ABOUT KORE1
Specializing in professional and technical recruiting, KORE1 is committed to supporting top IT, Engineering, Creative, Scientific, Accounting and Finance professionals in their career paths. We build deep relationships with leading companies, connecting them to exceptional talent every day. With extensive industry expertise and unmatched opportunities, our goal is to provide a unique experience for our contractors and consultants as they prepare for their next role. We are passionate about matching the right people with the right companies.