Freelance Data Pipeline Engineer Jobs in Raleigh, NC

Job Summary:
VARITE INC is a company specializing in data solutions, and they are seeking a Data Engineer. The role involves designing and implementing scalable data architectures, developing complex data pipelines, and ensuring data governance and quality management.
Responsibilities:
• Data Architecture and Strategy - Design and implement scalable, efficient data architectures. Lead the development of data strategy aligned with business objectives. Evaluate and integrate new technologies to enhance data capabilities
• Hands-on Data Pipeline Development - Implement complex data pipelines for real-time and batch processing by ingesting, transforming data from staging layer to domain to consumption layers using Data Vault 2.0 models. Optimize data flows for high-volume, high-velocity data environments. Develop advanced ETL processes for diverse data sources.
• Data Governance, Security & Quality Management - Establish and enforce data governance policies, best practices and implement data quality checks, standardization. Enforce Snowflake RBAC/ABAC, network policies/rules, masking/row access policies, tags, data classification, and least-privilege role hierarchies. Operationalize audit-ready controls (change management, approvals, runbooks, separation of duties, evidence capture).
• Data warehousing & modeling - Build performant Snowflake objects (databases, tables, streams, tasks, materialized views, warehouses) and optimize clustering/micro-partitioning
• Performance Optimization and Troubleshooting - Analyze and optimize system performance for large-scale data operations. Troubleshoot complex data issues and implement robust solutions
• Testing & Automation – Write unit test cases, validate the data integrity & consistency requirements, build automated data pipelines using GitLab, Github, CICD tools.
• Code Deployment & Release Management - Adopt release management processes to promote code deployment to various environments including production, disaster recovery, and support activities.
• Mentorship and Knowledge Sharing - Mentor junior data engineers and provide technical guidance. Contribute to the development of best practices and standards. Collaborate with cross-functional teams to drive data literacy
Qualifications:
Required:
• Strong hands-on experience in building robust metadata-driven, automated, secured, governed data pipeline solutions leveraging modern cloud data technologies, tools for large data platforms.
• Strong experience designing, building, and operating secure, audited, and cost-efficient data pipelines using medallion architecture on Snowflake—from raw ingestion to Data Vault 2.0 models and onward to business-friendly consumption layers (mart/semantic).
• Strong hands-on experience building, managing, packaging, deploying DBT Cloud data pipelines with advanced macros, SQL; automating testing strategies; and maintaining lineage.
• Strong experience in Snowflake architecture, data classification, tagging & masking automation, Universal Search, performance tuning, security and cost optimization.
• Strong hands-on experience building snowflake objects (tables, streams, tasks, materialized views) and optimize clustering/micro-partitioning.
• Strong experience using Astronomer Airflow for orchestrating, scheduling data pipelines; and Terraform (with HashiCorp practices) for IaaC automation of pipelines.
• Strong hands-on experience building data pipelines, data integrity solutions across multiple data sources and targets like SQL Server, Oracle, Mainframe-DB2, files, Snowflake.
• Experience using S3, Lambda, SQS, SNS, Glue, RDS AWS services.
• Excellent proficiency in Python, Pyspark, advanced SQL for ingestion frameworks and automation.
• Experience in implementing logging, monitoring, alerting with Snowflake using ACCOUNT_USAGE/INFORMATION_SCHEMA, event tables, and integrating with Splunk, Datadog.
• Strong experience designing, implementing tokenization, RBAC, masking policies, dynamic & conditional masking, data access controls across Snowflake and supporting systems.
• Strong experience in adopting release management guidelines, code deployment to various environments, implementing disaster recovery strategies, leading production activities.
• Experience in SCIM/SSO (Okta), MFA, ephemeral credentials, service-account hardening.
• Must have one or more certifications in the relevant technology fields.
• Team Player: Support peers, team, and department management.
• Communication: Excellent verbal, written, and interpersonal communication skills.
• Problem Solving: Excellent problem-solving skills, incident management, root cause analysis, and proactive solutions to improve quality.
• Partnership and Collaboration: Develop and maintain partnership with business and IT stakeholders.
• Attention to Detail: Ensure accuracy and thoroughness in all tasks.
• Experience Required: 8-10 years
Preferred:
• Internal dbt packages, reusable Terraform modules, project templates experience is a plus.
• Experience with Iceberg/external tables; Kafka or event-driven ingestion patterns is a plus.
• Experience FinOps for Snowflake, cost charge-back/show-back, warehouse auto-tuning is a plus.
• Power BI/Looker metric-layer design; semantic modeling concepts experience is a plus.
• Great Expectations, Monte Carlo/Anomalo/Atlan/Collibra/BigID integrations experience is a plus.
• DBT mesh, custom materializations, Slim CI, state comparison experience is a plus.
• Financial banking experience is a plus.
Company:
Varite is an information technology company providing networking, business analytics and cloud services Founded in 2000, the company is headquartered in San Jose, USA, with a team of 1001-5000 employees. The company is currently Late Stage.