Job Summary:
Tekfortune is a fast-growing consulting firm specialized in staffing services for leading organizations. They are seeking a Data Engineer Quality Engineer to ensure the reliability and performance of data platforms and pipelines, focusing on validating data workflows built on AWS, Kafka, SQL, and Python.
Responsibilities:
• Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.
• Create and maintain data quality checks (nulls, duplicates, schema drift, referential integrity).
• Verify business rules and transformations using SQL-based validations.
• Design, develop, and execute test strategies for Databricks-based data pipelines and analytics workflows
• Establish data reconciliation and end-to-end traceability between source and downstream systems.
• Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).
• Validate transformations written in SQL and Python.
• Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers.
• Test reprocessing, backfills, and historical data loads.
• Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks
• Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly-once/at-least-once semantics.
• Test producer and consumer logic, serialization formats (Avro, JSON, Protobuf).
• Validate topic configurations, partitions, offsets, retention policies, and schema changes.
• Simulate and test late arrivals, duplicate events, and consumer failures.
• Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services.
• Validate IAM roles, permissions, and secure data access.
• Verify data lifecycle policies, encryption, and storage optimizations.
• Build and maintain automated data testing frameworks using Python.
• Develop reusable test utilities, fixtures, and synthetic datasets.
• Integrate data tests into CI/CD pipelines for pre-merge, scheduled, and post-deployment validation.
• Enable automated alerts for data quality failures.
• Validate pipeline performance for large-scale datasets.
• Test throughput, latency, and concurrency under peak workloads.
• Validate retry logic, error handling, idempotency, and recovery mechanisms.
• Perform soak, regression, and failover testing
• Validate data pipeline metrics, logs, and alerts using CloudWatch, Prometheus, Grafana, or equivalent tools.
• Partner with teams to define data SLAs and SLOs.
• Participate in incident response, root-cause analysis, and postmortems related to data quality issues
Qualifications:
Required:
• 7+ years of experience in QA, SDET, or Data Quality Engineering roles.
• Strong hands-on experience with SQL for complex data validation and analysis.
• Proficiency in Python for test automation and data validation.
• Experience testing data pipelines and ETL/ELT workflows.
• Hands-on experience with Kafka or other streaming platforms.
• Solid understanding of AWS data services (S3, Glue, Redshift, Lambda, Athena, etc.).
• Experience working with large datasets and distributed systems.
• Strong debugging, analytical, and problem-solving skills.
• Need a consultant with 10+ years of overall experience.
• Need a strong Data Engineer with Data Bricks experience.
• Excellent communication skills are a must.
• Must be willing to face an in-person interview.
• Must be able to work 3-4 days from office in Dallas.
Company:
Tekfortune Inc. Founded in 2014, the company is headquartered in Piscataway, USA, with a team of 51-200 employees. The company is currently Growth Stage.