Lead Data Engineer
- Expired: April 05, 2022. Applications are no longer accepted.
POSITION OVERVIEW :
The Lead Data Engineer reports to the Vice President of Software Development and is a core member of the data team. This lead engineer drives how data is stored, consumed, managed, and integrated within and across products. This role will design and implement databases, data pipelines, data search and visualizations across big datasets. They will need to have extensive experience in all phases of data modeling, from conceptualization to database optimization. This role requires expert knowledge in designing and implementing solutions with structured and unstructured data in a cloud-based environment.
This is a full time, exempt position, is fully remote, and is based in the United States.
ESSENTIAL JOB FUNCTIONS :
+ Provide hands-on technical leadership in data engineering design and implementations including data ingestion, data models, data structures, data storage, high-throughput data processing, data pipelines, and data monitoring at scale
+ Architect and build solutions to collect, transform, and process data at scale in a cloud-based environment
+ Develop and follow data engineering best practices with considerations for high data availability, computational efficiency, cost, and quality
+ Build and maintain environments processes, functionalities, and tools to improve all stages of solution development, e.g., proof of concepts, prototypes, and production
+ Create data models and database designs
+ Ensure that requirements for data retrieval are met
+ Identify areas for improvement in existing data storage and delivery systems
+ Develop and execute on long term data strategies across products
+ Creating new, experimental frameworks to collect data
+ Building tools to automate data collection
+ Develop techniques to search for patterns in large data sets, correlating similar data to find actionable results
+ Serve as a SME for all major data environments
+ Drive consistency in data usage across the organization
+ Know at a high level how data moves into and out of all enterprise systems
+ Provide guidance to keep the team current on new technologies, standards, and practices
+ Evaluate and make recommendations on new technologies, tools, and data
ANCILLARY JOB FUNCTIONS :
+ Perform database administration tasks which include database updates, snapshots, replication setup etc
+ Troubleshoot database incidents, including deadlocks, performance tuning slow-running queries, and data inaccuracies
+ Serve as a resource to outward-facing staff to determine rough LOE of new initiatives
+ Assist in concept development for new data products
+ Maintain data documentation such as ER diagrams and Data Mapping Specifications, especially in the absence of other support.
+ Assist in building and training, deploying AI Models for the data science team
+ Work well in distributed development environment across time zones
+ Mentor resources in the data engineering team
JOB COMPETENCIES/SKILL REQUIREMENTS/QUALIFICATIONS :
+ 5+ years of experience in a data engineering role
+ Deep expertise in Python, MySQL and AWS services including RDS, Dynamo DB, Redshift, Athena, S3, Lambdas, EC2, EMR, RedShift, SQS, Kafka, Data pipelines, Step Functions etc
+ In-depth understanding and hands on experience using Elastic Search
+ Strong programming skills and ability to utilize a variety of software / languages / tools, e.g.,Python, SQL, etc.
+ Experience with microservice development using Docker, ECS/Kubernetes
+ Experience with ETL tools (Eg : AWS Glue)
+ Experience with CICD deployment of data engineering solutions
+ Deep knowledge of and experience with at least one RDBMS and a Columnar database technology (e.g. MySQL, DynamoDB)
+ Experience working in real time-data and streaming solutions preferred
+ Experience in building machine learning models preferred
+ Ability to break down and understand complex business problems, define a solution and implement it using advanced quantitative methods.
+ Solid oral and written communication skills, especially around analytical concepts and methods.
+ Significant experience in data modeling and database design
+ Understanding of software development life cycle
+ Knowledge of data visualization tools such as Tableau and Splunk
**Requisition ID:** 81
TechnologyView all jobs at WMC Global