Skip to Main Content

Big Data Engineer II

DataDelivers
Schaumburg, IL
  • Posted: over a month ago
  • Full-Time
Job Description

The Big Data Engineer II primary responsibilities are to create, implement, and maintain various automated data solutions within DataDelivers big data ecosystem. The Engineer will work closely with other teammates to design optimum solutions using best practices. The Engineer is responsible to monitor and measure the data ecosystem's performance, troubleshoot as needed, and report on status and metrics.

The Big Data Engineer II also collaborates with DataDelivers data scientists and account team to creates ETL, batch, and automated processes to best suite DataDelivers and its client's needs. They proactively follow and contribute to the teams coding and programming standards and offer suggestions to how better improve them. They have a high standard for the quality of code and documentation they themselves produce.

Responsibilities:

  • Work closely with SMEs, make solution recommendations, and implement agreed upon solutions using best practices.
  • Select and integrate any Big Data tools and frameworks required to provide requested capabilities.
  • Design and implement ETL, batch, real-time, and automated processes.
  • Monitor performance and advise of any necessary improvements and changes.
  • Provide technical support to members of TS and SA team, as well as project support across client engagements.
  • Work with geographically dispersed teams, embracing Agile and DevOps strategies for themselves and others while driving adoption to enable greater technology and business value.
  • Stays current with relevant technology in order to maintain and/or improve functionality for authored applications.
  • Assume other responsibilities as requested/required.
  • Manage AWS landscape of EC2 clusters, Glue Jobs, Athena Tables, S3 data lakes and related services.
  • Acts as a subject matter expert for systems worked on. Ensures DataDelivers data solutions are using the latest versions and code base.
  • Actively listen to and work with end users to gather feedback and input, and make suggestions and solutions based on said feedback.



Qualifications:

  • (5 years of relevant experience) or (2 years of relevant experience and an advanced degree in Computer Science/IT or related field)
  • Good understanding of distributed computing principles.
  • Proficiency with Big Data frameworks such as Hadoop, Spark, MapReduce.
  • Proven experience ingesting data from multiple data sources such as REST API, SFTP flat files, Streaming data etc.
  • Proven experience with Big Data querying tools such as Athena/Presto, Hive, and Impala.
  • Proven experience with NoSQL databases, such as DynamoDB, Redshift, Databricks, etc.
  • Proven experience with various ETL techniques and frameworks, such as Flume, Glue Jobs, Step Functions.
  • Proven experience with AWS Lambda and leveraging it in various solutions such as Glue, Step Functions, CloudWatch, S3 Events, etc.
  • Extensive experience with Python scripts & libraries
  • Experience with AI and Data Science toolkits - Sagemaker, Mahout, SparkML, H2O, etc. highly desired.
  • Experience desired with Database Warehousing Design Concepts; Dimensional.
  • Modeling, Star/Snowflake Schemas, ETL/ELT, Data Marts, Analytic Playgrounds, Reporting techniques.
  • Experience working with Agile software development methodologies, namely Scrum.
  • Proven experience with team collaboration, release management, system and performance monitoring.
  • Ability to work well with people from many different disciplines and varying degrees of technical experience.
  • Excellent analytical, problem resolution, organization and time management skills.
  • Ability to handle multiple tasks at a time.


Benefits:
  • We constantly strive to achieve a strong work/life balance; we are an employee-centric culture, with room for flexibility in work location and hours where possible
  • We provide excellent benefits, including 401K, medical, dental, vision, disability, and life insurance, and generous compensated time off policy
  • We offer a competitive compensation package with opportunities for performance rewards based on company success

DataDelivers

Address

Schaumburg, IL
60195 USA

Industry

Technology

What email should the hiring manager reach you at?

By clicking the button above, I agree to the ZipRecruiter Terms of Use and acknowledge I have read the Privacy Policy, and agree to receive email job alerts.