Remote Microsoft Data Engineer Jobs in Edison, NJ

We're seeking an experienced Data Engineer to join our engineering team. We are looking for someone who enjoys solving interesting problems and wants to work with a small team of talented engineers on a product used by thousands of publishers.

Position Title & Overview

The Data & Analytics team consists of data analysts, data engineers and analytics engineers working to build the most effective platform and tools to help uncover opportunities and make decisions with data here. We partner with Product, Support, Ad Operations and other teams within the Engineering department to understand behavior, develop accurate predictors and build solutions that provide the best internal and external experience possible.

A Data Engineer will help build and maintain our data infrastructure. Building scalable data pipelines, managing transformation processes, and ensuring data quality and security at all steps along the way. This will include writing and maintaining code in Python and SQL, developing on AWS, and selecting and using third-party tools like Rundeck, Metabase, and others to round out the environment. You will be involved in decisions around tool selection and coding standards.

Our current data engineering toolkit consists of custom Python data pipelines, AWS infrastructure including Kinesis pipelines, Rundeck scheduling converting to Prefect orchestration, dbt for transformation and Snowflake as our data warehouse platform. We are open to new tools and expect this position to be a part of deciding the direction we take.

Essential Responsibilities

• Create data pipelines that make data available for analytic and application use cases
• Develop self-healing, resilient processes that do not require constant care and feeding to run smoothly
• Create meaningful data quality notifications with clear actions for interested parties including other internal teams and other members of the data and analytics team
• Leading projects from a technical standpoint, creating project Technical Design Documents
• Support data analysts and analytics engineers ability to meet the needs of the organization
• Participate in code reviews, understanding coding standards, ensuring test coverage and being aware of best practices
• Build or implement tooling around data quality, governance and lineage, in the dbt framework and Snowflake but external to that as needed
• Provide next level support when data issues are discovered and communicated by the data analysts
• Work with data analysts and analytics engineers to standardize transformation logic in the dbt layer for consistency and ease of exploration by end users
• Enable analytics engineers and data analysts by providing data modeling guidance, query optimization and aggregation advice

Must Have:

• 3 years of experience in a data engineering role
• Strong Python skills (Understands tradeoffs, optimization, etc)
• Strong SQL skills (CTEs, window functions, optimization)
• Experience working in cloud environments (AWS preferred, GCS, Azure)
• Experience managing complex dbt environments with hundreds or more flows
• An understanding of how to best structure data to enable internal and external facing analytics
• Familiarity with calling APIs to retrieve data (Authentication flows, filters, limits, pagination)
• Experience working with DevOps to deploy, scale and monitor data infrastructure
• Scheduler experience either traditional or DAG based
• Experience using LM-powered tools for code generation, documentation, and architectural diagramming
• Comfortable working with multi-TB cloud data warehouses (Snowflake preferred, Redshift, Big Query)
• Experience with other DBMS systems (Postgres in particular)
• Ability to travel up to approx 15%
• Experience building enterprise-facing data products (APIs, data shares, custom client reporting solutions), not just internal analytics pipelines.

Nice to haves:

• Experience with web analysis such as creating data structures that support product funnels, user behavior, and decision path analysis
• Understanding of Snowflake external stages, file formats and snowpipe
• Experience managing the semantic layer in either dbt or Snowflake
• Experience with orchestration tools particularly across different technologies and stacks
• Knowledge of Ad Tech, Google Ad Manager and all of its fun quirks (so fun)
• The ability to make your teammates laugh (it wouldn't hurt if you were fun to work with is what I'm saying)
• Familiarity with event tracking systems (Snowplow, etc)
• Experience with one or more major BI tools (Omni, Sigma, Metabase, etc.)