Azure Data Engineering Architect

Indiglobe IT Solutions Pvt. Ltd.

Brentwood, TN

Expired: 25 days ago Applications are no longer accepted.

Full-Time

Job Description

Description

Lead Azure Data Engineer Architect role is to strategically design, develop and implement Azure Cloud based Data Warehouse, Data Lake, Data Lakehouse, Data Marts BI solution using modern data engineering techniques and advanced analytics methods to provide information insights to end users. The role requires modeling, collecting, harmonizing, aggregating, storing and reconciling data from variety of data sources for our petroleum refineries IT/OT technology platform including time series Database, SCADA data, batch, real time, enterprise applications, Rest API’s , streaming IOT data.& The role will help design and build data pipelines, streams, data warehouse, data lake, reporting application, data generators and a whole range of tools to provide information and insight.

The ideal candidate will be responsible for designing, developing, and maintaining scalable data processing pipelines in a cloud-based environment. The candidate should have a deep understanding of Azure ADF, Databricks, streaming analytics, data engineering best practices, and a proven track record of delivering high-quality real time solutions.

Responsibilities

Strategy Planning

Assess and cultivate long-term strategic goals for data warehouse, data lake, BI& advance analytics development in conjunction with end users, managers, clients, and other stakeholders.
Coordinate and work with other technical staff to develop Azure Data BI architecture, coding standards, and quality assurance policies and procedures.
Promote a culture of self-serve data analytics by minimizing technical barriers to data access and understanding.
Is responsible for data collaboration and communication with various stakeholders.

Operational Management

Lead the integration efforts for Azure data warehouse, data lakehouse - sourcing data from disparate enterprise systems including applications both on premises cloud, IOT devices, Data Service (Rest API’s), time-series database, semi-structure, unstructured data external data sources.
Organize and lead projects in the implementation and use of BI software tools and systems.=
Design, code, test, and document all new or modified BI systems, applications, and programs.
Develop the semantic layer, metadata, reports, and report definitions.
Assist in the design of databases to ensure interoperability with DW BI solutions.
Produce ELT/ETL design guidelines to ensure a manageable ELT/ETL infrastructure for the analytics system.
Participate in design discussions about new features and ensure real-time technology integrates seamlessly with other pieces of the platform
Applying technical knowledge to architect solutions that meet business needs, AI/ML roadmaps, and contributing to architectures that enable the ability to scale to support additional modelling use cases (e.g. Azure ML, Azure Databricks, Azure EventHub/IoT Hub, etc)
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Work with project managers to ensure that data entry, retrieval, change, and delete functions meet business requirements for project completion.
Act as the primary owner and maintainer of new and existing Lakehouse/DW/BI data models
Design, implement, and maintain CI/CD pipelines using Azure DevOps to integrate analytical solutions into cloud based custom developed applications
Support Cloud strategy team to integrate analytical capabilities into an overall cloud architecture and business case development
Facilitate Master/ Reference Data Management and overall Data Governance process
Conduct job duties and responsibilities according to the organization’s business systems development methodology and its Systems Development Life Cycle (SDLC) methodology.

Design and deliver end-user training and training materials; provide technical support as necessary.

Troubleshoot Data Integration BI tools, systems, and software; performance-tune these applications as necessary.
Defines and lead proof of concept activity associated with cloud data analytics technology assessment.
Act as evangelist for Azure Cloud data analytics benefits across the organization; increase usage to relevant departments.
Act as a subject matter expert on all things cloud data architecture and pipeline development

Position Requirements

Knowledge Experience

Minimum of 12 to 15 years of Data Engineering, Data Warehouse/ Business intelligence with hands-on experience leading and delivering multiple full life cycle implementations.
Comprehensive data warehouse analysis and design experience, with full knowledge of data warehouse methodologies and data modeling.
Hands-on experience deep understanding in designing, development and implementation of Azure Data Analytics architecture, data modeling, transformations, and development.
In-depth knowledge and minimum 5 to 7 years’ experience developing implementing data engineering solutions using Azure services for& Data Warehouse, Data Lakehouse, Advance analytics- Primarily- Azure Data Factory, ADLS, Azure Databricks, Azure Functions, Logic Apps, IoT Hub, Event Hub Azure SQL
Experience with Data pipeline and workflow management tools Azure DevOps processes.
Strong proficiency in programming languages such as Python, PySpark, or Scala
End to End Implementations expertise in& Azure Data Warehouse Data Lakehouse
Solid understanding of database structure principles, Master and Reference Data management
Experience expertise in optimizing the ETL/ELT data pipeline technology options to minimize the on-going operational cost of Azure data engineering storage solutions.
8+ years experience& in agile Data warehouse development methodology.
8+ years’ experience designing and developing dimensional models for data warehousing
3+ years’ experience in MSBI suite primarily SSIS
3+ years of implementing operational data model is plus not essential
3+ years of experience with big data batch/streaming processing.
5+ years’ experience using Pyspark, Python or Scala
Experience delivering reporting solution using visualization tools like PowerBI
Experience in implementing SAP data analytics solution in Azure is plus not essential
Experience using data modeling tool like ERWin
Ability to lead and work with remote offshore teams.
Experience integrating analyzing data from time-series databases, SCADA data, IOT, voice to text, images videos for creating Industrial Intelligence solution.
Knowledge/expertise on Azure ML/AI predictive modeling capabilities is a plus
Experience using proven methods to solve business problems using Azure Data and Analytics services in combination with building data pipelines, data streams and system integration
Hands-on experience with analytics and big data technologies within Microsoft Azure Cloud platform in tools such as Azure Machine Learning Azure Cognitive Services is plus not essential
Experience with data profiling, cataloguing, and data mapping for technical design using a use case-based approach that drives the construction of technical data flows
Experience with Azure Administration Security (Connection setup, User setup, Object security) is plus
Understanding of statistical and machine learning modeling with experience applying these modeling techniques to business problems is a plus not essential.
Ability to set up data and experimental platforms using Azure cloud service
Ability to work independently with business stakeholders to collect and understand requirements for reporting analytics.
Full project management and software development life cycle experience.
Experience with data process flowcharting techniques.
Oil Gas industry experience preferred but not essential.
Experience knowledge of SAP S4 and SAP BW4HANA is a plus not essential.
Excellent quantitative, analytical mathematical skills is a plus
Strong understanding of the organization’s goals and objectives

Formal Education Certification

Bachelor’s degree in computer science, information systems, or a related field, or equivalent work experience.
Certifications in Azure Data Engineering and Analytics space is a plus

Personal Attributes