Flexible Remote Azure Data Engineer Jobs in Indiana

Data Engineer II

Job Location - Indianapolis, IN

The Data Engineer is a key technical contributor responsible for designing, building, and maintaining robust data infrastructure and pipelines that enable seamless data integration, transformation, and analysis. This role involves optimizing cloud resource usage, ensuring data quality and governance, and implementing scalable, efficient data solutions aligned with business objectives. As a member of the data services team, the Data Engineer collaboratively designs database schemas, develops ETL workflows, and ensures compliance with data privacy and regulatory standards. They proactively diagnose and resolve complex technical issues, optimize queries, and contribute to process improvements.

Status - Part-Time or Full-Time considered.

Salary - Salary for this position is competitive and will be determined based on the candidate's experience, expertise, and qualifications.

Benefits for Full-Time Employees - Paid Holidays, Paid Time Off, Employer Retirement Contributions, Health Savings Account Contributions, Health, Vision, and Dental Insurance Coverage, Profit sharing and/or annual bonuses dependent on company performance, Flexible work arrangements including remote work, Professional development, Access to life insurance benefits, short and long-Term disability insurance, and an employee assistance program

Technical Skills and Knowledge You Bring to the Role

The ideal candidate is highly skilled in Python, SQL, and cloud-based data storage technologies, with a strong focus on automation and continuous learning. They take ownership of tasks within cross-team initiatives, mentor more junior team members, and recommend innovative tools and solutions to enhance performance. This role is integral to driving the scalability, reliability, and efficiency of our clients' data systems.

• Programming Languages: Proficiency in Python and SQL for data processing and query optimization. Experience with PySpark and one or more additional languages like Scala, Java, or Bash for managing data workflows.
• Data Storage and Databases: Strong knowledge of relational databases (e.g., PostgreSQL, MySQL, Oracle). Experience with modern data warehouses such as Snowflake, Amazon Redshift, or Google BigQuery. Familiarity with data lakes (e.g., Amazon S3, Azure Data Lake) and lakehouse solutions (e.g., Delta Lake, Apache Iceberg).
• Big Data Frameworks: Hands-on experience with Apache Spark for distributed data processing, including leveraging Apache Spark through Databricks. Knowledge of Apache Kafka or similar tools for real-time data streaming.
• Cloud Platforms: Experience with cloud technologies such as AWS (S3, Glue, Redshift), Microsoft Azure (Data Factory, Synapse), or Google Cloud Platform (BigQuery, Dataflow).
• Data Governance and Security: Understanding of data governance frameworks, compliance (GDPR, HIPAA), and tools like Unity Catalog, Apache Atlas, and Great Expectations.
• Pipeline Monitoring and Optimization: Experience with monitoring tools, such as Apache Airflow, for pipeline performance. Ability to optimize and troubleshoot data pipelines for scalability and efficiency.

Key Responsibilities

In this role, you will work alongside the Delineate team to:

• Develop database schemas for moderately complex data models, optimizing for query performance. Design and implement data models utilizing concepts like dimensional modeling (Kimball) and normalized data structures (Inmon) to store data for analytical reporting in alignment with business requirements.
• Design and implement automated data validation and quality checks to ensure data accuracy, consistency, and anomaly detection. Collaborate with cross-functional teams to maintain data integrity across systems and pipelines.
• Implement and enforce data lifecycle management practices, including data retention, archiving, and deletion. Ensure policies are applied consistently across platforms.
• Contribute to data integration strategies by designing and optimizing ETL workflows to integrate diverse data sources. Implement data transformation processes to improve data usability and streamline integration.
• Develop and automate scalable data pipelines, ensuring continuous and reliable data flow. Optimize pipeline monitoring processes to quickly detect and address failures or delays.
• Enhance ETL workflow performance through optimization techniques and independently refine code to improve efficiency and resource utilization.
• Conduct thorough root cause analyses for moderately complex issues, identifying underlying problems and proposing effective solutions to prevent recurrence.
• Lead the resolution of moderately complex incidents, ensuring swift recovery and minimal disruption to operations.
• Write and optimize efficient SQL queries to improve performance on moderately complex datasets and ensure data processing efficiency.
• Apply analytical skills to independently tackle and resolve moderately complex technical challenges, delivering innovative and practical solutions.
• Write complex and modular data processing scripts in Python for performance, ensuring efficient, repeatable, and traceable data transformations and retrieval.
• Maintain version control practices (e.g., Git) while harnessing DevOps principles to automate build, test, and deployment processes, ensuring continuous integration across development and production environments.
• Conduct thorough code reviews to uphold standards and best practices. Develop scripts to automate workflows, enhancing efficiency and reducing manual errors.
• Optimize storage usage in cloud environments for cost-efficiency and performance. Maintain and improve data warehousing systems, ensuring query performance and operational reliability.
• Ensure the efficient operation of data lakes and integrate them with data processing tools. Optimize storage and processing within a data lakehouse architecture for advanced analytics.
• Evaluate and recommend new tools and technologies to improve team efficiency and workflow. Suggest process improvements and embrace changes in methodologies and tools.
• Incorporate data governance best practices within workflows, maintaining high standards of data privacy and regulatory compliance across all tasks and projects.
• Ensure security measures are consistently applied across all data systems, proactively identifying and addressing areas where additional measures are required. Implement improvements to enhance data protection.