Data Engineer
Data Engineer Address: 301 Lindenwood Dr Ste 330, Malvern, Pennsylvania Remote: 3 days onsite Hours: 8-5 Position Type: FULL-TIME
KEY RESPONSIBILITIES
- Data Architecture & Pipeline Development
- Design, construct, and maintain robust data pipelines using Azure Data Factory, Azure Synapse, and Azure Databricks.
- Develop ETL/ELT processes to ingest, transform, and store structured and unstructured data from various sources.
- Ensure data pipelines are scalable, secure, and optimized for performance.
- Document architecture, database, data flow, and algorithm details.
- Database Management
- Manage and optimize relational databases using Azure SQL and flat files in cloud native, Azure environment.
- Write complex SQL queries for data extraction, transformation, and analysis.
- Implement indexing, partitioning, and performance tuning strategies.
- Establish and maintain processes aligned to master data management; data quality, security, storage, and database tuning throughout the data lifecycle.
- Machine Learning Integration
- Collaborate with Data Analyst to operationalize ML models using Azure Machine Learning.
- Support feature engineering, model versioning, and deployment pipelines.
- Utilize Azure ML tools like AutoML, HyperDrive, and Model Registry for experimentation and tracking.
- Cloud Infrastructure & Automation
- Leverage Azure Functions, Logic Apps, and Event Hubs for event-driven data processing.
- Automate data workflows and monitoring using CI/CD pipelines and scripting (PowerShell, PowerAutomate).
- Ensure compliance with security and governance policies across all data assets.
- Collaboration & Communication
- Work closely with cross-functional teams including product, engineering, testing, and information security.
- Translate business requirements into technical specifications and data solutions.
- Document data flows, architecture, and operational procedures.
SKILLS AND EXPERIENCE
Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field. 5+ years of experience in data engineering roles. Proficiency in SQL and experience with Azure data services, (Synapse, Azure ML). Familiarity with machine learning concepts and model deployment practices. Strong problem-solving skills and attention to detail. Excellent communication and collaboration abilities.
PREFERRED QUALIFICATIONS
Experience with Azure DevOps, GitHub Actions, or other CI/CD tools. Working knowledge of .Net. Knowledge of data governance frameworks and SOC 2 compliance. Exposure to financial services, credit union, banking data environments.