1. Data Pipeline Development: a. Assist in the design, development, and maintenance of scalable data pipelines to process and integrate data from various sources. b. Implement ELT (Extract, Load, Transform) processes to ensure data is collected, cleaned, and stored accurately.
2. Database Management: a. Support the management and optimization of databases, including data warehousing solutions. b. Ensure data integrity, security, and accessibility.
3. Data Integration: a. Collaborate with data scientists and analysts to integrate new data sources and support data-driven projects. b. Assist in the implementation of data integration solutions and APIs.
4. Performance Optimization: a. Monitor and troubleshoot data pipeline performance, identifying and resolving bottlenecks. b. Implement performance tuning techniques to improve data processing efficiency.
5. Data Quality and Validation: a. Perform data quality checks and validation to ensure the accuracy and consistency of data. b. Develop and maintain documentation for data processes and workflows.
6. LLM experience: a. Prepare quantitative and qualitative data for easy discovery by LLMs. b. Design and manage vector search infrastructure to support Retrieval-Augmented Generation (RAG) pipelines.
7. Collaboration and Support: a. Work collaboratively with cross-functional teams to understand data needs and provide technical support. b. Contribute to the continuous improvement of data engineering practices and processes.
Qualifications: Education:
o Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field or a Master's degree in Computer Science, Information Technology, Engineering, or a related field with 3-5 years of professional experience.
Technical Skills:
o Proficiency in SQL and Python.
o Understanding of ELT processes and data pipeline development. o Experience with cloud platforms (Google Cloud, BigQuery) is a plus. o Prior experience with dbt and Air Flow is preferred. o Familiarity with CI/CD (Docker and Terraform) and concepts of data integrity. o Familiarity with data visualization tools (e.g., d3, plotly or Carto preferred).
Soft Skills:
o Strong analytical and problem-solving skills. o Excellent communication and teamwork abilities. o Detail-oriented with a commitment to accuracy