The Consumer Services Data Engineering Team of our client is building a new data warehouse and need a talented engineer to help drive this huge milestone across the line. The data engineer will have experience sourcing in social media data into our newly designed data warehouse.
· MS/BS in Computer Science, or related technical discipline
· 5+ years of industry experience, 3+ years of relevant big data/dimensional/relational
· DB experience
· 5+ year experience in Python and Snowflake. Strong programming experience in Python
· Demonstrated experience using APIs to extract data with consideration of authentication and authorization methods to adhere to privacy and security policies
· Ability to architect, design and implement solutions with AWS Virtual Private Cloud, EC2, AWS Data Pipeline, AWS Cloud Formation, Auto Scaling, AWS Simple Storage Service, EMR and other AWS products.
· Data Modelling is required as the engineer will source the data and build the data model.
· Extensive experience working with Hadoop and related processing frameworks such as Spark, Hive, Sqoop, etc.
· Experience with workflow orchestration tools like Apache Airflow
· Experience with performance and scalability tuning
· Experience in Agile/Scrum application development using JIRA.
· Experience working in a public cloud environment, particularly AWS
· Experience with BitBucket
· Implemented coding standards and long-term best practices
Required Soft Skills:
· Demonstrated experience and ability to deliver results on multiple projects in a fast-paced, agile environment
· Excellent problem-solving and interpersonal communication skills
· Strong desire to learn and share knowledge with others
· Passionate about data and striving for excellence
· Desire to learn and understand the business and communicate with business stakeholders to accomplish business rules transformations and data validation while coding
· Desire and ability to work collaboratively with your teammates to come up with the best solution to a problem. Specially with architects, product managers, scrum masters, engineers.
Nice to Have:
· Familiarity with practices like Continuous Development, Continuous Integration and Automated Testing
· Familiarity with build tools such as Cloud Formation and automation tools such as Jenkins or Circle CI
· Call centre data globally
· Understands “social media” language and sourcing in social media data – i.e. Twitter, Facebook, Instagram
· ErWin experience is a plus
· Experience working and configuring data, custom fields and reporting in Sprinklr social application is a plus.
· Understand the technical details of the Sprinklr platform to extract the data using their APIs
· Interrogate Sprinklr’s authentication and authorization methods to make sure it adheres to Client’s privacy and security policies
· Work with architects, product managers, and scrum masters to deliver sprint goals every two weeks
· Work with the data enablement team who are our business stakeholders to understand data requirements pertaining to metrics and quality
· Design and implement features in collaboration with team engineers, product owners, data analysts, and business partners using Agile / Scrum methodology
· Model the social media data elements and make sure it integrates with the existing data model
· Work with the Salesforce engineer to determine how to integrate Sprinklr social media data with Salesforce application
· Contribute to overall architecture, frameworks and patterns for processing and storing large data volumes
· Design and implement distributed data processing pipelines using Spark, Hive, Sqoop, Python, and other tools and languages prevalent in the Hadoop ecosystem
· Build utilities, user defined functions, and frameworks to better enable data flow patterns
· Research, evaluate and utilize new technologies/tools/frameworks centred around high-volume data processing
· Define and apply appropriate data acquisition and consumption strategies for given
· Technical scenarios
· Build and incorporate automated unit tests and participate in integration testing efforts
· Work with architecture/engineering leads and other teams to ensure quality solutions are implemented, and engineering best practices are defined and adhered to
· Work across teams, even third party vendors, to resolve operational and performance issues.