Summary

he Data Engineer will play a key role in designing, developing, and maintaining our data infrastructure, enabling the organization to extract valuable insights from diverse datasets. The candidate will collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to ensure the seamless flow of data and facilitate data-driven decision-making.

Duties & Responsibilities:

1. Implement scalable data lake/lakehouse solution on AWS.
2. Design and implement big data pipelines using batch and streaming tools.
3. Develop data modelling workflows with DBT.
4. Orchestrate workflows with Airflow.
5. Collaborate with global teams to ensure data governance standards are met.
6. Build and deploy end-to-end CI/CD pipelines, incorporating DevOps best practices.

Requirements:

1. 3+ years experience in a similar role.
2. Bachelor's degree or higher in a quantitative/technical field (e.g. Computer Science, Engineering or similar).
3. Great at communicating ideas and working in a team.
4. Solid proficiency experience with Python to develop data ingestion pipelines.
5. Advanced knowledge of REST APIs and Websockets.
6. Experience with Delta Lake infrastructures and large datasets processing with Apache Spark / PySpark.
7. AWS experience preferred, with proficiency in a wide range of AWS services (e.g., EC2, S3, Glue, Fargate, Lambda, IAM, EMR, Kinesis)
8. Working knowledge of DBT for Data Modeling.
9. Expert knowledge of SQL and experience in using cloud-based DWH, such as Redshift/Snowflake/Google Big Query.
10. Knowledge designing CI/CD pipelines and IaC experience, Terraform preferred.
11. Experience with building data pipeline processes and orchestration tools such as Airflow.
12. Good understanding of cloud infrastructure platforms.
13. Self-starter and detail-oriented. You can complete projects with minimal supervision.

Skills

• AWS
• Database Management
• Development
• Python
• Software Engineering
• SQL