Data Engineer

Type werk: Full-time
Salaris:
70.000 - 110.000 GBP/Jaarlijks

Join our dynamic team in London, working on cutting-edge ML-based solutions for a leading European retail chain. We're scaling our successful demand forecast and delivery planning systems to service all 100 stores of a major supermarket within the chain. This exciting opportunity involves migrating our existing on-premises architecture to Google Cloud Platform (GCP) for enhanced scalability and reliability.

Key Responsibilities:

Migrate existing data pipelines from Airflow to GCP infrastructure.
Transition feature engineering framework to BigQuery for scalability.
Develop and manage a scalable prediction API for machine learning models.
Transition current infrastructure to GCP, addressing architecture challenges.
Optimize system structure to minimize costs.
Design and implement an observability pipeline for the system.In addition to these core responsibilities, you'll play a vital role in providing technical guidance to our engineering team on various projects.

Requirements: Essential:

Proficiency in implementing and monitoring DAGs in Airflow.
Experience with a cloud provider (AWS/GCP) and knowledge of various compute and storage resources.
Understanding of containerized systems and Kubernetes.
Familiarity with Terraform for provisioning complex infrastructures.
Solid Python skills for developing data pipelines and basic API implementation (Flask or FastAPI).
Understanding of observability pipelines (e.g., Prometheus+Grafana, ELK stack).
Proficiency in SQL and experience with DBT or equivalent frameworks.Desirable:

Knowledge of scalable data processing systems like Spark.
Familiarity with Machine Learning in Python.
Strong skills in Continuous Integration/Continuous Deployment (CI/CD).
Experience working with large datasets (TBs).If you're passionate about leveraging data to drive business insights and enjoy working in a collaborative environment, we'd love to hear from you. Apply now to be part of our innovative team reshaping the future of retail logistics

PrintRapporteer

Solliciteer voor deze baan