Senior Data Engineer
- Remote (UK-based candidates
- £500 - £550 per day - Outside IR35
- 6 months +
- Start date: Decemeber/January
My client, a leading retailer currently has a need for a Senior Data Engineer (Azure / Databricks / Data Factory ) to join a newly formed Scrum team. The initial project team will work on the end-to-end implementation of Data Pipelines using Azure Data Factory and Databricks.
The scrum team's role will be to develop the integration and Data pipeline components to ensure customer data from other systems is fed into the new Data Warehouse using Azure cloud technologies.
As the Senior Data Engineer (Azure / Databricks / Data Factory) you will have several responsibilities including:
- Work with Product Owner / Business Analysts / Business Stakeholders to understand functional requirements and interact with other teams to estimate, architect, design, develop, test & release features.
- Work with team members to collaborate on feature delivery (design, code reviews, testing etc), attend key team meetings (as required), update Jira tickets, and participate in delivery planning & reporting activities.
- Iterative development (based on requirements/feedback) of:
- Data pipelines and ETL processes using Azure Data Factory, Data Flow, Databricks / Spark (using Python & associated frameworks), Azure storage (data lake, blob) etc.
- Database engines/technologies (Synapse, SQL MI, Azure SQL, Cosmos DB etc.)
- Code management builds and releases automation (CI/CD) using Azure DevOps and Atlassian tooling as required.
The essential skills required for the Senior Data Engineer (Azure / Databricks / Data Factory ) include:
- Azure Data Factory
- End to end Data Pipeline experience
- SQL Data Warehouse
- Strong Data Warehouse and Microsoft BI background
- Data Modelling
- Agile / Scrum methodology
- Data Quality / Profiling experience