senior Data Engineer ( AWS )

Posted 15 November 2022
Salary £450.00 - £500.00 per day
LocationLondon
Job type Temporary
Discipline Data Engineering & Data Science
ReferenceBBBH96288_1668594538

Job description

Senior Data Engineer ( AWS / Redshift / Python )

  • Outside IR35
  • £450-£500 Per Day
  • Hybrid - London ( 1 day in the office a week )
  • 3-6 Month Contract

You will help to establish the infrastructure in AWS to handle enterprise-scale volumes of data from numerous sources, build out the data lake and data warehouse for operational and analytical discovery as well as helping to develop data pipelines for a number of different applications.

Key Responsibilities

  • Improving the maturity of the data estate with better use of CloudFormation and CI/CD
  • Help to build and support an enterprise-level data warehouse in Redshift
  • Build ETL/ELT data pipelines with AWS Glue and Lambda to clean, transform and load data from its raw format to multiple endpoints and for multiple use cases
  • Maintain and improve data governance within the cloud estate
  • Produce data exports and queries for teams throughout the business

Essential Skills & Experience

  • Excellent experience working with AWS, including Lambda, Glue, SQS, SNS, S3, ECS, CloudWatch and Step Functions
  • Good experience working with Amazon Redshift or PostgreSQL
  • Skilled in SQL
  • Proficient in developing with Python
  • Good experience of CI/CD and AWS SAM
  • Understanding of the AWS Well-Architected Framework
  • Experience working in a fast-paced agile environment
  • Understanding of how to architect resilient, highly available and efficient data pipelines in Databricks with Pyspark.

Desirable Skills & Experience

  • Experience working with DynamoDB, including DynamoDB Streams
  • Familiarity with business intelligence tools such as Tableau or QuickSight
  • Understanding of the Kimball methodology for data warehousing
  • Awareness of GDPR/PECR/DPA
  • Knowledge of Amazon Kinesis, IAM, Athena, Glue
  • Proficient in building Spark/PySpark jobs to process big data volumes on structured and semi-structured data
  • AWS certification is a plus