Data Engineer/Software Developer

This job has now expired please search on the home page to find live IT Jobs.

Job Description - Data Engineer/Software Developer

Location - London

Rate - up to £510.00 based on experience

Duration - initially 6 months with possibility of extension

Technologies

You will have demonstrable experience of developing reproducible ETL pipelines (using an infrastructure as code framework), using tools such as Python, SQL and ideally, Spark. You will be an experienced user of a version control system, preferably Git. You will have experience working with semi-structured data sources and/or data from administrative or transactional systems and transforming this data into structures suitable for use by analysts or business intelligence tools. You will have a experience or awareness of DevOps ways of working utilising automated testing and an understanding of cloud infrastructure ideally Docker and Kubernetes.

The key purpose of the role is to

1 Design processes that automatically transform data from operational systems to improve the accessibility, structure, timeliness and connectivity of data. This will build capability and dependability of data across the department.

2 Use modern software development practices to improve the maintainability, agility, reproducibility and quality of our data transformation pipelines, including building automated tests.

3 Work closely with analysts and stakeholders to understand user need, implementing data models enabling them to work faster and generate improved insight.

4 Transform government by helping to embed modern development practices.

5 Share knowledge of tools and techniques with the wider team and community

6 Help build and maintain a diverse, inclusive culture across the development community, growing awareness, inclusivity, and balance.

7 Challenge entrenched practices and influence decision makers, and look for deeper underlying problems to solve, and larger opportunities for digital transformation.

Key skills and experience:

1 Experience of software development or data engineering, specifically using Python and, ideally, experience using Spark.

2 A working knowledge of cloud infrastructure, ideally AWS including S3 and Athena.

3 An understanding of architectures for data warehousing and data lakes.

4 Experience in working in a devops environment utilising continuous integration.

5 A clear understanding of working in an Infrastructure as code framework, ideally with experience of Kubernetes and Docker.

6 The ability to manage technical debt appropriately and proportionately.

7 The ability to quickly learn new languages and frameworks and willingness to pick the right tool for the job.

8 A systematic approach to solving problems.

9 Knowledge of open source principles and technologies.

10 Enthusiasm for sharing knowledge and working in a multi-disciplinary team.

11 Experience of mentoring and supporting your colleagues, whether one-to-one or in groups.

General Qualities

The role requires someone who is dedicated to the task and works with a focused attention to detail. The candidate should also be a good communicator who is happy to seek out and engage with data SMEs on their own initiative as required.