We are looking for Senior Data Engineer to join Data Engineer team squad. You will be responsible for platform administration, operational support, small change and release management. Larger, more complex changes are delivered by an L3 squad and project teams.
Key Responsibilities:
Release Management and Support activity (~60%):
- Assist the team with Production code deployment and data platform support
- Act as a gatekeeper Production following JERAGM’s change management process.
- Support production data pipelines and reports.
- Provide detailed analysis of data issues in Production with a quick turnaround time.
- On occasion, provide support for applications in Public Holidays and in weekend releases.
- Compile and present monthly Ops reports.
- Help the Data Engineering Manager by sharing evidence based on data-driven analysis.
- Set up security for end users when required.
Data Engineering (~40%):
- Design and deliver data solutions using an agile, iterative approach based on Scrum.
- Develop high quality data pipelines using Spark, Python, SQL and Azure services.
- Collaborate with the Data Engineering Manager on technical architecture and design.
- Analyse and estimate IT changes, providing input on technical opportunities, constraints, and trade-offs.
- Create documentation and present to both technical and non-technical audiences.
- Handover 1st line support to the JERAGM IT Ops teams and 2nd/3rd line support to the L2 squad.
- Conduct detailed testing for development activities and demonstrate results according to the delivery methodology and coding standards.
- Create complex charts and reports using Python libraries, Databricks, and Power BI.
Technical Skills
- 8 plus years of relevant experience.
- Strong proficiency in writing complex SQL Queries, Stored Procedure etc.
- Proficient in Python, or the ability to learn basic Python for the interview process.
- Experience with Azure or a willingness to learn and deepen knowledge of Azure platforms.
- Exposure to Azure Databricks, or an eagerness to gain experience working with Databricks.
- Experience orchestrating data pipelines using Azure Data Factory or SSIS.
- Knowledge of Python packages such as Pandas, Numpy, and Seaborn.
- Understanding of Big Data, MapReduce, Spark, and file formats like Parquet, Avro, and ORC.
- Familiarity with reporting tools such as Power BI or Tableau.
Support:
- Experience in a support role with experience using JIRA or similar tools to track issues and manage them through a small team.
- Experience managing releases under a change control process.
- Experience of communicating with users and other stakeholders.
Software engineering and delivery:
- Source code management e.g. Azure DevOps, Git
- Agile delivery methodologies such as SCRUM or Kanban
