As a Senior Data Engineer, you will be a part of a highly qualified team, leading the delivery of modern data platforms using Azure Services and Databricks.
- Designing and delivering software using an agile and iterative approach based on Scrum or Kanban
- Following and contributing to improvement of client's software engineering practices e.g., source code management, test-driven development, automated testing, CI/CD
- Supporting the Lead Data Engineer in technical architecture and design
- Building an understanding of how data is used within client's commercial activity, and using this knowledge when working with Business Analysts or users to identify system requirements
- Analyzing and estimating IT changes giving input on technical opportunities, constraints, and trade-offs
- Creating documentation and presenting to technical and non-technical audiences · Providing handover to Support teams and providing third line support for short periods after releases
- Owning their own learning so that they remain a technical subject matter expert.
Technical Skill
- Having solid experience in data modeling, data warehousing, SQL Server database design and development (SQL, relational and dimensional modelling) Analytical querying using technologies such as SSAS cubes or columnar data stores, Python. · ETL tools such as SSIS, Alteryx, Azure Data Factory · Building data pipelines using code e.g. Pyspark · Modern cloud-based data architectures and technologies such as Data Lake, Delta Lake, Data Lakehouse e.g. Azure Data Lake Gen 2, Azure Delta Lake Lakehouse architecture using Databricks, Hudi or Iceberg
- NoSQL using CosmosDB, MongoDB, DynamoDB etc.
- Python scripting, including data engineering and machine learning libraries · Scripting languages such as Powershell, bash, DAX, M-language etc. (Note - We primarily use SQL, Pyspark and Python in our day-to-day work along with some coding around DAX, Bash, Powershell and other scripting languages. )
- Distributed computing mainly around Spark § Big data files formats e.g. Parquet, Avro, ORC etc.
- Big data technologies such as Hadoop, Hive, Map-Reduce, Partition design, performance tuning in Spark etc.
- Reporting and visualisation tools such as SSRS, Tableau, Power BI or Excel
