Job Duties:
- Develop and implement techniques and analytics applications to transform raw data into meaningful information using data-oriented programming languages and visualisation software.
- Apply data mining, data modelling, natural language processing, and machine learning to extract and analyse information from large structured and unstructured datasets.
- Visualise, interpret, and report data findings and may create dynamic data reports as well.
Job Requirements:
- Experience on Azure Cloud.
- Experience in developing data ingestion, data processing through Data bricks and analytical pipelines for relational databases, NoSQL and data warehouse solutions
- Strong experience in data curation/aggregation/preparation using Azure Data Lake, Azure Data Factory and distilling data into meaningful insights with hands-on expertise on Azure and Data bricks
- Designs, modifies, and builds new and scalable data processes
- Expert in Data Engineering and building data pipelines, implementing Algorithms in a distributed environment
- Develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity.
- Extensive experience providing practical direction within the Azure Native services
- Good experience in Spark SQL is preferred with Azure services.
- Extensive hands-on experience implementing data migration and data processing using Azure services: ADLS, Azure Data Factory, Azure Functions, Synapse/DW, Azure SQL DB, Data bricks Azure Data Catalog, etc.
- Cloud migration methodologies and processes including tools like Azure Data Factory, Data Migration Service, SSIS, Event Hub, Kafka, etc.
- Experience in using various File Formats and compression techniques
- Experience working with Developer tools such as Azure DevOps, GitLab
- Experience with private and public cloud architectures, pros/cons, and migration considerations
- Experience developing and deploying ETL solutions on Azure
- Familiarity with the technology stack available in the industry for metadata management: Data Governance, Data Quality, MDM, Lineage, Data Catalog etc.
- Experience of working in Agile delivery
- Proven ability to work creatively and analytically in a problem-solving environment
- Excellent communication (written and oral) and interpersonal skills
- Excellent organizational, multi-tasking, and time-management skills
- Sr Azure Data Engineer with strong experience on Azure ADF, Synapse, Python, exposure to Spark
- Strong understanding of Azure DevOps, exposure to Data modelling
