Job Scope:
We are seeking an experienced Data Engineering Lead specializing in Azure and Databricks to design, develop, and manage cloud-based data solutions. In this role, you will be responsible for building scalable data pipelines, optimizing data architectures, and ensuring the efficient use of Azure and Databricks services. The ideal candidate will possess deep technical knowledge of Microsoft’s Azure ecosystem, Databricks, and related tools, with a solid focus on delivering high-performance data solutions.
Key Responsibilities:
Azure Architecture and Strategy:
- Design and implement Azure-based data architectures that are scalable, efficient, and aligned with business objectives.
- Develop a data strategy leveraging Azure Data Lake, Azure Synapse Analytics, and Azure Databricks for advanced data processing, storage, and analytics.
Databricks Platform Leadership:
- Lead the development of data pipelines and big data solutions using Azure Databricks, with a solid focus on leveraging Apache Spark for scalable data transformations.
- Optimize Databricks clusters for performance, cost management, and efficient resource utilization.
Data Integration and Extract, Transform, Load (ETL):
- Design, develop, and maintain robust ETL/ELT processes using Azure Data Factory (ADF) and Azure Databricks to ensure seamless data integration across various data sources.
- Lead the transformation of raw data into meaningful and actionable insights through PySpark or Spark SQL in Databricks.
Data Governance and Management:
- Implement and enforce data governance best practices using Azure Purview to ensure compliance, data quality, and integrity across the organization.
- Manage Azure Data Lake storage policies to ensure secure and scalable data management.
Performance Optimization:
- Continuously monitor and improve the performance of Azure SQL Database, Azure Synapse Analytics, and Databricks environments.
- Implement data partitioning, indexing, and other optimization techniques for fast and reliable data access.
Collaboration and Stakeholder Communication:
- Collaborate closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data needs and provide tailored solutions.
- Communicate technical specifications clearly to both technical and non-technical stakeholders.
Team Leadership and Mentoring:
- Lead and mentor a team of data engineers in best practices related to Azure and Databricks platforms.
- Provide guidance on Azure DevOps practices, such as setting up continuous integration/continuous delivery (CI/CD) pipelines for automated data pipeline deployments.
Security and Compliance:
- Implement best practices for securing data at rest and in transit using Azure Key Vault, Azure Active Directory, and role-based access controls.
- Ensure compliance with data privacy regulations and internal security policies.
Job Qualifications:
Educational Background:
- Bachelor’s or Master’s Degree in Computer Science, Information Systems, or a related field.
Microsoft Azure Expertise:
- Extensive experience with Azure Data Services, including Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Data Lake, and Azure Storage.
- In-depth knowledge of Azure Databricks for data engineering, including proficiency in Apache Spark, PySpark, and Delta Lake.
ETL/ELT and Data Integration:
- Expertise in Azure Data Factory (ADF) for building complex data pipelines and integrating data from various structured and unstructured sources.
- Proficiency in designing and managing data lakes and data warehouses using Azure Synapse Analytics.
Programming and Tools:
- Excellent experience with SQL, Python, and PySpark for building and optimizing data workflows.
- Proficient in Azure DevOps for implementing CI/CD pipelines and managing code repositories.
Data Governance and Security:
- Familiarity with Azure Purview for data governance and cataloging.
- Experience with Azure Key Vault, Azure AD, and RBAC (Role-Based Access Control) to manage security and compliance across data platforms.
Optimization and Performance Tuning:
- Proficient in performance optimization techniques for Databricks and Azure SQL (e.g., partitioning, indexing, caching).
- Knowledge of cost optimization strategies for Azure resources, ensuring efficient use of cloud services.
Team Leadership and Communication:
- Demonstrated experience leading and mentoring data engineering teams.
- Good communication skills to articulate technical concepts and solutions effectively to both technical and non-technical stakeholders.
Additional requirements:
- Microsoft certifications such as Azure Data Engineer or Azure Solutions Architect.
- Familiarity with Azure Stream Analytics and real-time data streaming with Event Hubs or Kafka.
- Experience with PowerBI or Azure Analysis Services for reporting and analytics.
Interested candidates can click apply for more information.
** We regret to inform that only shortlisted candidates will be notified. **
We respect your privacy and all communication will be treated with confidentiality. If you wish to know more about this position or explore other roles, please prepare your updated profile and get in touch with our consultants.
Please note that your response to this advertisement and communications with us pursuant to this advertisement will constitute informed consent to the collection, use and/or disclosure of personal data by Manpower for the purpose of carrying out its business, in compliance with the relevant provisions of the Personal Data Protection Act 2012.
Rallonza Bernice Mae Nocum | EA License No. 02C3423 Personnel | Registration No. R1442141
