Job Scope:
We are seeking for a Data Analytics Architect specializing in Azure and Databricks to design, develop, and manage cloud-based data solutions. In this role, you will be responsible for building scalable data pipelines, optimizing data architectures, and ensuring the efficient use of Azure and Databricks services. The ideal candidate will possess deep technical knowledge of Microsoft’s Azure ecosystem, Databricks, and related tools, with a solid focus on delivering high-performance data solutions.
Key Responsibilities:
Azure Architecture and Strategy:
- Design and implement Azure-based data architectures that are scalable, efficient, and aligned with business objectives.
- Develop a data strategy leveraging Azure Data Lake, Azure Synapse Analytics, and Azure Databricks for advanced data processing, storage, and analytics.
Databricks Platform Leadership:
- Lead the development of data pipelines and big data solutions using Azure Databricks, with a solid focus on leveraging Apache Spark for scalable data transformations.
- Optimize Databricks clusters for performance, cost management, and efficient resource utilization.
Data Integration and Extract, Transform, Load (ETL):
- Design, develop, and maintain robust ETL/ELT processes using Azure Data Factory (ADF) and Azure Databricks to ensure seamless data integration across various data sources.
- Lead the transformation of raw data into meaningful and actionable insights through PySpark or Spark SQL in Databricks.
Data Governance and Management:
- Implement and enforce data governance best practices using Azure Purview to ensure compliance, data quality, and integrity across the organization.
- Manage Azure Data Lake storage policies to ensure secure and scalable data management.
Performance Optimization:
- Continuously monitor and improve the performance of Azure SQL Database, Azure Synapse Analytics, and Databricks environments.
- Implement data partitioning, indexing, and other optimization techniques for fast and reliable data access.
Collaboration and Stakeholder Communication:
- Collaborate closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data needs and provide tailored solutions.
- Communicate technical specifications clearly to both technical and non-technical stakeholders.
Team Leadership and Mentoring:
- Lead and mentor a team of data engineers in best practices related to Azure and Databricks platforms.
- Provide guidance on Azure DevOps practices, such as setting up continuous integration/continuous delivery (CI/CD) pipelines for automated data pipeline deployments.
Security and Compliance:
- Implement best practices for securing data at rest and in transit using Azure Key Vault, Azure Active Directory, and role-based access controls.
- Ensure compliance with data privacy regulations and internal security policies.
Job Qualifications:
- Degree in Information Technology or a related field.
- Extensive experience with Azure Data Services, including Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Data Lake, and Azure Storage.
- In-depth knowledge of Azure Databricks for data engineering, including proficiency in Apache Spark, PySpark, and Delta Lake.
- Expertise in Azure Data Factory (ADF) for building complex data pipelines and integrating data from various structured and unstructured sources.
- Proficiency in designing and managing data lakes and data warehouses using Azure Synapse Analytics.
- Excellent experience with SQL, Python, and PySpark for building and optimizing data workflows.
- Proficient in Azure DevOps for implementing CI/CD pipelines and managing code repositories.
- Proficient in performance optimization techniques for Databricks and Azure SQL (e.g., partitioning, indexing, caching).
- Knowledge of cost optimization strategies for Azure resources, ensuring efficient use of cloud services.
- Demonstrated experience leading and mentoring data engineering teams.
- Good communication skills to articulate technical concepts and solutions effectively to both technical and non-technical stakeholders.
Interested candidates can click apply for more information.
** We regret to inform that only shortlisted candidates will be notified. **
We respect your privacy and all communication will be treated with confidentiality. If you wish to know more about this position or explore other roles, please prepare your updated profile and get in touch with our consultants.
Please note that your response to this advertisement and communications with us pursuant to this advertisement will constitute informed consent to the collection, use and/or disclosure of personal data by Manpower for the purpose of carrying out its business, in compliance with the relevant provisions of the Personal Data Protection Act 2012.
Rallonza Bernice Mae Nocum | EA License No. 02C3423 Personnel | Registration No. R1442141