Job Description:
A Cloud Data Engineer specializing in AWS, Databricks, and Informatica IDMC is responsible for building and maintaining a robust, integrated, and governed data infrastructure that leverages the strengths of the analytics platforms to extract valuable insights from data while ensuring data security, compliance, and high-quality data management.
Roles And Responsibilities:
• Develop ETL (Extract, Transform, Load) processes to cleanse,
transform, and enrich data, making it suitable for analytical purposes using Databricks' Spark capabilities and Informatica IDMC for data transformation and quality.
• Monitor and optimize data processing and query performance in both AWS and Databricks environments, making necessary adjustments to meet performance and scalability requirements. Utilize Informatica IDMCfor optimizing data workflows.
• Implement security best practices and data encryption methods to
protect sensitive data in both AWS and Databricks, while ensuring
compliance with data privacy regulations. Employ Informatica IDMC for data governance and compliance.
• Implement automation for routine tasks, such as data ingestion,
transformation, and monitoring, using AWS services like AWS Step
Functions, AWS Lambda, Databricks Jobs, and Informatica IDMC for
workflow automation.
• Maintain clear and comprehensive documentation of data infrastructure, pipelines, and configurations in both AWS and Databricks environments, with metadata management facilitated by Informatica IDMC.
• Collaborate with cross-functional teams, including data scientists,
analysts, and software engineers, to understand data requirements and deliver appropriate solutions across AWS, Databricks, and Informatica IDMC.
• Identify and resolve data-related issues and provide support to ensure data availability and integrity in both AWS, Databricks, and Informatica IDMC environments.
• Optimize AWS, Databricks, and Informatica resource usage to control costs while meeting performance and scalability requirements.
• Stay up-to-date with AWS, Databricks, Informatica IDMC services, and data engineering best practices to recommend and implement new technologies and techniques.
Requirements / Qualifications
• Bachelor’s or master’s degree in computer science, data engineering, or a related field.
• Minimum 4 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC.
• Proficiency in programming languages such as Python, Java, or Scala for building data pipelines.
• Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
• Strong knowledge of SQL and NoSQL databases.
• Familiarity with data modeling and schema design.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• AWS certifications, Databricks certifications, and Informatica
certifications are a plus
Next Step:
- Prepare your updated resume (please include your current salary package with full breakdown such as base, incentives, annual wage supplement, etc.) and expected package.
- Simply click on 'Apply here' to drop your resume or email at [email protected]
- All shortlisted candidates will be contacted.
Saghana Sithara | Registration Number: R1550224
