Job Summary:
- We are seeking a highly skilled and experienced Big Data Engineer to join our team. The ideal candidate must have
- a minimum of 4 years of experience managing data engineering jobs in big data environment e.g., Cloudera Data
- Platform. Overall experience can be 5 to 10 years. Consultant/ Senior Consultant preferably. The successful
- candidate will be responsible for designing, developing, and maintaining the data ingestion and processing jobs as
- per business requirements. Candidate will also be integrating data sets to provide seamless data access to users.
Key Responsibilities:
• Analyze the Authority’s data needs and document the requirements.
• Refine data collection/consumption by migrating data collection to more efficient channels.
• Plan, design and implement data engineering jobs and reporting solutions to meet the analytical needs.
• Develop test plan and scripts for system testing, support user acceptance testing.
• Work with the Authority’s technical teams to ensure smooth deployment and adoption of new solution.
• Ensure the smooth operations and service level of IT solutions.
• Support production issues - Track record in implementing systems with high availability, high performance,
high security.
What we are looking for:
• Good understanding and completion of projects using waterfall/Agile methodology.
• Good understanding of analytics and data warehouse implementations.
• Ability to troubleshoot complex issues ranging from system resource to application stack traces.
• Strong SQL, data modelling and data analysis skills are a must.
• Hands-on experience in big data engineering jobs using Python, PySpark, Linux, and ETL tools like Informatica.
• Track record in implementing systems using Hive, Impala and Cloudera Data Platform will be preferred.
• Hands-on experience in DevOps deployment and data virtualization tools like Denodo will be preferred.
• Understanding/ Hands on experience of reporting or visualization tool like SAP BO and Tableau will be beneficial but not required.
• Passion for automation, standardization, and best practices
• Good written and verbal communication and interpersonal skills, ability to understand the business
requirement, communicate confidently with stakeholders.
