Job Duties
• Research, design, and develop computer and network software or specialised utility programs.
• Analyse user needs and develop software solutions, applying principles and techniques of computer science, engineering, and mathematical analysis.
• Update software, enhances existing software capabilities, and develops and direct software testing and validation procedures.
• Work with computer hardware engineers to integrate hardware and software systems and develop specifications and performance requirements.
Key Responsibilities
- Responsible for the development of ETL pipeline in open source to carry out data ingestion
- Write programs to extract data from data lake and curated data layer to meet business objectives
- Collaborate with different teams to gather the understanding of the application to design the ETL pipeline
- Gather business and functional requirements, and translate these requirements into robust, scalable, operable solutions that work well within the overall data architecture
- Participate in the full development life cycle, end-to-end, from design, implementation, and testing, to documentation, delivery, support, and maintenance and produce comprehensive, usable dataset documentation and metadata
Must Have Skills
- Programming languages: Python / Spark
- Experience with Linux utilities & SQL.
- Experience in using PySpark for data transformation.
- Knowledge of AWS services such as Redshift, Glue, Cloudformation, EC2, S3, Lambda.
Good to Have Skills
- ETL Tool Experience
- AWS Exposure
