Responsibilities:
- Perform data extraction, cleaning, transformation, and flow.
- Design, build, launch and maintain efficient and reliable large-scale batch and real time data pipelines with data processing frameworks.
- Integrate and collate data silos in a manner which is both scalable and compliant.
- Collaborate with Project Manager, Frontend Developers, UX Designers and Data
- Analyst to build scalable data-driven products.
- Be responsible for developing backend APIs & working on databases to support the applications.
- Bridge the gap between engineering and analytics.
- Work in an Agile Environment that practices Continuous Integration and Delivery
- Work closely with fellow developers through pair programming and code review process
Requirements:
- 3+ years of experience with building production-grade data pipelines, ETL/ELT data integration with proper documentation
- Proficient in general data cleaning and transformation (e.g. pandas, pysparks, SQL)
- Knowledge in database design and various databases (e.g. structured dbase(postgres, mysql etc and unstructured -mongodb etc)
- Knowledge about system design, data structure and algorithms
- Knowledge with rest api and web requests/protocols in general
- Comfortable coding in at least 2 scripting language (eg. Python/SQL)
- Comfortable working in both windows and linux development environments
- Interest in data engineering in a big data environment using Cloud platforms (i.e. AWS, Azure, Google Cloud)
- Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, and Data Warehouse.
www.dadaconsultants.com
EA Registration Number: R21103926
Data provided is for recruitment purposes only
Business Registration Number: 201735941W
Licence Number: 18S9037
