As a Data Engineer, you will be required to:
• Perform data extraction, cleaning, transformation, and flow.
• Design, build, launch and maintain efficient and reliable large-scale batch and realtime data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner which is both scalable and compliant.
• Collaborate with Project Manager, Frontend Developers, UX Designers and Data
• Analyst to build scalable data-driven products.
• Be responsible for developing backend APIs & working on databases to support the applications.
• Bridge the gap between engineering and analytics.
• Work in an Agile Environment that practices Continuous Integration and Delivery
• Work closely with fellow developers through pair programming and code review process
EXPERIENCE AND SKILLS NEEDED
As a Data Engineer, you need to bring to the team:
• Experience with building production-grade data pipelines, ETL/ELT data integration with proper documentation
• Proficient in general data cleaning and transformation (e.g. pandas, pysparks, SQL)
• Knowledge in database design and various databases (e.g. structured dbase(postgres, mysql etc and unstructured -mongodb etc)
• Knowledge about system design, data structure and algorithms
• Knowledge with rest api and web requests/protocols in general
• Comfortable coding in at least 2 scripting language (eg. Python/SQL)
• Comfortable working in both windows and linux development environments
• Interest in data engineering in a big data environment using Cloud
• platforms (i.e. AWS, Azure, Google Cloud)
• Familiar with data modelling, data access, and data storage infrastructure like Data
• Mart, Data Lake, and Data Warehouse.
• Proficient in creating comprehensive data dictionaries to ensure clear and consistent data definitions.
• Skilled in designing Entity-Relationship Diagrams (ERDs) to visually represent data models and relationships.
• Capable of collaborating with stakeholders to understand business processes and develop effective data solutions.
• Familiar with BI Tools such as Tableau
• Familiar with Docker containers
