You will play a crucial role in designing, developing, and maintaining our organisation's data infrastructure and pipelines that enable our organisation to extract valuable insights.
Responsibilities
- Identify data from different source systems and develop ingestion, data processing and data warehouse pipeline.
- Supports the design, implementation and maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information in a scalable, repeatable, and secure manner.
- Defining optimal solutions to data collection, processing, and warehousing.
- Focuses on collecting, parsing, managing, analysing, and visualising large sets of data to turn information into insights accessible through multiple platforms.
- Collaborate with stakeholders to understand organisation requirements then build, design and manage interpret solutions for reporting and dashboard.
- Ensure business rules and data definition are standardised across users and reports/dashboards.
- Collaborate with cross-functional teams to ensure data integration and interoperability across different systems and platforms.
- Review and ensure organisation’s data policy, data security standard and data governance guidelines are adhered to.
- Implement and monitor data security and privacy measures on existing data solutions.
- Create and maintain comprehensive documentation of data processes, pipelines, and infrastructure.
- Stay up to date with industry trends and emerging technologies related to data engineering, applying new knowledge to improve existing processes.
- Other duties as required.
Requirements
- Proficiency in one or more programming languages commonly used in data engineering, such as Python, Java, or Scala.
- Familiarity with relational databases (e.g., MsSQL, MySQL) and NoSQL databases (e.g., MongoDB).
- Strong understanding of SQL for querying, data manipulation, and database management.
- Familiarity with data warehousing concepts, dimensional modelling, and data schema design.
- Basic knowledge of Extract, Transform, Load (ETL) tools and frameworks.
- Understanding of various data formats like CSV, JSON, Avro, Parquet, etc.
- Basic understanding of cloud computing platforms like AWS, Azure, or GCP, and their relevant data services.
- Familiarity with version control systems, such as Git.
- Excellent communication and interpersonal skills.
- Ability to work independently and as part of a team.
Designation & salary will commensurate with experience.
