As a Data Engineer, you need to:
• Create and maintain data architectures.
• Build and maintain platforms required for storage, extraction, transformation, and loading of data from a wide variety of data sources.
• Improve discoverability and distribution of analytical data to data analysts and data scientists.
• Implement best practices in data security throughout the lifecycle of analytical data.
EXPERIENCE AND SKILLS NEEDED
As a Data Engineer, you need:
• Background in Information Systems (IS), Computer Science or related field in a reputed institution.
• Experience in designing and maintaining data platforms such as data lakes, data warehouses and the associated data pipelines.
• Proficient in creating and maintaining complex data pipelines using tools such as Microsoft SQL Server Integration Services (SSIS).
• Experience in Commercial Cloud such as Amazon web Services (AWS), Google cloud Platform (GCP) and Microsoft Azure.
• Knowledge of data platforms such as Spark, Flink, Kafka, SQL and NoSQL databases, Airflow.
• Knowledge of the latest developments in data engineering and best practices in data management.
• Knowledge of cloud technologies in data platforms is a plus e.g. Snowflake,
AWS Redshift, Google BigQuery, Azure Synapse.
