As a Data Engineer, you need to:
• Create and maintain data architectures that meets the needs of HDB. • Build and maintain platforms required for storage, extraction, transformation, and loading of data from a wide variety of data sources. • Improve discoverability and distribution of analytical data to data analysts and data scientists. • Implement best practices in data security throughout the lifecycle of analytical data. EXPERIENCE AND SKILLS NEEDED As a Data Engineer, you need: • Background in Information Systems (IS), Computer Science or related field in a reputed institution. • Experience in designing and maintaining data platforms such as data lakes, data warehouses and the associated data pipelines. • Proficient in creating and maintaining complex data pipelines using tools such as Microsoft SQL Server Integration Services (SSIS). • Experience in Commercial Cloud such as Amazon web Services (AWS), Google cloud Platform (GCP) and Microsoft Azure. • Knowledge of data platforms such as Spark, Flink, Kafka, SQL and NoSQL databases, Airflow. • Knowledge of the latest developments in data engineering and best practices in data management. • Knowledge of cloud technologies in data platforms is a plus e.g. Snowflake, AWS Redshift, Google BigQuery, Azure Synapse.
