A well-funded technology startup is seeking a Data Engineering Lead to be part of their team here in Singapore.
The Role
We are seeking a skilled and experienced Data Engineering Lead with 8+ years of industry experience to join our dynamic team. As a Data Engineering Lead, you will be responsible for leading a team of data engineers in designing, developing, and maintaining our data infrastructure and processing pipelines using Python, Spark on aws. Your expertise in data engineering and strong programming skills will contribute to the efficient processing, analysis, and storage of large-scale datasets. This is an exciting opportunity to work on challenging projects and collaborate with cross-functional teams in a fast-paced environment.
Responsibilities:
- Design and develop scalable and robust data processing pipelines using Python and Spark, ensuring efficient data ingestion, transformation, and storage
- Provide leadership and guidance to junior resources
- Take ownership of data team's deliverables
- Collaborate with data scientists, analysts, and stakeholders to understand data requirements and implement appropriate data engineering solutions.
- Optimize and tune data processing workflows to enhance performance, reliability, and scalability.
- Build and maintain data warehouses, data lakes, and other data storage systems to support analytics and reporting needs.
- Develop and maintain ETL (Extract, Transform, Load) processes to integrate data from various sources into a unified format for analysis.
- Monitor data infrastructure and pipelines, identify and resolve issues, and ensure data quality and integrity.
- Implement and maintain data security and privacy measures to protect sensitive information.
- Stay updated with industry trends and emerging technologies in data engineering and provide recommendations for process improvements and tool selection.
- Collaborate with cross-functional teams to define and implement data governance policies and best practices.
- Document data engineering processes, standards, and workflows.
Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- Has succesully lead a team of junior data engineers to implement data layers of solutions and products
- 8+ years of relevant industry experience as a Data Engineer.
- Strong programming skills in Python, including experience with data manipulation, data structures, and algorithmic problem-solving.
- Proficiency in Apache Spark for big data processing, including experience with Spark SQL, Spark Streaming, and Spark MLlib.
- Solid understanding of distributed computing principles and concepts.
- Experience with data modeling, data warehousing, and ETL processes.
- Familiarity with cloud platforms like AWS, Azure, or GCP, and their data services (e.g., Amazon EMR, Azure Databricks).
- Proficient in SQL and experience with relational databases (e.g., PostgreSQL, MySQL).
- Strong analytical and problem-solving skills with the ability to handle complex data scenarios.
- Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.
- Attention to detail and a strong commitment to data quality, integrity, and security.
- Experience with version control systems (e.g., Git) and agile development methodologies is a plus.
