Position Overview: We are seeking an experienced Senior Data Engineer with a strong background in backend technologies and data engineering, particularly within the banking sector. The ideal candidate will have experience with Scala, Python, Java, and various big data technologies. They should also have hands-on experience with Azure and on-premise data lakes, data pipeline creation, and performance tuning.
Strong background in building data strategies and implementing them across various platforms. Experience with data warehousing, data processing, and handling large datasets is essential.
Key Responsibilities:
- Data Pipeline Development: Design, develop, and maintain data pipelines using Scala Spark, PySpark, and Spark SQL. Utilize a test-driven development (TDD) approach to ensure high-quality solutions.
- Microservices Development: Build and maintain microservices using Spring Boot to expose APIs for compute and configuration data.
- Regulatory Reporting: Create and enhance regulatory reports, managing data in Hive and Impala. Develop and maintain views and services for business users.
- Performance Optimization: Address performance issues and tech debt in existing modules. Perform housekeeping tasks such as purging and cleaning up HDFS partitions.
- Collaboration: Work with cross-functional teams to gather requirements, design scalable solutions, and document requirements in JIRA.
- CI/CD and Release Management: Build and manage CI/CD pipelines using Jenkins and Kubernetes. Oversee release activities, including documentation, user sign-off, regression, and performance testing.
- Quality Assurance: Write unit tests and code coverage to improve code quality and system resilience, using tools like Mockito for Scala and Pytest for Python.
Required Skills:
- Technologies: Scala, Python, Java (Spring Boot), Apache Spark, PySpark, SQL (Hive, Impala), Apache Kafka, Azure Databricks.
- Cloud Stack: AWS, Azure, EMR Cluster.
- On-Premise: Cloudera (HDP), 48 Node Cluster, Storage Formats (Parquet, AVRO).
- Tools: Airflow, TWS Scheduler, Shell Scripting (UNIX), Jenkins, Kubernetes.
- Experience: Minimum of 7 years in Data Engineering with a strong focus on banking or financial services. Proven experience in data lake migration, performance tuning, and regulatory reporting.
Preferred Qualifications:
- Experience with data modeling and transformation in Azure Data Lake and on-premise environments.
- Ability to work with large datasets and optimize data storage and retrieval.
- Strong problem-solving skills and the ability to manage production issues efficiently.
- Experience with large-scale data engineering projects.
- Strong background in data transformation and reporting.
Interested candidates, please click "APPLY" to begin your job search journey and submit your CV directly through the official PERSOLKELLY job application platform - GO Mobile.
We regret to inform you that only shortlisted candidates will be notified.
Gelangre Reyanelle Gelario | REG No : R1870995
PERSOLKELLY SINGAPORE PTE LTD | EA License No : 01C4394
This is in partnership with Employment and Employability Institute Pte Ltd (“e2i”). e2i is the empowering network for workers and employers seeking employment and employability solutions. e2i serves as a bridge between workers and employers, connecting with workers to offer job security through job-matching, career guidance and skills upgrading services, and partnering employers to address their manpower needs through recruitment, training and job redesign solutions. e2i is a tripartite initiative of the National Trades Union Congress set up to support nation-wide manpower and skills upgrading initiatives. By applying for this role, you consent to e2i’s PDPA.
By sending us your personal data and curriculum vitae (CV), you are deemed to consent to PERSOLKELLY Singapore Pte Ltd and its affiliates to collect, use and disclose your personal data for the purposes set out in the Privacy Policy available at https://www.persolkelly.com.sg/policies. You acknowledge that you have read, understood, and agree with the Privacy Policy.