Our client is looking for an experienced and highly skilled Senior Big Data Developer to join the dynamic team. The ideal candidate will have over 10 years of extensive working knowledge and hands-on expertise in big data technologies, especially in Spark on Scala or PySpark, and Hive. This role will involve the design, development, and performance tuning of Spark applications while working with cutting-edge big data tools and distributed systems.
As a key player, you will work closely with our data engineering team to provide robust, scalable solutions for data processing and analytics.
Responsibilities:
- Spark Expertise: Design, develop, and optimise Spark applications using Scala, PySpark, and Hive.
- Big Data Engineering: Work with big data tools and techniques, leveraging the Hadoop ecosystem for processing large datasets.
- Programming: Utilize strong programming skills in Java, Scala, or Python to build high-performance applications.
- Distributed Systems: Develop a deep understanding of distributed systems and ensure system efficiency.
- Database Management: Work with RDMS, Data Warehouses, and Unix Shell scripting for end-to-end data management and optimization.
- Automation: Implement workflows using Airflow and Ctrl-M for scheduling and monitoring.
- Collaboration: Work with cross-functional teams to ensure seamless data pipeline integrations.
Qualifications:
- 10+ years of experience working with Spark on Scala or PySpark and Hive.
- Strong expertise in Java, Scala, or Python programming.
- Proficient in the Hadoop ecosystem (HDFS, YARN, MapReduce, HBase, etc.).
- Solid understanding of distributed systems and their performance considerations.
- Familiarity with RDMS, Data Warehouses, and Unix Shell scripting.
- Experience with workflow management tools like Airflow and Ctrl-M.
- Strong analytical and problem-solving skills.
- Experience in performance tuning for Spark applications.
- Familiarity with cloud-based data platforms.
- Experience in a fast-paced, agile development environment.
EA REG:R1108137
EA LIC:95C5411
