We are seeking a talented and experienced Spark Scala Developer with expertise in Elasticsearch to join our team. As a Spark Scala Developer, you will play a crucial role in designing, developing, and optimizing big data solutions using Apache Spark, Scala, and Elasticsearch. You will collaborate with cross-functional teams to build scalable and efficient data processing pipelines and search applications. Knowledge and experience in the Compliance / AML domain will be a plus.
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Strong experience in developing Spark applications. Experience with Spark Streaming is a plus
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Proficiency in Scala programming language and familiarity with functional programming concepts
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Experience integrating and working with Elasticsearch for data indexing and search applications
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Solid understanding of Elasticsearch data modeling, indexing strategies, and query optimization
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Experience with distributed computing, parallel processing, and working with large datasets
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Familiarity with big data technologies such as Hadoop, Hive, and HDFS
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Proficient in performance tuning and optimization techniques for Spark applications and Elasticsearch queries
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Strong problem-solving and analytical skills with the ability to debug and resolve complex issues
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Familiarity with version control systems (e.g., Git) and collaborative development workflows
ā¢Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Excellent communication and teamwork skills with the ability to work effectively in cross-functional teams