Description:
• Design, develop, and maintain scalable data pipelines using Spark and Scala to process large datasets.
• Implement and manage Elasticsearch clusters for efficient data retrieval and analysis.
• Collaborate with data scientists and analysts to understand data requirements and provide support for analytical solutions.
• Utilize Quantexa for entity resolution and network analytics to enhance data quality and insights.
• Monitor and optimize data workflows for performance, reliability, and scalability.
• Document data processes and ensure best practices in data engineering are followed.
• Participate in code reviews and contribute to the continuous improvement of our data engineering practices.
Requirements:
• Proven experience as a Data Engineer, with a strong focus on Spark and Scala.
• Proficiency in Elasticsearch for search and analytics applications.
• Familiarity with Quantexa or similar tools for data enrichment and entity resolution.
• Experience with cloud platforms (AWS, Azure, GCP) is a plus.
• Strong SQL skills and experience with relational databases.
