We are seeking a highly skilled Lead Data Engineer to oversee our data engineering team and help us design and implement advanced data pipelines and solutions. The ideal candidate will have a strong background in Apache Spark and Scala development, with extensive experience in building, managing, and optimizing big data processing pipelines. This is a leadership role requiring deep technical expertise, project management skills, and the ability to mentor and guide junior engineers.
Key Responsibilities:
• Lead the design and implementation of large-scale data processing systems using Spark and Scala.
• Architect, develop, and optimize data pipelines for efficient data transformation, aggregation, and processing of large datasets.
• Work with cross-functional teams to define data engineering solutions that meet business and operational requirements.
• Lead a team of data engineers, provide technical guidance, and ensure adherence to coding standards and best practices.
• Collaborate with data scientists and analysts to understand data needs and deliver high-quality datasets for analytics.
• Work with cloud platforms (AWS, Azure, GCP) to design and implement scalable data solutions.
Requirements:
• 6+ years of experience in data engineering with at least 2 years in a lead role.
• Expertise in Apache Spark and Scala for building scalable, high-performance data processing pipelines.
• Proficiency in big data technologies like Hadoop, Hive, HBase, and Kafka.
• Strong experience with cloud platforms (AWS, Azure, or GCP) for data processing and storage.
• Experience in SQL and NoSQL databases such as PostgreSQL, Cassandra, or MongoDB.
• Deep understanding of data warehousing concepts and data modeling techniques.
• Experience in managing and leading engineering teams, with strong mentoring and coaching skills.
