Job Description:
About our team & Job Description Summary::
Our team operates at the cutting edge of the merchants data & insights domain, focused on developing innovative merchant data marts and mid-tier microservices. We leverage modern technologies like Java, Scala, ETL processes, BigQuery (BQ), BigTable (BT), DataProc, Apache Spark, Airflow, Google Cloud, AI/ML, and Java-based microservices to create scalable, data-driven solutions. We are seeking talented engineers to join us in delivering robust data infrastructure and services.
Role and Responsibilities:
Role Overview:
We are looking for an experienced Member of Technical Staff with at least 10 + years of experience who excels in building and maintaining data marts, ETL pipelines, and microservices using Java or Scala, and Google Cloud technologies. In this role, you will play a critical part in driving data architecture, developing microservices, and ensuring seamless integration with merchant data platforms.
Key Responsibilities:
- Design, develop, and maintain ETL pipelines to process large-scale merchant data efficiently.
- Build and optimize merchant data marts using BigQuery and BigTable for data storage and retrieval.
- Leverage Apache Spark and Google Cloud Dataproc to handle complex data processing tasks.
- Implement microservices using Java/scala, ensuring scalability and high performance for mid-tier applications.
- Develop and orchestrate data workflows using Apache Airflow to streamline and automate processes.
- Collaborate with data scientists and AI/ML engineers to integrate machine learning models into data pipelines for deeper insights into merchant data.
- Maintain and optimize Google Cloud infrastructure to ensure high availability, security, and performance of data systems.
- Troubleshoot and enhance existing data systems and microservices to improve efficiency and reliability.
- Ensure best practices for data governance, security, and compliance are followed throughout the data lifecycle.
- Identify opportunities for refactoring, API changes, solution designs and other types of tech debt remediation.
- Influence management for driving technical decision-making for the right selection of technology by giving forthright, understandable, and actionable advice.
- Drive decision-making with team members with different views for simple to complex problems with the business requirements to help synthesize the best solution.
- Engage with business team, highly analytical and have experience designing and implementing A/B tests, with approach like data-based experiments. You know exactly what and how to track business metrics and KPIs.
Required Skills and Qualifications:
- Masters/bachelor’s degree in computer science, Data Engineering, or a related field.
- Minimum of 10 to 14 years of experience in data engineering, microservices development, or a similar role.
- Proficiency in Java for both data engineering tasks and developing microservices.
- Strong hands-on experience with Scala, ETL processes, and building data marts.
- Expertise with Google Cloud technologies, including BigQuery (BQ), BigTable (BT), DataProc, and Apache Spark.
- Experience in building and managing workflows using Apache Airflow.
- Exposure to AI/ML frameworks and integrating them with data processing pipelines.
- Solid understanding of cloud infrastructure and microservices architecture.
- Strong problem-solving and analytical skills with a focus on creating scalable solutions.
- Must be a self- starter, work independently or as a team member, and have strong communication skills in interfacing with business partners and the Technology.
- Experience with using Agile/Scrum methodology for software development.
Preferred Qualifications:
- Experience in the fintech or merchant data domain.
- Familiarity with data governance and compliance standards for large-scale data systems.
- Prior experience working with AI/ML models for data insights.
