Key Responsibilities:
· Collaborate with cross-functional teams to understand data requirements and design robust data systems.
· Develop, implement, and maintain scalable data pipelines to collect and transform data from various sources to end systems.
· Optimize and enhance existing data pipelines for better performance, scalability, and reliability.
· Ensure data quality and integrity through data validation, monitoring, and error handling.
· Troubleshoot and resolve data-related issues promptly.
· Collaborate with data scientists and analysts to provide necessary datasets and tools for advanced analytics and reporting.
· Guide junior team members in data engineering and software development best practices.
· Implement data security and privacy best practices to protect sensitive information.
· Stay updated with the latest trends and technologies in data engineering and contribute to the continuous improvement of our data infrastructure.
Skills & Qualifications:
· Bachelor’s degree in Computer Science, Information Technology, or a related field (or equivalent work experience).
· 5+ years of experience in data engineering or a related role with a proven record of designing and implementing complex data solutions. Familiarity with real-time processing is a plus.
· Proficiency in programming languages such as Python, Java, or Scala and data processing/orchestration frameworks like Apache Spark, Apache Kafka, and Apache Airflow.
· Experience with cloud platforms and related services such as AWS, Azure, and Databricks.
· Experiences in managing and optimizing data lake solutions.
· Solid knowledge of data modeling concepts and database systems (SQL and NoSQL).
· Knowledge of integrating ML models into data pipelines.
· Familiarity with software development lifecycle (SDLC), including CI/CD, version control, and testing.
· Understanding of data security and compliance standards.
· Excellent problem-solving and communication skills.
· Ability to work independently and collaboratively in a fast-paced environment.
· Strong attention to detail and a commitment to delivering high-quality work.
· Certification in relevant data engineering technologies is a plus.
