We are looking for a skilled Data Engineer with a strong background in Python, PySpark, Spark, and SQL. The ideal candidate will have experience in building and optimizing data pipelines and architectures. This role requires a collaborative professional who can work across teams to design and implement robust data solutions.
Key Responsibilities:
- Design, build, and maintain efficient, scalable, and reliable data pipelines using Python, PySpark, Spark, and SQL.
- Collaborate with data scientists, analysts, and other stakeholders to understand data needs and deliver appropriate data solutions.
- Develop and optimize ETL (Extract, Transform, Load) processes to ensure data quality and integrity.
- Implement data processing workflows for batch and real-time data integration.
- Monitor and troubleshoot data pipelines, ensuring data accuracy and availability.
- Perform data exploration and analysis to support business decisions.
- Document data processes, architectures, and workflows.
Proven 4+Years of experience as a Data Engineer or in a similar role.
Strong proficiency in Python and PySpark for data manipulation and transformation.
Hands-on experience with Apache Spark for large-scale data processing.
Advanced SQL skills for data querying, transformation, and performance tuning.
Familiarity with cloud platforms (AWS, Azure, Google Cloud) and their data services.
Experience with data warehousing concepts and technologies (e.g., Redshift, BigQuery, Snowflake).
Knowledge of best practices for data architecture and data governance.
