We are seeking a skilled and motivated Data Engineer with expertise in Google Cloud Platform tools such as BigQuery, Composer, Dataproc, and ETL processes. The ideal candidate will have a strong background in designing, implementing, and optimizing data pipelines and workflows. You will play a key role in ensuring our data infrastructure is scalable, efficient, and reliable to support our data-driven decision-making processes.
Key Responsibilities:
- Design and Development:
- Develop, construct, test, and maintain scalable data pipelines using Big Query, Composer, Dataproc, and other GCP tools.
- Design and implement ETL processes to extract, transform, and load data from various sources into the data warehouse.
- Ensure the architecture supports business requirements and allows for easy reporting and analysis.
- Data Management:
- Monitor and manage data workflows and pipelines to ensure data quality, integrity, and reliability.
- Perform data profiling and analysis to identify and address data quality issues.
- Optimization and Performance:
- Optimize ETL processes and data pipeline performance to handle large-scale data efficiently.
- Implement best practices for data governance, security, and compliance.
- Collaboration
- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Troubleshooting and Support:
- Identify, diagnose, and resolve data-related issues and discrepancies.
- Provide support and troubleshooting for data pipelines and ETL processes.
Minimum of 5 years of experience as a Data Engineer or in a similar role.
Proven experience with Google Cloud Platform, specifically Big Query, Composer, Dataproc, and GCP Tools.
Strong experience with ETL processes and tools.
