We are looking for a skilled Data Engineer with a strong background in AWS technologies. This is a remote role supporting us on a 6-month extendable contract basis.
The ideal candidate will have experience in Python and Airflow with a deep understanding of Apache Airflow and its integration within the AWS ecosystem, enabling efficient data pipeline orchestration and management.
Roles and Responsibilities:
- Design, develop, and manage sophisticated data pipelines using Python to ensure efficient data processing and orchestration.
- Work with cross-functional teams to understand data requirements and create effective solutions within the AWS ecosystem.
- Implement data integration and transformation processes to maintain high performance and reliability of data pipelines.
- Enhance and optimize existing data pipelines and Airflow configurations for better efficiency, scalability, and maintainability.
- Diagnose and address issues with data pipelines to ensure continuous operation and minimal downtime.
- Partner with Data Scientists and Analysts to facilitate smooth data access and processing for analytics and reporting.
- Utilize AWS services such as S3, Glue, EMR, Redshift, and related technologies to design and refine data infrastructure.
- Develop and keep updated documentation for data pipelines, processes, and system architecture.
Qualifications:
- Bachelor's degree in computer science, Engineering, or a related field.
- Proficiency in Python and SQL for data processing and manipulation.
- 3 to 6 years of experience in data engineering, particularly with Apache Airflow and AWS technologies.
- Strong understanding of AWS services, including S3, Glue, EMR, Redshift, and AWS Lambda.
- Experience in optimizing and scaling data pipelines for performance and efficiency.
- Excellent problem-solving skills and the ability to thrive in a fast-paced, collaborative environment.
- Solid grasp of data modeling, ETL processes, and data warehousing concepts.
- Strong communication skills with the capability to explain technical concepts to non-technical stakeholders.
Argyll Scott Consulting Pte Ltd
