Responsibilities:
- Design, develop, and manage data pipelines and ETL (Extract, Transform, Load) processes using AWS and Azure services to efficiently extract data from various sourcesand load it into data storage systems.
- Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions that leverage both AWS and Azure cloud capabilities.
- Build and maintain data warehouses, data lakes, and other data storage solutions on AWS and Azure, ensuring data organization and accessibility.
- Implement data validation, cleansing, and transformation processes to ensure high data quality and integrity.
- Optimize data processing systems for performance, scalability, and reliability on both AWS and Azure cloud platforms.
- Monitor, troubleshoot, and resolve issues in data pipelines and systems to ensure uninterrupted data flow.
- Implement security and compliance measures to safeguard sensitive data throughout the data lifecycle on both cloud platforms.
- Stay up-to-date with the latest developments in data engineering technologies on AWS and Azure and recommend relevant tools and solutions.
- Document data infrastructure, processes, and workflows to facilitate knowledge sharing and future reference.
- Collaborate with DevOps and IT teams to deploy and operate data pipelines and systems seamlessly on both cloud platforms.
- Evaluate and select data tools, technologies, and frameworks that align with both AWS and Azure environments.
Requirements:
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- Proven experience as a Data Engineer or in a similar role, with a strong understanding of data engineering concepts and best practices.
- Proficiency in programming languages such as Python, Java, Scala, or similar languages.
- Extensive hands-on experience with AWS services like S3, Redshift, Glue, EMR, Lambda, and relevant Azure services.
- Familiarity with data warehousing solutions on both platforms, such as Amazon Redshift,Google BigQuery, and Snowflake.
- Experience with data processing frameworks like Apache Spark, Apache Flink, or similar technologies on both AWS and Azure.
- Strong knowledge of SQL and NoSQL database systems, data formats (JSON, Parquet, Avro), and data serialization.
- Ability to troubleshoot complex data issues and implement effective solutions on both AWS and Azure cloud environments.
- Excellent communication skills and the ability to collaborate effectively within crossfunctional
- teams.
- Experience with DevOps practices, CI/CD pipelines, and version control systems (e.g.,Git) is advantageous.
- Relevant certifications in AWS (e.g., AWS Certified Data Analytics - Specialty) and Azure (e.g., Azure Data Engineer Associate) are a plus.
