Job Description:
We are on the lookout for a talented Data Engineer to join our team. In this role, you'll be at the forefront of designing, developing, and maintaining cutting-edge data solutions. Your expertise in data pipelines, ETL processes, and big data technologies will be crucial as you optimize data generation, collection, and processing. You'll be expected to leverage your deep knowledge of Python, PySpark, and Cloudera Hadoop to drive innovation and provide solutions to complex data challenges.
Key Responsibilities:
Design and Develop Data Solutions:
Create and maintain data pipelines and ETL processes to efficiently handle data generation, collection, and processing.
Optimize Data Processes:
Develop innovative solutions to enhance data flow and system performance.
Manage Data Pipelines:
Build and maintain robust data pipelines, ensuring data quality and integrity across systems.
Implement ETL Processes:
Execute advanced ETL processes for effective data migration and deployment.
Collaborate Across Teams:
Work closely with cross-functional teams to identify and resolve data quality issues.
Lead and Mentor:
Take on leadership responsibilities, making key decisions for your team and guiding them in problem-solving across multiple teams.
Required Qualifications:
Expertise in Python:
Proven proficiency in Python for data engineering tasks.
Cloudera Hadoop:
Expert-level experience with Cloudera Hadoop, including Hive and Impala.
Data Engineering Skills:
Advanced proficiency in data engineering, including building and maintaining data pipelines.
PySpark Proficiency:
Deep experience with PySpark for big data processing.
Big Data Engineering:
Hands-on experience with big data engineering jobs using Python, PySpark, and Linux.
ETL Processes:
Experience in implementing and managing advanced ETL processes.
Data Solutions:
Track record of developing and optimizing data solutions for large-scale environments.
Preferred Qualifications:
Cloudera Data Platform:
Experience with Cloudera Data Platform will be highly regarded.
System Implementation:
Proven success in implementing data systems using Hive and Impala.
Cross-Functional Collaboration:
Strong ability to work with various teams to tackle data quality and integration challenges.
