Job Description:
We are looking for a skilled Data Engineer with 8+ years of experience in handling databases and data engineering projects in the Banking, Aviation, and Telecommunication industries. The ideal candidate will have expertise in evaluating technology stacks for building analytics solutions on the cloud and designing technology roadmaps for data ingestion, data lakes, data processing, and visualization.
Key Responsibilities:
1. Data Ingestion and Processing:
• Extract structured and semi-structured raw data from multiple sources using tools like Pyspark, Hive, and Sqoop.
• Develop, test, and maintain data infrastructure, ingestion pipelines, and data stores.
2. Data Lake and Cloud Solutions:
• Design and implement data lakes and analytics solutions on cloud platforms such as AWS (S3, EC2, EMR, RDS).
• Build and maintain data pipelines using CI/CD methodologies, Docker containerization, and Git version control.
3. Database Management:
• Setup and construct database models and architectures using PostgreSQL, Oracle, and other relational databases.
• Optimize queries for faster report generation and efficient data processing.
4. ETL Processes:
• Design and manage ETL processes for data migration and integration using tools like Informatica.
• Perform data cleansing and harmonization to ensure data quality and consistency.
5. Big Data Technologies:
• Utilize Hadoop ecosystem tools (Hadoop, Hive, Spark) for big data processing and analysis.
• Optimize Hive SQL queries and Spark jobs for performance improvements.
6. Data Visualization and Reporting:
• Create dashboards and reports using tools like SAS, Kibana, and Amazon Redshift.
• Implement real-time transaction monitoring and fraud detection using visualization techniques.
7. Technical Design and Problem Solving:
• Develop technical designs that meet system objectives while minimizing operational impacts.
• Provide solutions for complex data engineering problems and handle batch failures.
8. Collaboration and Communication:
• Work with business analysts to gather requirements, validate data, and perform integration testing.
• Communicate effectively with team members and stakeholders to ensure project success.
• Experience: 8+ years of experience in data engineering and database management.
Skills:
• SQL, PL/SQL, Python
• Hadoop, Hive, Sqoop, Spark
• AWS (S3, EC2, EMR, RDS)
• SAS, Machine Learning, Data Modeling
• Oracle, PostgreSQL, MSSQL, Teradata
Additional Skills:
• Strong analytical, problem-solving, and organizational abilities.
• Quick learner with a keen interest in staying updated with industry trends.
• Excellent written and oral communication skills.
• Ability to work independently and as part of a team.
