Overview of Role:
This data engineer's main role is to create and maintain data pipelines that facilitate data flow from various sources to the secured data storage systems.
Responsibilities:
- Responsible for developing and maintaining data pipelines for efficient data extraction, transformation, and loading (ETL) processes.
- Responsible for designing and optimizing data storage solutions, including data warehouses and data lakes.
- Ensure data quality and integrity through data validation, cleansing, and error handling.
- Collaborate with data specialists and software engineers to understand data requirements and deliver relevant data sets (e.g., for business intelligence).
- Implement data security measures and access controls to protect sensitive information.
- Automate and improve data processes and workflows for scalability and efficiency.
- Monitor data infrastructure for performance and reliability to address issues promptly.
- Participate in data governance and compliance efforts to meet regulatory requirements.
- Continuously optimizing data architecture to support the company's evolving data needs.
- Collaborate with cross-functional teams to drive data-driven decision-making within the organization.
- Document data pipelines, processes, and best practices for knowledge sharing.
- Keeping abreast of industry trends and emerging technologies in data engineering.
Requirements :
- Degree in Computer Science or related field.
- Proficient in data modeling and database management systems, both relational (e.g., MySQL, PostgreSQL) and non-relational (e.g., MongoDB, Cassandra).
- Strong programming skills (in egs, Python, Java, or Scala).
- Experience with big data technologies like Hadoop, Spark, and Kafka.
- Experience with ETL (Extract, Transform, Load) processes.
- Familiar with data integration and ETL tools, such as Talend, Informatica, or Apache NiFi.
- Familiar with data warehousing and cloud platforms (e.g., AWS, Azure, or Google Cloud).
- Strong problem-solving and analytical skills.
- Good communication and collaboration abilities.
- Adaptability and a willingness to learn new technologies and techniques
