We are seeking a talented and a motivated Senior Data Engineer to join our dynamic team. As a Senior Data Engineer, you will be responsible for designing, implementing, and maintaining scalable data pipelines and infrastructure to support our data-driven initiatives. You will collaborate closely with cross-functional teams to understand data requirements, optimize data models, and ensure the reliability and performance of our data systems.
Key Responsibilities:
Design, develop, and maintain robust data pipelines to ingest, process, and transform large volumes of structured and unstructured data from various sources (e.g., Database, API, SFTP).
Build and optimize data models and schemas to support efficient data storage, retrieval, and analysis.
Implement data integration solutions to consolidate data from disparate systems and sources.
Work closely with Data Scientists and Analysts to understand their requirements and provide them with clean, reliable, and well-structured data.
Collaborate with DevOps and Infrastructure teams to deploy and manage data infrastructure in cloud environments (e.g., AWS, Azure).
Familiar with the concept of data security to ensure the data pipelines are secured and PII (Personally Identifiable Information) data are encrypted.
Monitor and optimize the performance, scalability, and reliability of data systems to ensure high availability and low latency.
Design and develop MLOps production pipelines; provide technical support to data scientists/ML engineers by getting their ML/DL models deployed at scale and meeting SLAs on both cloud and on-premises GPU and CPU instances.
Develop and maintain documentation, standards, and best practices for data engineering processes and technologies.
Stay current with emerging technologies and trends in data engineering and contribute to the continuous improvement of our data architecture and practices.
Explore, evaluate, and champion the introduction of next-generation technologies in the data-ingestion workflow. Participate in project planning and provide technical guidance on cloud architecture for data projects.
Requirements
Requirements:
BS in Computer Science or other related discipline is required. Advanced degrees in Computer Science (PhD, MS) are highly desirable.
Proven experience (3 years) working as a Data Engineer or in a similar role.
Strong proficiency in programming languages such as Python. Conversant with data structures and algorithm design.
Hands-on experience with distributed computing frameworks and big data technologies such as Spark, Kafka, etc.
Proficiency in SQL and experience with relational databases (e.g., Azure SQL DB, AWS Redshift).
Experience with data modelling, ETL/ELT processes, and data warehousing concepts.
Familiarity with cloud platforms and services (e.g., AWS S3, AWS Step Functions, AWS Airflow, Azure Data Logic Apps, Azure Blob Storage).
Experience with setting up SSO using AD and implementing RBAC (Role Based Access Control) for data security.
Excellent problem-solving skills and attention to detail.
Effective communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.
Experience with version control systems (e.g., Git) and CI/CD pipelines is a plus.
