Responsibilities
- Design, build, and maintain data pipelines to ingest, process, and transform large volumes of data from various sources into our data warehouse.
- Develop and optimize our data warehousing solutions using Snowflake, ensuring data is structured for fast and efficient querying and analysis.
- Utilize Hadoop and Databricks for processing large datasets, implementing ETL processes, and ensuring data is readily available for analysis and reporting.
- Implement data validation, cleansing, and enrichment processes to ensure the accuracy and reliability of the data.
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver data solutions that meet their needs.
- Continuously monitor and optimize the performance of data pipelines and storage solutions, identifying and addressing bottlenecks or inefficiencies.
Requirements:
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- 3+ years of experience in data engineering or a related field.
- Proven experience with Snowflake, Databricks, and Hadoop is mandatory.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud.
- Strong SQL skills and experience with database management.
- Proficiency in programming languages such as Python, Java, or Scala.
- Strong understanding of data warehousing concepts and ETL processes.
- Experience with data modeling and schema design.
- Familiarity with big data processing frameworks like Apache Spark.
- Knowledge of data governance, security, and compliance best practices.
- Experience with other data platforms and tools such as Redshift, BigQuery, or Apache Kafka.
