Responsibilities:
- Engage in both exploratory data analysis to identify trends and data patterns
- Perform data preparation and data cleaning part of data migration project.
- Understand Source data and support Source to Target mapping rules
- Design and implement the ETL framework for the data warehouse
- Build and maintain DBT pipelines to transform source data into reporting data models for our visualization team
- Build the Data Ingestion framework and ingest data from source files to load into Staging layer using DBT
- Code orchestration pipelines leveraging Python to connect data ingestion, DBT, Power BI
- Perform data profiling and identify any data quality issues and report to source systems
- Using best practices build the ETL components as per mapping rules to move data into ODS; SOR layer and Data marts
- Ensure the ETL jobs are set up; configured and scheduled
- Develop data pipeline observability test scripts in DBT to monitor anomaly and freshness, and detect data quality issues
- Perform Unit testing on ETL components built and provide unit test results. Support System Integration testing and User Acceptance testing and fix any issues
- Proven experience as a Data Engineer, with a focus on ETL development, Azure Data Factory, and DBT Cloud.
- Strong proficiency in SQL and experience working with relational databases.
- In-depth knowledge of data modeling and data warehousing concepts.
- Experience with cloud-based data platforms, particularly Azure.
- Certification in Azure Data Factory or DBT.
