Responsibilities:
- Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization platforms.
- Perform data extraction, cleaning, transformation, and flow, including web scraping as required.
- Build, launch, and maintain efficient, reliable large-scale batch and real-time data pipelines using data processing frameworks.
- Collaborate with cross-functional teams, including Project Managers, Data Architects, Business Analysts, and others, to develop scalable data-driven products.
- Develop backend APIs and manage databases to support applications.
- Work in an Agile environment, practicing Continuous Integration and Delivery (CI/CD).
Experience and Skills Required:
- Proficiency in data cleaning and transformation tools (e.g., SQL, pandas, R).
- Expertise in building ETL pipelines using tools such as SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, and AWS Lambda.
- Strong database design skills and experience with various databases (e.g., SQL, PostgreSQL, MongoDB, MySQL).
- Familiarity with cloud technologies (AWS, Azure, Google Cloud).
- Knowledge of system design, data structures, and algorithms.
- Proficiency in scripting languages (e.g., SQL, Python).
