DATA ENGINEER
Responsibilities:
- Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization systems.
- Perform data extraction, cleaning, transformation, and flow, including web scraping.
- Design, build, launch, and maintain efficient and reliable large-scale batch and real-time data pipelines using data processing frameworks.
- Integrate and collate data silos in a scalable and compliant manner.
- Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, UX Designers, and Data Analyst to build scalable data-driven products.
- Develop backend APIs and work on databases to support applications.
- Work in an Agile Environment that practices Continuous Integration and Delivery.
- Participate in pair programming and code review processes with fellow developers.
Experience and Skills Needed:
- Proficient in general data cleaning and transformation (e.g., SQL, pandas, R, etc.).
- Experienced in building ETL pipelines (e.g., SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Eventbridge, AWS Glue, Spring).
- Proficient in database design and various databases (e.g., SQL, PostgreSQL, AWS S3, Athena, MongoDB, Postgres/GIS, MySQL, SQLite, VoltDB, Cassandra, etc.).
- Experience with cloud technologies (e.g., AWS, Azure, Google Cloud Platform).
- Passionate about data engineering in a big data environment using cloud platforms.
- Experience with building production-grade data pipelines, ETL/ELT data integration.
- Knowledge in system design, data structures, and algorithms.
- Familiar with data modeling, data access, and data storage infrastructure (Data Mart, Data Lake, Data Virtualization, Data Warehouse).
- Understanding of REST APIs and web requests/protocols in general.
- Familiarity with big data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ).
- Comfortable with W3C Document Object Model and customized web scraping (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js, etc.).
- Comfortable in at least one scripting language (e.g., SQL, Python).
- Experience in both Windows and Linux development environments.
- Interest in being the bridge between engineering and analytics.
4o mini
