Job Description
• Design, develop, optimize, and maintain data architecture and pipelines that adhere to ETL principles and business goals.
• Solve complex data problems to deliver insights that help the organization achieve its goals.
• Code in Python with tools like Apache Spark to build a multi-cluster data warehouse.
• Interact with other technology teams to define, prioritize, and ensure smooth deployments for other operational components.
• Advise, consult, mentor, and coach other data and analytics professionals on data standards and practices.
• Foster a culture of sharing, reuse, design for scale stability, and operational efficiency of data and analytical solutions.
• Codify best practices for future reuse in the form of accessible, reusable patterns, templates, and code bases to facilitate data capturing and management.
• Strong experience in mapping attributes, data profiling , data cleansing , and technical data quality etc.
• Strong experience in Ansi SQL and in-depth knowledge with structured , semi-structured & unstructured data
• Must have good knowledge of data lake and working experience of migration projects in cloud with providers like AWS or Microsoft AZURE or GCP
• Good to have experience in working with No SQL, Spark SQL using AWS Glue, EMR, and columnar data store.
• Good to understand data security features like data masking, data encryption , role based & fine grain access control mechanisms etc.
Qualifications
• 4+ years of relevant experience in data engineering/analytics space
• Expertise in SQL and data analysis and strong hands-on expertise with at least one programming language: Python.
• Strong knowledge in one or more of the following big data tools: Hive, Hadoop Impala, Spark, Kafka
• Strong expertise in ETL, reporting tools, data governance, data warehousing, and hands-on experience.
• Experience developing solutions for cloud computing services and infrastructure.
• Experience developing and maintaining data warehouses in big data solutions.
• Up to date on industry trends within the analytics space from a data acquisition processing, engineering, and management perspective
• Experience in agile development.
• Strong people skills, specifically in collaboration and teamwork
• High level of curiosity, creativity, and problem-solving capabilities
