Senior Data Engineer
Institution: National Healthcare Group
Family Group: Administration
The Senior Data Engineer leads the creation, maintenance, and strategic development of our advanced data analytics infrastructure, which is vital to the hospital's data-driven decision-making processes. This role oversees the acquisition, storage, retrieval, and optimization of data from various sources to support clinical, operational, and research initiatives. The position directly supports the work of the Group Chief Data Officer (GCDO) for Data Analytics and provides leadership to the data engineering team.
Leadership and Management:
• Lead and mentor a team of data engineers, fostering their professional growth and ensuring high-quality deliverables.
• Develop and implement best practices, standards, and architectural decisions for data engineering processes.
• Collaborate with cross-functional leadership to align data engineering strategies with organizational goals.
• Drive innovation in data engineering practices and technologies within the organization.
Strategic Planning and Execution:
• Develop and execute the long-term vision for the data engineering function, aligning with the institution's overall data strategy.
• Identify opportunities for process improvements and automation to enhance team efficiency and data quality.
• Manage resource allocation and capacity planning for data engineering projects.
Data Sourcing and Collection:
• Oversee the identification and evaluation of source systems.
• Guide the team in gathering requirements on frequency, volume, and types of data.
• Ensure robust data privacy and compliance measures are in place, liaising with legal and compliance teams as necessary.
Data Ingestion and Architecture:
• Lead the design and implementation of scalable, efficient data ingestion pipelines using Informatica IDMC, Spark, and Python.
• Establish architectural standards for reliability and fault tolerance of ingestion pipelines.
• Oversee data transformation strategies during ingestion.
• Manage vendor relationships and collaborations for data ingestion projects.
Data Lake Management and Integration:
• Direct the structuring of data in the DataBricks Lakehouse to ensure optimal usability and performance.
• Oversee data partitioning, indexing, and archival strategies.
• Lead integration efforts with DataBricks Lakehouse, ensuring seamless accessibility for data scientists and analysts.
Data Quality, Validation, and Security:
• Establish comprehensive data quality frameworks and validation protocols.
• Implement and oversee security measures, including access controls, data encryption, and regular audits.
• Collaborate with compliance officers to ensure adherence to healthcare data regulations.
Performance Management and Optimization:
• Set and monitor KPIs for data ingestion pipeline performance.
• Lead optimization efforts to meet and exceed SLAs.
• Conduct regular performance reviews and implement improvement strategies.
Strategic Collaboration and Communication:
• Serve as the primary liaison between the data engineering team and other departments, including data science, business analytics, and IT.
• Present data engineering strategies, progress, and challenges to senior management.
• Foster a culture of knowledge sharing and continuous learning within the team.
Qualifications:
• 8+ years of experience in data engineering, with at least 3 years in a leadership role.
• Strong technical expertise in data engineering tools and technologies, including Informatica IDMC, Spark, Python, and DataBricks.
• Excellent communication and interpersonal skills, with the ability to influence and collaborate across all levels of the organization.
Nice to Have:
• Proven track record of successfully leading data engineering projects in a healthcare setting.
• Advanced knowledge of healthcare data standards, privacy regulations, and compliance requirements
