Job Summary:
We are seeking an experienced Senior Solutions Consultant to lead the design, development, and implementation of large-scale data pipelines and architectures. The successful candidate will have 8+ years of hands-on experience in data engineering, with expertise in Spark, Python, SQL, and cloud-based data integration services.
Responsibilities:
- Design, develop, and implement scalable and efficient data processing systems using technologies such as Databricks, Spark and Azure Synapse.
- Collaborate with cross-functional teams to gather requirements, define data models, and ensure data quality and integrity.
- Develop and maintain complex data pipelines that handle large volumes of data from various sources, including relational databases, NoSQL databases, and cloud storage.
- Implement data integration processes using cloud-based data integration services.
- Design and implement real-time streaming data processing systems using Apache Kafka, Apache Flink, or similar technologies.
- Develop and deploy data quality and validation processes to ensure accuracy and consistency of data.
- Optimize data storage and retrieval performance by designing efficient schema, indexing, and caching strategies.
- Troubleshoot complex data engineering issues and provide technical leadership in resolving them.
Requirements:
- 8+ years of hands-on experience in data engineering with expertise in Spark, Python, SQL, and AWS/Azure/GCP Cloud Data Services.
- Familiarity with cloud-based data services such as Databricks, Azure Synapse, Azure Data Factory, Amazon Redshift, AWS Glue etc.
- Familiarity with data visualisation tools such as Power BI, Tableau, or Databricks.
- Experience in developing and deploying large-scale data processing systems using technologies like Apache Spark, Hadoop, or similar distributed computing frameworks.
- Strong understanding of database concepts, including schema design, indexing, caching, and query optimization.
- Proficiency in programming languages such as Python, Java, or Scala.
- Excellent problem-solving skills with ability to work under pressure to meet deadlines.
- Strong communication and collaboration skills to work effectively with cross-functional teams.
- Experience with cloud-based security, governance, and compliance tools.
