The engineer is responsible for designing, implementing, and maintaining Elasticsearch clusters and related components in enterprise distributed cluster environment. His role involves working with large-scale data sets and optimizing search and indexing performance.
Responsibilities
- Cluster Design and Deployment: Designing and deploying Elasticsearch clusters based on specific requirements, including hardware, network, and security considerations.
- Configuration and Optimization: Configuring Elasticsearch settings, including index mappings, shard allocation, and cluster settings, to optimize performance and ensure efficient resource utilization.
- Data Ingestion and Indexing: Developing strategies for efficient data ingestion into Elasticsearch, including handling large data volumes, implementing data pipelines, and optimizing indexing processes.
- Search and Query Optimization: Tuning search queries, relevance scoring, and aggregations to improve search performance and response times.
- Monitoring and Troubleshooting: Implementing monitoring solutions to track cluster health, performance, and resource utilization. Identifying and resolving issues related to indexing, search, and cluster stability.
- Scalability and High Availability: Planning and implementing strategies for scaling Elasticsearch clusters horizontally or vertically to handle increased data volumes and user traffic. Ensuring high availability and fault tolerance through replication and data redundancy.
- Security and Access Control: Implementing security measures, such as authentication, authorization, and encryption, to protect Elasticsearch clusters and data from unauthorized access.
- Collaboration and Documentation: Collaborating with cross-functional teams, including developers, DevOps, and data engineers, to integrate Elasticsearch into applications and systems. Documenting cluster configurations, deployment processes, and troubleshooting guidelines.
- Performance Analysis and Optimization: Conducting performance analysis, identifying bottlenecks, and implementing optimizations to improve overall system performance and efficiency.
- Stay updated with Elasticsearch: Keeping up-to-date with the latest features, enhancements, and best practices in Elasticsearch and related technologies.
- Collaborate and work with the various stakeholder to develop technical solutions, plans and configuration for the observability capability and develop run book automation with the objective to achieve self-healing capability.
- Perform integration and develop full interoperability capabilities with various operations management platform including change management, service management, privileged access management system, etc.
- Provide support to existing monitoring system and bridge the transition to an AI-enabled observability platform.
The engineer plays a crucial role in ensuring the smooth operation and performance of Elasticsearch clusters, enabling efficient search and analysis capabilities for organizations dealing with large-scale data sets.
Requirements
- Possesses at least a Bachelor's Degree in Computer Science/Information Technology or equivalent.
- Solid experience on managing enterprise distributed cluster environment.
- At least 5 years of hands-on working experience in:
- Enterprise logging platform (such as ElasticSearch, Splunk)
- Automation scripting (such as Ansible, Terraform, Powershell, etc)
- Demonstrate strong knowledge in both application and infrastructure domain with ability to develop automation scripts.
- Possess good technical knowledge in implementing, troubleshoot, performance tuning of hardware, operating system and system services.
- Strong analytical skill, creative and thinking out of the box in problem solving.
- Excellent command of written and spoken English.
- A good team player and able to work effectively at all levels of an organization with the ability to influence others to move towards consensus.
- Proven ability to operate under pressure and meet challenging deadlines with minimum supervision.
