About the Project:
There is a leading learning management system designed for corporate onboarding, development, and assessment. Offering flexible integration capabilities, it serves as an e-learning platform for training, certification, internal communications, and data analysis. The initial hosting on Hetzner servers posed scalability challenges, leading to a transition from bare-metal to instance-based infrastructure.
Goals of the Project:
- Automate horizontal scaling
- Ensure a highly available and fault-tolerant infrastructure
Challenge:
The surge in popularity for e-learning solutions resulted in increased traffic, raising concerns about maintenance costs and the need for horizontal scaling on the existing Hetzner servers.
Solution:
Kubernetes Implementation:
Leveraged Kubernetes for its autoscaling capabilities, preventing infrastructure failure during high traffic periods.
Ensured fault tolerance, enhancing overall availability and user experience.Instance-based Infrastructure:
Utilized Hetzner instances for a repetitive system, reducing scaling time and simplifying management.Monitoring and Alerting System:
Implemented Prometheus and Grafana for efficient monitoring and alerting, aligning with modern DevOps practices.Centralized Logging System:
Transitioned from ELK Stack to Locky for centralized logging, improving manageability and reducing infrastructure costs.Database Enhancements:
Deployed MySQL DB and Elasticsearch for data navigation.
Implemented ClickHouse for fast data computing and improved application availability.Backup System:
Established a backup system for Elasticsearch and ClickHouse.
Implemented a policy to delete unnecessary or old backups, optimizing system performance and efficiency.Cost Reduction Measures:
Automated deletion of unnecessary backups.
Utilized auto-scaling to ensure efficient resource usage, reducing infrastructure maintenance costs.
Technologies Used:
- Kubernetes
- Prometheus
- Grafana
- Locky (Logging System)
- MySQL DB
- Elasticsearch
- ClickHouse
- Auto-scaling solutions