Talks > 20-21/04/2016 Jo Vanvoorden

A view inside your HPC infrastructure with elasticsearch, logstash and kibana

At KU Leuven we set up a complete elasticsearch – logstash – kibana stack to monitor the hpc infrastructure.

Logfiles are shipped to a central location where they are parsed and indexed to our central elasticsearch cluster. This information is then presented trough a kibana web frontend. The web frontend allows system administrators and support engineers to easily search the logfiles of all compute nodes and login nodes. Job schedule information, abnormal behavior, executed commands, loaded modules, and many more information is now searchable through a simple web interface. Making a visual representation of the results is now only a matter of a few mouse clicks. Dashboards can be created for reporting and debugging purposes. All information can be accessed and used in real time.

This talk will give an overview of the technical setup and design constraints. Deployment of the infrastructure using our puppet and docker based infrastructure and to end it a real time demo of the analytics.

Related Talks

Visit our forum

One of the main goals of this project is to motivate new initiatives and collaborations in the HPC field. Visit our forum to share your knowledge and discuss with other HPC experts!

About us

HPCKP (High-Performance Computing Knowledge Portal) is an Open Knowledge project focused on technology transfer and knowledge sharing in the HPC, AI and Quantum Science fields.

Promo HPCNow