At KU Leuven we set up a complete elasticsearch – logstash – kibana stack to monitor the hpc infrastructure.
Logfiles are shipped to a central location where they are parsed and indexed to our central elasticsearch cluster. This information is then presented trough a kibana web frontend. The web frontend allows system administrators and support engineers to easily search the logfiles of all compute nodes and login nodes. Job schedule information, abnormal behavior, executed commands, loaded modules, and many more information is now searchable through a simple web interface. Making a visual representation of the results is now only a matter of a few mouse clicks. Dashboards can be created for reporting and debugging purposes. All information can be accessed and used in real time.
This talk will give an overview of the technical setup and design constraints. Deployment of the infrastructure using our puppet and docker based infrastructure and to end it a real time demo of the analytics.
