System monitoring is one of the topics that has got a lot of attention lately.
A lot of options already exists in the open but every-time I search a for a system monitoring tool I am presented with options suitable for large scale cluster of machines, capturing metrics over time, showing trends of resource utilization over time and requiring a storage system and moderate to heavy installation. These systems are good for Ops teams interested in managing the stable operation of systems and optimization of resource utilization over time.
My requirements on the other hand are more on the side of debugging and require a real time peak in to the monitored system. To be frank my favourite tool for the job has been to use top command on the system to look at the current status of the system and it still one of the invaluable tools linux provides. It provides info on the load average, ram utilization and per process metrics with various selection and sorting options.
More than often I start with a couple of physical machines and start by running virtual machines on them. A quick and easy monitoring tool is what I need to debug. Till now I would have a tmux session with windows running top command over ssh to these physical machines. I am mostly interested in the status VMs (kvm processes) running on the system and the overall health of the hypervisor.
One problem with top is it does not cover the network and disk metrics (which are a good to have feature), but most importantly although top provides the per process stats but I could not find a way to get alerts when a process (KVM instance in my case) starts to go high on resources.
Glances a better top
I recently tried glances and was really impressed with its capability. The most important part is the easy installation steps. Although I created a virtual environment and installed the app within it, it is not a requirement. To install glances all you need to do is to install the python package using pip and that’s it, you are ready to go
pip install glances
It has the same look and feel as top (although not as may selection and sorting option) but what it provides is a complete view of the system in real time including disk and network stats.
In addition you can define process level alerts for resource utilization.
It can act as standalone, client and a server mode. In the standalone mode it can monitor the status of the local system.
While in the server mode (with the -s option) it can be sending statistics to a remote client.
Or you can run it in the web server mode to monitor the system over a web browser(make sure you have bottle install for the webserver mode to work).
glances -w Glances Web User Interface started on http://0.0.0.0:61208/
In the client mode you can create a list of servers that you need to monitor and start the client in the browse mode (with –browse option). The client will present you with a list of monitored server with a summary of their status. You can connect to any of them to get the detailed view.
The client connected to a remote server has the same look and feel as a standalone glances instance
My current installation
I am monitoring three physical machines with glances. I have glances running in the server mode on the physical machines and also in the web server mode (Currently you need to run 2 instances of Glances if you need both server and web server mode, there is a open request to enable both modes on the same instance of Glances)
I have installed glances in a virtual environment to keep its dependencies separated for the rest of the system.
I am running it under supervisord to make sure we have the process always running and restarted after reboot.