Monit is the excellent monitoring app in the bag of any seasoned Linux admin. Here is one way I use Monit for an interactive health check to restart a down web service.
Scenario: I have a web app that I want to keep running and self-restart if there are any bad HTTP status codes.
This web app is running via supervisor to facilitate various environmental variables and to make it a service, so I want to restart supervisor to restart my app.
It is possible to use Monit to monitor the HTTP status codes, and if one is an error status code, restart supervisor thereby restarting my app.
(For the HTTP status code checks, you need at least Monit 5.8.)
Install and configure monit locally on the web server itself. Create a check to monitor the process of the service you wish to restart and a host check as the trigger. Below, I am monitoring the supervisor process which will restart if the host check fails with any status code above 400.
check process supervisor with pidfile /var/run/supervisord.pid start program = "/etc/init.d/supervisor start" stop program = "/etc/init.d/supervisor stop" if failed host lilybin.com port 80 protocol http and status < 400 then restart
How does it work?
Works great! Here is an example log snipit when monit had to restart the service.
[UTC Oct 16 23:16:15] error : 'supervisor' failed protocol test [HTTP] at INET[lilybin.com:80] via TCP -- HTTP error: Server returned status 404 [UTC Oct 16 23:16:15] info : 'supervisor' trying to restart [UTC Oct 16 23:16:15] info : 'supervisor' stop: /etc/init.d/supervisor [UTC Oct 16 23:16:16] info : 'supervisor' start: /etc/init.d/supervisor [UTC Oct 16 23:20:17] info : 'supervisor' connection succeeded to INET[lilybin.com:80] via TCP
The monit wiki has many more good examples: