I'm working on a monitoring tool (on and off for a few years now).
It includes all concepts I have learned over the years, for example:
A good set of options to relay information on state changes, or each time. ONBROKEN only get's called once, ONERROR multiple times, until it is fixed..
The "yeah, the website is down until you have 2 measurements of down..." means I allow for ONDEGRADED
Of course, there are maintenance things, like looking for a if-this-file-exists-we-do-not-escalate (unless this file is older than xxx, then delete it and just monitor), or calculating weekends.
The thing I am proud of adding is decay. The first few minutes an error is important, but if it's down for a day... meh, do not propagate the error each cycle, instead, just send an hourly reminder (just in case the event is closed instead of increasing the duplicate counter in a monitoring DB).
This way, the monitoring does not overload the central server with monitoring events, but still runs locally each cycle, so it is able to react fast.
It also has timezones per monitoring check, so you can monitor websites in Singapore (that do not have summer/winter time) without having to change your active hours each 1/2 year...
It features TEST dependencies, so that if your Oracle process is down, you will not need to check for Oracle data-consistency.
And the configuration file is ini based, with weird injections (so you can define a test with minimal parameters (only the process name), and the rest gets added automatically (the severity, the error action, etc)). This implementation of ini allows for multiple same named fields, which get converted into an array and are executed in that order.
It's honed like a Damascus steel sword, keeping the code small (under 40k) and passing it through
NYT Profiler many times to get the optimal speed.
It is used for when monitoring tools charge per monitoring item, you can collate the tests into one errormessage and save some money. Or when you have an Active-Active cluster, where components can run dynamically on either side of the nodes and need intelligent checks that can communicate with each other instead of going through central rules "if down here, but up there, then drop down event".
It also divides the responsibilities of defining good monitoring to the application teams, and leaving the responsibility for the default OS stuff to the monitoring team, while still being able to throttle the application monitoring, if required.
Also, the application team can log into a system, list a single directory, and see the status (there is a read-only webserver or should I say "*AAS", which implements
418 I'm a teapot, but it's not really production ready (security was bolted on), although it was deployed successfully)
Code:
[TEST_EXAMPLE]# a simple test example
MY_FILE=/tmp/ff.txt
RUNCHECK=` ! -f "${:MY_FILE}"
ONOK= echo "yes, test ${:} says file ${:MY_FILE} exists"
ONERROR= echo "no, file ${:MY_FILE} does not exist"
ONERROR= echo "yeah, and this gets executed too if an error is encoutered"
.
