EMU—Event Management Utility

Event management is a backbone of every enterprise management system, since it is the flow of events that describes various activities in the enterprise. By coordinating or acting on those events, enterprise starts being managed.

One of us, Jarra, has been contracting to a large computer company that was deploying Unicenter TNG for monitoring outsourced systems. Most of the systems were UNIX-based. Unicenter TNG agents, as is well known, are SNMP-based. After a considerable amount of time was spent tweaking the agents to come close to what he had decided to monitor and how to go about it, he arrived at the following conclusions:

We began thinking about how to accomplish system monitoring with ultra-thin clients, while maintaining their status at the event manager. That was when we thought of using a principal parameter we coined “time-to-live”. At that very moment, EMU (Event Management Utility) was born. Each event message sent to EMU must have time-to-live set. It determines how long the message is kept in the database before it is deleted. If time-to-live is slightly more than the polling interval, the message will stay in the database until we stop resending it. In other words, the message will be in the database for the duration of an alarm only.

EMU is a comprehensive event management system with no limitations on what it can do for you. It harnesses the full power of Linux/UNIX utilities and scripting languages. Remember, software is here for us, and not the other way around.

Event management systems usually consist of four components: monitoring manager, monitoring agents, event manager and event console.

Monitoring agents typically poll a resource, such as a file system, at regular intervals in order to track its status changes. These changes are communicated to the manager. The manager processes incoming resource status changes and either forwards them to the event manager or updates the business rules it maintains. The monitoring manager can poll for resource status information as well.

Resources may be of any kind, such as file systems, processes, disk, swap, applications or logged-in users.

Some monitoring agents, such as Unicenter TNG, are built around SNMP. In those systems, agents use traps to notify monitoring managers of status changes. Traps are sent using the UDP protocol, hence their delivery is not guaranteed. This drawback has to be made up for by regular polling by the manager, thereby collecting resource status independent of traps.

SNMP agents maintain status information of each resource they monitor. This information is typically maintained by the monitoring manager as well. Both agents and the manager keep resource status stored in a database. This is a key design feature of SNMP-based monitoring. Let us consider a file system example. If a threshold is set at 90%, an alarm is raised on exceeding this threshold. The status of the resource is changed to critical. It must stay in the critical state until the file system is cleaned up. Then the status is changed back to normal.

A need to maintain resource status and resource hierarchy results in rather complex SNMP agent coding. As a result, it is not trivial to extend existing agents. Furthermore, C coding skills and access to the source code are necessary.

The monitoring agent forwards events to the event manager, which usually performs some kind of event processing, such as matching messages against predefined strings and triggering an action in case of a match. Unicenter TNG provides a language for matching messages and taking action on them. The language is extremely simple, and thus inadequate for performing more complex correlations and actions.

The event console is the front end for administration staff. It allows us to “view” events. Its design determines its usability and efficiency. Events representing one alarm should be displayed as one event, rather than getting a separate visible event at each poll. To keep track of events easily, one strives to keep their display representations to a minimum. Otherwise, they scroll out of view before IT staff can take note of them.