Table of Contents |
---|
opEvents provides two mechanisms to handle repeated event occurrences in a practical fashion, namely stateful event deduplication and programmable event suppression.
...
- The
stateful
property indicates the type of state source, and is a free-form string.
For example, if the event is related to an interface,stateful
should be set to "Interface"; if it's about a service, the value "Service" would be most appropriate, etc.
You may use any state source type you want in your parser rules, but avoid overloading already existing ones like "Node" and "Interface". - The
element
property indicates which (of potentially many) state sources the event relates to.
For state type interface, a unique interface identifier should be used (i.e. the ifDescr).
Like above your parser rules may capture or set the element to anything you desire, as long as the combination of node name,stateful
andelement
is a suitably unique identifier for the particular stateful thing you're trying to track. - The
state
property indicates whether the observed state is "good" or "bad".
opEvents treats the valuesup
,ok
,good
,normal
orclosed
as "good", anything else as "bad".
This comparison is made case-insensitively, i.e. "Good
" will work just as well as "OK
".With the release of opEvents 4.5.0, you can now define custom values for the "good" state by adding the
opevents_up_states
setting in opCommon.json underopevents
.
This new setting allows you to dynamically configure additional "good" states to suit your specific requirements, providing greater flexibility and customization in state management.
For example:"opevents_up_states" : ["up","ok","good","normal","closed","online","added","enabled"]
For state tracking opEvents then combines the node name and the values of stateful
and element
into a lookup key, and associates that key with the state
value.
Any repeat events with the same lookup key and the same state value are ignored.
Stateful Deduplication, Forwarded Events and Reorder Protection
Anchor | ||||
---|---|---|---|---|
|
If you use an Event Action or Escalation Policy with create_remote_events
to forward events to another opEvents server elsewhere, then you might occasionally find that such forwarded events arrive out of order, i.e. an earlier 'down' event might be received after the later 'up' event. This can happen because of network congestion, action processing on the sending side being asynchronous and subject to process limits and similar reasons.
Out of order reception of stateful events can cause state desynchronisation at the receiving server, as the up event would be processed first and thus be deduplicated and discarded, while the down event later on causes a transition to state down which isn't cleared.
opEvents versions 2.4.2 and newer provide a reorder protection mechanism to handle such out of order situations better - which comes at the cost of temporarily delaying the processing of some forwarded stateful events.
To enable reorder protection, two steps need to be taken:
- you need to set the configuration property
state_reorder_window
to a positive number (e.g. 30) on the receiving server, - and you must make sure that your forwarded events do carry an
authority
property, to denote the event as originating from a remote authoritative source.
If both of these conditions are met, opEvents on the receiving server will temporarily postpone processing of a forwarded stateful event, if the event would be discarded by stateful deduplication.
This allows earlier but externally delayed related events to enter the processing queue in the correct sequence, if any such do arrive within the configured time window after the out-of-order postponed event.
If a state-changing remote event does arrive within the time window configured by state_reorder_window
, then the correct sequencing of transitions is restored and processing of postponed events resumes immediately. Otherwise, processing resumes after the time window elapses.
The state_reorder_window
should not be set too large as it causes undesirable event processing delays; a value of 10 to 30 seconds should suffice in most environments.
Programmable Suppression
...