Creating custom SNMP OIDs in InterSystems IRIS

This post is dedicated to the task of monitoring aa IRIS instance using SNMP. Some users of IRIS are probably doing it already in some way or another. Monitoring via SNMP has been supported by the standard IRIS package for a long time now, but not all the necessary parameters are available "out of the box". For example, it would be nice to monitor the number of CSP sessions, get detailed information about the use of the license, particular KPI’s of the system being used and such. After reading this article, you will know how to add your parameters to IRIS monitoring using SNMP.

What we already have

IRIS can be monitored using SNMP. A full list of what’s supported can be found in the files of the <Install_dir>/SNMP/. You should find a file called ISC-IRIS.mib there. In particular, we’d like to know what information we can get about licenses and sessions. The table contains corresponding OID’s provided that the hierarchy starts from the root for InterSystems— 1.3.6.1.4.1.16563

OID	Name	Description	Data type
.4.1.1.1.10	irisSysLicenseUsed	The current number of licenses used on this IRIS instance	INTEGER
.4.1.1.1.11	irisSysLicenseHigh	The high-water mark for licenses used on this IRIS instance	INTEGER
.4.2.15	irisLicenseExceed	A request for a license has exceeded the licenses available or allowed	Trap message
.4.1.1.1.6	irisSysCurUser	Current number of users on this IRIS instance	INTEGER

The package lacks many important parameters, such as, for instance, the number of CSP sessions, license information and, of course, does not have application-specific KPI’s.

Here is an example of what we’d like to know:

The number of CSP users
Limitations of our license in terms of the user count
License expiry date

Let’s also add a few parameters for performance analysis. The parameters themselves are in the package, but we want to know the increment per minute, for example:

The increase of the number of "global" references per minute
The number of executed command per minute
The number of routine calls per minute

How to add "your" parameters

You can rely on the Monitoring InterSystems IRIS Using SNMP document.

The IRIS version of our test instance (IRIS) is 2020.3.0.200.0com. The operating system is Ubuntu 18.04.4 LTS (docker image intersystemsdc/iris-community:2020.3.0.200.0-zpm is used, root user is enabled).

Here is our agenda:

Create a class for collecting metrics.
Register and activate a new class in IRIS using ^%SYSMONMGR.
Create a user MIB using MonitorTools.SNMP class methods. We’ll use 99990 as a temporary PEN (Private Enterprise Number), but will need to register with IANA afterwards. This procedure is free, takes a week or two and requires some email exchange along the lines of "what do you need your own PEN for?".
Start a monitoring service with a connected IRIS subagent.
Use snmpwalk to make sure we have access to all our newly-created OID’s.
Add our OID’s to to a third-party monitoring system. Let’s use Zabbix, for example. Zabbix documentation is available here. Let’s make sure that monitoring is up and running.
Add the start of the system monitor in our TEST namespace to the system startup list.

Let’s now follow the agenda, point by point.

1. Create a class for collecting metrics

The metrics collection class extends %Monitor.Adaptor. In the Terminal we switch to the %SYS namespace and export the hidden Monitor.Sample class:

%SYS>do $system.OBJ.Export("Monitor.Sample.cls","/tmp/Monitor_Sample.xml")
Exporting to XML started on 10/16/2020 09:33:55
Exporting class: Monitor.Sample
Export finished successfully.

Let’s assume that the TEST namespace is our working area. Let’s switch to it and import the Monitor.Sample class here:

TEST>do $system.OBJ.Load("/tmp/Monitor_Sample.xml", "ck")

Now, we create a class that describes the implementation of a monitoring mechanism for the 6 metrics described in the "What we already have" section.

Class monitoring.snmp.Metrics Extends %Monitor.Adaptor
{
/// Give the application a name. This allows you to group different
/// classes together under the same application level in the SNMP MIB.
/// The default is the same as the Package name.
Parameter APPLICATION = "Monitoring";
/// CSP sessions count
Property Sessions As %Monitor.Integer;
/// License user limit
Property KeyLicenseUnits As %Monitor.Integer;
/// License key expiration date
Property KeyExpirationDate As %Monitor.String;
/// Global references speed
Property GloRefSpeed As %Monitor.Integer;
/// Number of commands executed
Property ExecutedSpeed As %Monitor.Integer;
/// Number of routine loads/save
Property RoutineLoadSpeed As %Monitor.Integer;
/// The method is REQUIRED. It is where the Application Monitor
/// calls to collect data samples, which then get picked up by the
/// ^SNMP server process when requested.
Method GetSample() As %Status
{
      set ..Sessions = ..getSessions()
      set ..KeyLicenseUnits = ..getKeyLicenseUnits()
      set ..KeyExpirationDate = ..getKeyExpirationDate()

      set perfList = ..getPerformance()
      set ..GloRefSpeed = $listget(perfList,1)
      set ..ExecutedSpeed = $listget(perfList,2)
      set ..RoutineLoadSpeed = $listget(perfList,3)

      quit $$$OK
}
/// Get CSP sessions count
Method getSessions() As %Integer
{
     // This method will only work if we don't use WebAddon:
    // quit $system.License.CSPUsers() 
    //
    // This will work even if  we use WebAddon:
    set csp = ""
    try {
        set cn = $NAMESPACE
        znspace "%SYS"
        set db = ##class(SYS.Stats.Dashboard).Sample()
        set csp = db.CSPSessions
        znspace cn
    } catch e {
        set csp = "0"
    }
    quit csp
}
/// Get license user's power
Method getKeyLicenseUnits() As %Integer
{
    quit $system.License.KeyLicenseUnits()
}
/// Get license expiration date in human-readable format
Method getKeyExpirationDate() As %String
{
    quit $zdate($system.License.KeyExpirationDate(),3)
}
/// Get performance metrics (gloref, rourines etc.)
Method getPerformance(param As %String) As %Integer
{
    set cn = $NAMESPACE
    znspace "%SYS"
    set m = ##class(SYS.Monitor.SystemSensors).%New()
    do m.GetSensors()
    znspace cn
    quit $listbuild(m.SensorReading("GlobalRefsPerMin"), 
                    m.SensorReading("RoutineCommandsPerMin"), 
                    m.SensorReading("RoutineLoadsPerMin"))
}
}

Make sure that the GetSample() method really fetches the necessary data for us:

TEST>set metrics = ##class(monitoring.snmp.Metrics).%New()

TEST>write metrics.GetSample()
1
TEST>zwrite metrics 
metrics=3@monitoring.snmp.Metrics  ; <OREF>
+----------------- general information ---------------
|      oref value: 3
|      class name: monitoring.snmp.Metrics
| reference count: 2
+----------------- attribute values ------------------
|      ExecutedSpeed = 2653596
|        GloRefSpeed = 35863
|  KeyExpirationDate = "2021-10-30"
|    KeyLicenseUnits = 5
|   RoutineLoadSpeed = 38
|           Sessions = 5
+-----------------------------------------------------

2. Let’s register and activate the new class in IRIS using ^%SYSMONMGR

Open the terminal and switch to the TEST namespace:

# iris session iris -U TEST
TEST>do ^%SYSMONMGR
1. Select item 5, Manage Application Monitor.

2. Select item 2, Manage Monitor Classes.

3. Select item 3, Register Monitor System Classes.
Exporting to XML started on 10/16/2020 10:05:39
Exporting class: Monitor.Sample
Export finished successfully.

Load started on 10/16/2020 10:05:39
Loading file /usr/irissys/mgr/Temp/VonCEUzQ8gWgfQ.stream as xml
Imported class: Monitor.Sample, using worker jobs
Compiling class Monitor.Sample
Compiling table Monitor.Sample
Compiling routine Monitor.Sample.1
Load finished successfully.

4. Select item 1, Activate/Deactivate Monitor Class
Class??
Num MetricsClassName Activated
1) %Monitor.System.AuditCount N
…
15) monitoring.snmp.Metrics N
Class? 15 monitoring.snmp.Metrics
Activate class? Yes => Yes

5. Select item 7, Exit

6. Select item 6, Exit

7. Select item 1, Start/Stop System Monitor

8. Select item 2, Stop System Monitor
Stopping System Monitor… System Monitor not running!

9. Select item 1, Start System Monitor
Starting System Monitor… System Monitor started

10. Select item 3, Exit

11. Select item 4, View System Monitor State
Component                     State
System Monitor                     OK
%SYS.Monitor.AppMonSensor          OK

12. Select item 7, Exit

3. Create a user MIB

A user MIB is created with the help of MonitorTools.SNMP class methods. For this example, let’s use a fake PEN (Private Enterprise Number), 99990, but the PEN will have to be registered with IANA afterwards. You can view registered numbers here. For example, InterSystems’ PEN is 16563.

16563
InterSystems
Robert Davis
rdavis&intersystems.com

We will use the MonitorTools.SNMP class and its CreateMIB() method to create a MIB file. This method takes 10 arguments:

Argument name and type	Description	Value
AppName As %String	application name	Value of the APPLICATION parameter of the metrics.snmp.Metrics class
Namespace As %String	our namespace	TEST
EntID As %Integer	company PEN	99990 (fiction)
AppID As %Integer	application OID inside the company	42
Company As %String	company name (capital letters)	fiction
Prefix As %String	prefix of all SNMP objects we create	fiction
CompanyShort As %String	short company prefix (capital letters)	fict
MIBname As %String	name of the MIB file	ISC-TEST
Contact As %String	contact information (address, in particular)	Let’s leave the default value: Earth, Russia, Somewhere in the forests, Subject: ISC-TEST.mib
List As %Boolean	equivalent to verbose. Show task progress for the MIB file	1

And here comes the creation of the MIB file:

%SYS>do ##class(MonitorTools.SNMP).CreateMIB("Monitoring","TEST",99990,42,"fiction","fict","fiction","ISC-TEST",,1)
Create SNMP structure for Application - Monitoring
     Group - Metrics
          ExecutedSpeed = Integer
          GloRefSpeed = Integer
          KeyExpirationDate = String
          KeyLicenseUnits = Integer
          RoutineLoadSpeed = Integer
          Sessions = Integer

Create MIB file for Monitoring
     Generate table Metrics
          Add object ExecutedSpeed, Type = Integer
          Add object GloRefSpeed, Type = Integer
          Add object KeyExpirationDate, Type = String
          Add object KeyLicenseUnits, Type = Integer
          Add object RoutineLoadSpeed, Type = Integer
          Add object Sessions, Type = Integer
MIB done.

There is new MIB ISC-TEST.mib in the <Install_dir>/mgr/TEST folder now.

4. Start the monitoring service with the connected IRIS subagent

Let’s open the System Administration -> Security -> Services -> %Service_Monitor (click) -> Service Enabled (check)

We also specify that we want to start the SNMP subagent when IRIS is started:

In Linux, we use the net-snmp package for SNMP monitoring. So we install it, configure it to be used with subagents and specify port 705 as the default one for the master agent to talk with subagents.
A small article about the snmpd.conf configuration file that complements the manual can be found on cyberciti. Here is your final set of settings:

# apt-get update
# apt-get install -y snmpd snmp
# grep '^[^#]' /etc/snmp/snmpd.conf
master agentx
agentXSocket TCP:localhost:705
com2sec local localhost public
group MyRWGroup v1 local
group MyRWGroup v2c local
group MyRWGroup usm local
view all included .1 80
view system included .iso.org.dod
access MyROGroup "" any noauth exact all none none
access MyRWGroup "" any noauth exact all all none
syslocation server (edit /etc/snmp/snmpd.conf)
syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
dontLogTCPWrappersConnects yes

Let’s restart the snmpd and snmptrapd daemons in Linux. After that, we start the SNMP service to activate the SNMP IRIS subagent:

%SYS>do start^SNMP

%SYS>; Check SNMP subagent status

%SYS>zwrite ^SYS("MONITOR")
^SYS("MONITOR","SNMP")="RUN"
^SYS("MONITOR","SNMP","NAMESPACE")="%SYS"
^SYS("MONITOR","SNMP","PID")=2035
^SYS("MONITOR","SNMP","PORT")=705
^SYS("MONITOR","SNMP","STARTUP")="SNMP agent started on port 705, timeout=20, winflag=0, Debug=0"
^SYS("MONITOR","SNMP","STATE")="Terminated - 10/16/2020 10:40:31.7147AM"
^SYS("MONITOR","SNMP","WINSTART")=0

5. Check that only our own, newly-created user OID’s are available

This can be done using snmpwalk — we’ll display the OID showing the number of CSP sessions:

# snmpwalk -On -v 2c -c public localhost 1.3.6.1.4.1.99990
.1.3.6.1.4.1.99990.42.1.1.1.1.4.73.82.73.83 = INTEGER: 1928761
.1.3.6.1.4.1.99990.42.1.1.1.2.4.73.82.73.83 = INTEGER: 226351
.1.3.6.1.4.1.99990.42.1.1.1.3.4.73.82.73.83 = STRING: "2021-10-30"
.1.3.6.1.4.1.99990.42.1.1.1.4.4.73.82.73.83 = INTEGER: 5
.1.3.6.1.4.1.99990.42.1.1.1.5.4.73.82.73.83 = INTEGER: 306
.1.3.6.1.4.1.99990.42.1.1.1.6.4.73.82.73.83 = INTEGER: 2

# If you get such result
# .1.3.6.1.4.1.99990 = No Such Object available on this agent at this OID
# try to restart SNMP subagent in IRIS in this way:
# do stop^SNMP
# do start^SNMP

The ISC-TEST.mib file contains the sequence of our OID’s:

FictMetricsR ::=
       SEQUENCE {
        fictExecutedSpeed   Integer32,
        fictGloRefSpeed   Integer32,
        fictKeyExpirationDate   DisplayString,
        fictKeyLicenseUnits   Integer32,
        fictRoutineLoadSpeed   Integer32,
        fictSessions   Integer32
       }

Accordingly, the number of sessions, for example, is the last OID 1.3.6.1.4.1.99990.42.1.1.1.6. You can compare it with the number of sessions shown on the SMP dashboard:

6. Let’s add our OID’s to an external monitoring system

Let’s use Zabbix. Zabbix documentation can be found here. A detailed Linux installation and configuration guide for Zabbix is available here. Zabbix was selected as a system that not only allows you to draw charts, but also monitor Plain Text (in our case, license expiry date and license units). After adding our 6 metrics to our local host items (type: SNMPv2 agent) and creating 4 graphs and 2 PlainText parameters (as screen elements), we should see the following picture:

Above is the information about license expiry and the number of available license slots. Graphs speak for themselves.

7. Let’s add the launch of the system monitor to the startup list of our TEST namespace

There is a pretty good document about user routines executed when IRIS starts and stops. They are called %ZSTART and %ZSTOP, accordingly.

What we are interested in is that the system monitor (^%SYSMONMGR) starts in the TEST namespace during the system start. By default, this monitor only starts on the %SYS namespace. Therefore, we will only look at the ^%ZSTART program. The source is in %ZSTART.mac (create and save it to the %SYS namespace).

ROUTINE %ZSTART

%ZSTART; User startup routine.
SYSTEM;
    ; IRIS starting
    do $zu(9,"","Starting System Monitor in TEST namespace by ^%ZSTART...Begin")
    znspace "TEST"
    set sc = ##class(%SYS.Monitor).Start()
    do $system.OBJ.DisplayError(sc)
    if (sc = 1) {
    do $zutil(9,"","Starting System Monitor in TEST namespace by ^%ZSTART...OK")
    } else {
    do $zutil(9,"","Starting System Monitor in TEST namespace by ^%ZSTART...ERROR")
    }
    ; Starting SNMP
    znspace "%SYS"
    do start^SNMP
    quit
LOGIN;
    ; a user logs into IRIS (user account or telnet)
    quit
JOB;
    ; JOB'd process begins
    quit
CALLIN;
    ; a process enters via CALLIN interface
    quit

Another way to do the same is using ^%SYSMONMGR:

%SYS>do ^%SYSMONMGR

1. Select item 3, Configure System Monitor Classes.

2. Select item 2, Configure Startup Namespaces.

3. Select item 2, Add Namespace.
Namespace? TEST

4. Select item 1, List Start Namespaces.
Option? 1
     TEST

5. Select item 4, Exit.

6. Select item 3, Exit.

7. Select item 8, Exit.

Let’s now restart IRIS (if possible) to make sure that SNMP stats continue to be collected after a restart.

This is it. Perhaps, some will question my choice of monitored parameters or code, but the task was to show the mere possibility of implementing such monitoring in principle. You can add extra parameters or refactor your code later.
Also, it's worth to make IRIS settings persistent in Docker using Installer, but it's not in a scope of this article.

Blog