Chapter 6. Operating the noitd daemon

Once noitd is configured and running, there are several methods of interacting with the service. The operator can interactively manage the instance via the integrated command line console (accessible via telnet on the port of your choice, as configured in noitd.conf via listener elements; by default port 32322) and programmatic access is available over the noitd wire protocol on port 43191.

The wire protocol is a binary protocol that is leverage for inter-system communication. While not entirely compliant with the HTTP specification, the protocol will give well formed HTTP responses to sessions that commence with the terms: "DELETE" "GET " "HEAD" "POST" "PUT ".

6.1. noitd Interactive Console

6.1.1. Interactive Console Command Line Operation

This section serves to document the basic methods of operation available through the interactive console's command line interface. It is currently in rough draft status.

6.1.1.1. General Operational Principles

The command line interface to the system available through the interactive console operates in a fashion similar to Cisco's IOS™, in that there are various "levels" or "sets"/"sub-sets" of commands within the interface. Operations and operands may be tab-completed or abbreviated to their un-ambiguous form as needed or desired (analogous to "configure terminal" becoming "conf t" in common IOS usage). Command history and editing is available.

6.1.1.2. Accessing the Interactive Console

The interactive console is accessed by telneting to the configured access port on the host running noitd. By default this is port 32322. The applicable configuration parameter in the noitd.conf XML file is the port attribute of the "listener" element, e.g.:

      <listener address="*" port="32322">
        

Please see the configuration section of the manual for more information.

6.1.1.3. Basic Command List

An example session of interaction showing the use of help and a few operating modes is illustrative:

      noit# help
      = Topics =
      ==> 'module'

      = Commands =
      ==> 'apply'
      ==> 'configure'
      ==> 'exit'
      ==> 'no'
      ==> 'reload'
      ==> 'restart'
      ==> 'show'
      ==> 'shutdown'
      ==> 'watch'
      ==> 'write'
      noit# help module
      Help for 'module':
      = Loaders and Modules =
      lua
      ping_icmp
      smtp
      ssh2
      varnish
      selfcheck
      resmon
      http
      snmp
      noit# configure {tab pressed twice}
      apply     help      terminal
      noit# conf t
      noit(conf:/)# ls
      == Section Settings ==
      @lockfile: /var/run/noitd.lock
      == Subsections ==
      eventer
      logs
      modules
      listeners
      rest
      checks
      filtersets
      config_templates
      noit(conf:/)# help
      -> 'terminal'
      = Topics =
      ==> 'module'

      = Commands =
      ==> 'apply'
      ==> 'attribute'
      ==> 'cd'
      ==> 'check'
      ==> 'config'
      ==> 'exit'
      ==> 'filterset'
      ==> 'ls'
      ==> 'no'
      ==> 'section'
      ==> 'write'
      noit(conf:/)# exit
      noit# show
      ==> 'apply'
      ==> 'check'
      ==> 'checks'
      ==> 'eventer'
      ==> 'watches'
      incomplete command.

      

Command Descriptions

help

help provides brief help information appropriate to a given context.

show

show shows various aspects of the system. See the "Showing System Information" section below for further detail.

watch

watch is much like show except it re-displays the given information periodically. See the "Showing System Information" section below for further detail.

exit

exit leaves the current "level" of session (e.g. returns to regular interactive session if used within a configuration sub-"level" or exits the current interactive session entirely and closes the connection if used within the regular interaction "level").

configure

configure causes the session to enter configuration mode if used with the terminal argument or with the apply argument allows for multiple configurations to be specified at once in an automated fashion (see below for coverage of apply and the "Configuring the System" section for more information about run-time configuration in general).

write

write writes the current operating configuration to: the permanent configuration store (noitd.conf by default, in the location specified either at startup with -c or in the default compile-time system directory) with file argument (memory is an alias), or to the current interactive display with the terminal argument.

reload

reload causes noitd to reload configuration information from the permanent configuration store (noitd.conf by default, in the location specified either at startup with -c or in the default compile-time system directory).

restart

restart causes noitd to restart, re-reading all configuration from the permanent configuration store in the process. As part of restarting, the current interactive session is terminated.

shutdown

shutdown causes noitd to shut down entirely.

apply

apply {variable name} {range specifier} {command} works much like a for {name} in {list}; do {command, optionally using $name}; done loop in the bash shell, in that it iterates over the given range, setting the variable name to each value in turn and executing the given command with $variablename set and available at each execution. A simple example is given below of executing an ssh2 check three times using apply:

                noit# apply i 1..3 show check 002d58ff-20ff-4db0-9420-782fc1748dc4
                ==== 002d58ff-20ff-4db0-9420-782fc1748dc4 ====
                name: ssh2 [from module]
                module: ssh2
                target: 127.0.0.1
                period: 60000
                timeout: 4000
                oncheck: [undef]
                filterset: default [inherited from @filterset]
                disable: [undef]
                currently: idle
                last run: 31.104 seconds ago
                availability/state: available/good
                status: 59698f91c167393deb952564fe316dca
                metrics:
                duration[I] = 192
                fingerprint[s] = 59698f91c167393deb952564fe316dca
                ==== 002d58ff-20ff-4db0-9420-782fc1748dc4 ====
                name: ssh2 [from module]
                module: ssh2
                target: 127.0.0.1
                period: 60000
                timeout: 4000
                oncheck: [undef]
                filterset: default [inherited from @filterset]
                disable: [undef]
                currently: idle
                last run: 31.104 seconds ago
                availability/state: available/good
                status: 59698f91c167393deb952564fe316dca
                metrics:
                duration[I] = 192
                fingerprint[s] = 59698f91c167393deb952564fe316dca
                ==== 002d58ff-20ff-4db0-9420-782fc1748dc4 ====
                name: ssh2 [from module]
                module: ssh2
                target: 127.0.0.1
                period: 60000
                timeout: 4000
                oncheck: [undef]
                filterset: default [inherited from @filterset]
                disable: [undef]
                currently: idle
                last run: 31.104 seconds ago
                availability/state: available/good
                status: 59698f91c167393deb952564fe316dca
                metrics:
                duration[I] = 192
                fingerprint[s] = 59698f91c167393deb952564fe316dca
              
no

no works much like "no" in IOS™, negating or nullifying another command. e.g. no watch {check id} {interval} to disable a given watch (see below for further details).

6.1.1.4. Showing System Information

The two main commands for showing system information are show and watch. show is useful for one-off queries of information, and watch is useful for periodically redisplaying the same information (e.g. to see snapshots of activity while some other process is affecting the monitored system/information).

6.1.1.4.1. show

show show accepts a number of arguments:

watches

show watches displays a list of all active watches currently being serviced by noitd.

checks

show checks displays a list of checks that noitd is currently servicing. Example output is illuminative:

                noit# show checks
                1b4e28ba-2fa1-11d2-883f-b9b761bde3fb 127.0.0.1`ping_icmp
                cnt=5,avail=100,min=0.0000,max=0.0000,avg=0.0000
                f7cea020-f19d-11dd-85a6-cb6d3a2207dc 127.0.0.1`selfcheck
                ok
                002d58ff-20ff-4db0-9420-782fc1748dc4 127.0.0.1`ssh2
                59698f91c167393deb952564fe316dca
              

Each check's primary identifier is a hexadecimal GUID. This may be tab-completed in common use from the longest unique string, much like commands. In the example above, for instance, "002" would be sufficient to tab complete a reference to the check 002d58ff-20ff-4db0-9420-782fc1748dc4. Each check is further identified as accessing a given IP/host and contains some description of what the check is doing as configured when the check was configured (see the "Configuring the System" section below, and/or noitd.conf). The second line for each check provides check-specific further information or status.

check

show check {check id}. See discussion above for referencing a particular check by id. This provides more in-depth information about a specific check. For example, using the check list mentioned above:

                noit# show check 1b4e28ba-2fa1-11d2-883f-b9b761bde3fb
                ==== 1b4e28ba-2fa1-11d2-883f-b9b761bde3fb ====
                name: ping_icmp [from module]
                module: ping_icmp
                target: 127.0.0.1
                period: 15000
                timeout: 14000
                oncheck: [undef]
                filterset: default [inherited from @filterset]
                disable: [undef]
                currently: running
                last run: 9.377 seconds ago
                availability/state: available/good
                status: cnt=5,avail=100,min=0.0000,max=0.0001,avg=0.0000
                metrics:
                available[n] = 1.000000000000e+02
                *count[i] = 5
                average[n] = 4.359999984445e-05
                *maximum[n] = 5.400000009104e-05
                *minimum[n] = 2.800000038405e-05
              
6.1.1.4.2. watch

watch {check id} {interval} redisplays information for a given check at a configurable interval. Interval is given in units of milliseconds. If interval is omitted, presently, there is a bug where the check begins to output at its configured interval directly to the cli, but does not appear as a watch in show watches and no watch {id} has no effect.

The currently active watches may be listed with show watches, and an active watch may be terminated with by issuing the command no watch {check id} {interval} where check id and interval are the ones used for the initial creation of the watch.

6.1.1.5. Configuring the System

The focus of this section is configuring noitd's behavior at run-time via the interactive console. For documentation of noitd.conf itself, see the main chapter on system configuration. As a general note, to store run-time configuration to the permanent configuration store, see the write file command above.

6.1.1.5.1. Configure Terminal ("conf t") Mode

To enter the interactive configuration mode, first telnet to noitd's telnet port and issue the "configure terminal" command, which may be abbreviated as "conf t" if desired. (Please see above information about default port, port configuration location, etc.)

Configure terminal mode presents a different array of options than regular mode, which are summarized below. (Some common commands have been ommitted as their function is identical within conf-t mode as within normal interactive mode, e.g. exit.)

cd

cd {location, e.g. ".." or "checks"} changes the point where the interactive configuration session is within the tree-like organizational structure of the configuration space in a fashion similar to cd within a filesystem directory tree. (See note below.)

section

section {location} operates similarly to cd, except that it will create the new location if it does not exist.

ls

ls lists the configured entities at a given point within the tree-like organizational structure of the configuration space in a fashion similar to ls within a filesystem directory tree. (See note below.)

status

status displays the fine-grained details about a particular leaf-node/entity within the configuration space.

check

check {check id} enters the configuration space for a particular leaf-node of the general configuration tree, i.e. the exact configuration for a particular check within the system. check new {params} may be used to set up a new check within the system, where params are dependent on what is required for a check using whichever module. For example, check new target 127.0.0.1 module http period 60000 timeout 10000 sets up a new http check on localhost running every minute and timing out after ten seconds. By default the new check will be named "target`module", and after entry of the above command you will be placed into the configuration space for the new check as though you had run check {the new id} upon it. Of note is that the new check is disabled by default (see status output), you must run no attribute disable to enable it.

attribute

attribute enables you to alter the fine-grained details about a particular leaf-node/entity within the configuration space; e.g. "no attr disable" to enable a check, or "attr port 2222" on an ssh2 check to have it check port 2222 instead of the default 22.

filterset

filterset is applicable only in the /filterset "directory" and has to do with the maintenance of filters, which are (briefly) exclusion patterns that filter out collected metrics from being reported upstream to stratcon (the noit daemon will still collect all of the data, but only the un-filtered metrics will propagate).

Configure terminal mode has a general tree-like organizational structure mimicing the xml in noitd.conf, hence commands like cd and ls. Within some leaf nodes of the tree (for instance, check entities' representations) status should be substituted for ls as the latter operates on entities rather than attributes of entities (which would be displayed by status).

An example configuration session is shown below to illustrate some of the above:

        noit# conf t
        noit(conf:/)# cd checks
        noit(conf:/checks)# ls
        == Section Settings ==
        @filterset: default
        @transient_min_period: 1000
        @transient_period_granularity: 500
        == Subsections ==
        foo
        == Checks ==
        check[@uuid="f7cea020-f19d-11dd-85a6-cb6d3a2207dc"] 127.0.0.1`selfcheck`selfcheck
        check[@uuid="1b4e28ba-2fa1-11d2-883f-b9b761bde3fb"] 127.0.0.1`ping_icmp`ping_icmp
        check[@uuid="002d58ff-20ff-4db0-9420-782fc1748dc4"] 127.0.0.1`ssh2`ssh2
        check[@uuid="70a3633f-1629-4417-a7db-d48b9558c48e"] 127.0.0.1`http`http
        noit(conf:/checks)# check 70a3633f-1629-4417-a7db-d48b9558c48e 
        noit(conf:127.0.0.1`http)# status
        ==== 70a3633f-1629-4417-a7db-d48b9558c48e ====
        name: http [from module]
        module: http
        target: 127.0.0.1
        period: 60000
        timeout: 10000
        oncheck: [undef]
        filterset: default [inherited from @filterset]
        disable: [undef]
        currently: idle
        last run: 46.432 seconds ago
        availability/state: available/good
        status: code=200,rt=0.001s,bytes=39
        metrics:
        tt_connect[I] = 0
        bytes[i] = 39
        code[s] = 200
        tt_firstbyte[I] = 1
        duration[I] = 1
        noit(conf:127.0.0.1`http)# exit
        noit(conf:/checks)# exit
        noit#