Sensu

Sensu is an open source computer monitoring framework. It is useful for monitoring servers, services, application health, and business KPIs. Get notified about failures before your users do. Collect and analyze custom metrics.

Sensu API examples

Note: These examples use jq for simple JSON parsing from the CLI.

• Setup environment variables for the various Sensu API endpoints:

$ USER=<redacted>
$ PASSWORD=<redacted>
$ PORT=4567
$ SENSU_HOST=localhost
$ ENDPOINT=${USER}:${PASSWORD}@${SENSU_HOST}:${PORT}

• Get a list of clients (i.e., hostnames):

$ curl -sH "Content-type:application/json" "${ENDPOINT}/clients" |\
       jq '[.[] | .name]'

• Get a list of checks for all "controller" nodes:

$ curl -sH "Content-type:application/json" "${ENDPOINT}/checks" |\
       jq '[.[] | select(.subscribers[] == "controller") | .name]'

• Create a stash (i.e., silence alert):

$ CLIENT=lab-controller.openstack.lan
$ CHECK=check-nova-boot
$ MESSAGE="silencing during maintenance"
$ EXPIRE=300 # seconds
$ curl -XPOST -H"Content-type:application/json" "${ENDPOINT}/stashes" \
       -d "{\"path\":\"silence/${CLIENT}/${CHECK}\",\
            \"content\":{\"message\":\"${MESSAGE}\"},\"expire\":${EXPIRE}}"

• Delete the above stash:

$ curl -XDELETE -H"Content-type:application/json" "${ENDPOINT}/stashes" \
       -d "{\"path\":\"silence/${CLIENT}/${CHECK}\"}"

• Create stashes for all checks on a given client (where the client is a "controller" subscriber):

$ for check in $(curl -sH"Content-type:application/json" \
      "${ENDPOINT}/checks" |\
      jq -r '[.[] | select(.subscribers[] == "controller") | .name] | .[]'); do
      curl -XPOST -H"Content-type:application/json" "${ENDPOINT}/stashes" \
           -d "{\"path\":\"silence/${CLIENT}/${check}\",\
                \"content\":{\"message\":\"${MESSAGE}\"},\"expire\":${EXPIRE}}"
  done

• Or, to completely silence a given client:

$ CLIENT=lab-controller.openstack.lan
$ curl -XPOST -H"Content-type:application/json" "${ENDPOINT}/stashes" \
       -d "{\"path\":\"silence/${CLIENT}\",\
         \"content\":{\"message\":\"${MESSAGE}\"},\"expire\":${EXPIRE}}"

• Get a list of the actual commands each check is running on a given client (where the client is a "controller" node):

$ for check in $(curl -sH"Content-type:application/json" "${ENDPOINT}/checks" |\
      jq -r '[.[] | select(.subscribers[] == "controller") | .name] | .[]'); do
      curl -sH"Content-type:application/json" \
      "${ENDPOINT}/checks/${check}" | jq -r '.command';
  done

Run Sensu checks as sudo

In order for a sensu check to be run as sudo, perform the following:

# Make sure to:
$ usermod -aG wheel sensu
# Make sure the following line exists in `visudo`:
# %wheel  ALL=(ALL)       NOPASSWD: ALL
$ cat << EOF >/etc/sudoers.d/sensu
Defaults:sensu !requiretty
Defaults:sensu secure_path = /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

sensu ALL = NOPASSWD: /etc/sensu/plugins/*.sh
EOF

External links

• Official website