Skip to content

EDR#

TLDR: See recipes at the bottom for ways to use EDR.

Description#

This app is responsible for getting EDR events (see kazoo_edr) out of Kazoo. It must be started per kazoo node you wish to receive events from.

Backends#

The EDR app is essentially a collection of backends. Each backend will receive EDR events matching a binding, and send them somewhere else (usually out of Kazoo).

Backends are registered in system_config/edr.

General registered backend configuration options#

Schema for edr system_config

Key Description Type Default Required
backends.[].bindings.[].account_id The account id to receive events from. If not specified or * then events for all accounts will be received string() * false
backends.[].bindings.[].app_name Which apps should we receive events from? If not specified or * then events for all apps will be received string() * false
backends.[].bindings.[].exact_severity If true, then only events with severity exactly matching the severity field will be received. If false then more severe events will also be received boolean() false false
backends.[].bindings.[].exact_verbosity If true, then only events with verbosity exactly matching the verbosity field will be received. If false then less verbose events will also be received boolean() false false
backends.[].bindings.[].include_descendants Should we receive events relating to descendant accounts if an account is specified? boolean() false false
backends.[].bindings.[].severity What severity are we interested in events for? string('ok' | 'warning' | 'critical') ok false
backends.[].bindings.[].verbosity What verbosity are we interested in events for? string('trace' | 'debug' | 'info' | 'warn' | 'error' | 'fatal') info false
backends.[].bindings The types of events the backend should be notified about array(object()) true
backends.[].enabled Should the backend be started on edr app start? boolean() true true
backends.[].name A unique identifier for the backend string(1..) true
backends.[].options.formatter.options Options passed on to the formatter module. These vary by module object() false
backends.[].options.formatter.type The formatter module to be used (e.g. json refers to edr_fmt_json.erl) string('json') false
backends.[].options.formatter Output format configuration which is used by some (but not all) backends object() false
backends.[].options Options passed on to the backend module. These vary by module object() false
backends.[].type The backend module to be used (e.g. amqp refers to edr_be_amqp.erl) string('amqp' | 'file' | 'http') true
backends Registered edr backends array(object()) [] false

Backend types#

AMQP (edr_be_amqp)#

This module forwards each event to RabbitMQ for use across nodes. kapi_edr_amqp can be used to bind to these events. The routing key used is the same as that which is used by edr_bindings for consistency edr.{SEVERITY}.{VERBOSITY}.{ACCOUNT_ID}.{APP_NAME}.

There are configuration options yet, however a configurable routing key would be useful in future.

file (edr_be_file)#

This module outputs each received event to a line in a log file. Each line is formatted by the specified formatter. The configuration options are as follows: - path (required): The path to the file. Ensure file (or directory, if the file does not exist) has the correct permissions - formatter: Configuration as to how to format each event. This defaults to JSON with the default options.

HTTP (edr_be_http)#

This module sends out a HTTP request for each EDR event it receives. The configuration options are as follows: - url (required): The URL the request will be sent to - method: The HTTP method the request should use. Valid options are put, post, and patch. This defaults to post. - headers: A JSON object containing headers and their values - async: Should the request be asynchronous? Currently only false is implemented. - connect_timeout: Connection timeout in milliseconds - timeout: Request timeout in milliseconds - formatter: Configuration as to how to format each event. Defaults to json with the default options.

Formatters#

A formatter is responsible for taking an event and formatting it for a backend. This is not configurable for every backend.

json (edr_fmt_json)#

The configuration options are as follows: - include_metadata: If set to true the output will be the whole EDR event. If set to false the output will just be the body. Defaults to true. - normalize: Should the metadata fields be normalized to snake case? Defaults to true - pretty: Should the output be nicely formatted? Defaults to false

Sup commands#

  • sup edr_maintenance register_backend {NAME} [{TYPE}] [{JSON_OPTS}] [{JSON_BINDINGS}] [{ENABLED}]: Register a backend in the system_config, and start it if it is enabled. It is probably easier to use the system_config API and run sup edr_maintenance enable_backend {NAME} if you wish to specify options or bindings.

  • sup edr_maintenance delete_backend {NAME}: The opposite of register_backend. Stop the backend if it is running, and remove it from the the system_config.

  • sup edr_maintenance enable_backend {NAME} Start the backend, and enable it if it isn't already enabled.

  • sup edr_maintenance disable_backend {NAME} Stop the backend, and disable it if it isn't already disabled.

  • sup edr_maintenance registered_backends Show the configured backends.

Recipes#

Practical uses of EDR. It would be great if people could submit PRs with their configurations for other people to use.

Multiple logs with different verbosities#

system_config/edr:#

{
   "backends": [
       {
           "name": "debug_log",
           "type": "file",
           "options": {
               "path": "/var/log/kazoo/edr_debug.log"
           },
           "bindings": [
               {
                   "account_id": "*",
                   "include_descendants": false,
                   "app_name": "*",
                   "severity": "ok",
                   "exact_severity": false,
                   "verbosity": "debug",
                   "exact_verbosity": false
               }
           ],
           "enabled": true
       },
       {
           "name": "info_log",
           "type": "file",
           "options": {
               "path": "/var/log/kazoo/edr_info.log"
           },
           "bindings": [
               {
                   "account_id": "*",
                   "include_descendants": false,
                   "app_name": "*",
                   "severity": "ok",
                   "exact_severity": false,
                   "verbosity": "info",
                   "exact_verbosity": false
               }
           ],
           "enabled": true
       }
   ]
}

EDR over Blackhole#

This involves forwarding EDR events specific to an account over AMQP, which are then pushed over websockets with Blackhole. The binding key is edr.{SEVERITY}.{VERBOSITY}.{APP_NAME}. Only events where the account_id matches that associated with the auth token will be received.

Note: You can request events with a verbosity or severity greater than what edr_be_amqp is bound to, however you won't receive any of them.

system_config/edr:#

{
   "backends": [
       {
           "name": "amqp",
           "type": "amqp",
           "options": {
           },
           "bindings": [
               {
                   "account_id": "*",
                   "include_descendants": false,
                   "app_name": "*",
                   "severity": "ok",
                   "exact_severity": false,
                   "verbosity": "info",
                   "exact_verbosity": false
               }
           ],
           "enabled": true
       }
    ]
}

Start the Blackhole module#

sup blackhole_maintenance start_module bh_edr

EDR events to HTTPBin#

An example of sending EDR events over HTTP.

{
   "backends": [
       {
           "name": "mock_bin",
           "type": "http",
           "options": {
               "url": "http://mockbin.org/bin/021cb9cb-b688-4f1e-9894-c80b4a19e654",
               "headers": {
                   "Content-Type": "application/json"
               }
           },
           "bindings": [
               {
                   "account_id": "*",
                   "include_descendants": false,
                   "app_name": "*",
                   "severity": "ok",
                   "exact_severity": false,
                   "verbosity": "info",
                   "exact_verbosity": false
               }
           ],
           "enabled": true
       }
    ]
}