Skip to content

KAZOO Support Channels

This documentation is curated by 2600Hz as part of the KAZOO open source project. Join our community forums here for peer support. Only features in the space are included as part of our 2600Hz Support Services plan.

Kazoo System Tracing#

It is possible to create Erlang traces and write them to file(s) so you can get an idea about what is happening in the VM. Lager provides this tracing functionality and kz_data_tracing exposes the wrapper.

Start a trace#

Creating a system trace needs to be surgical; too many events could overwhelm the VM and impact performance. Especially if tracing a production system, use filters to limit the scope of events traced.

Starting a trace is straight-forward:

1> {'ok', Trace1} = kz_data_tracing:trace_file().
{'ok', Ref1}
2> {'ok', Trace2} = kz_data_tracing:trace_file(Filters).
{'ok', Ref2}
3> {'ok', Trace3} = kz_data_tracing:trace_file(Filters, Filename).
{'ok', Ref3}
4> {'ok', Trace4} = kz_data_tracing:trace_file(Filters, Filename, Format).
{'ok', Ref4}

SUP-provided examples:

sup kazoo_data_maintenance trace_module Module
sup kazoo_data_maintenance trace_function Function
sup kazoo_data_maintenance trace_function Module Function
sup kazoo_data_maintenance trace_pid 'X.Y.Z'


Filters are 2-tuples that contain the patterns used on events to see if they meet the necessary criteria.

  • `{'module', Module}`: Filter for calls to `Module`
  • `{'function', Function}`: Filter for calls to `Function` When combined with `{'module', Module}`, just trace that Module:Function combo
  • `{'pid', Pid}`: Filter for calls by `Pid`
  • Custom metadata
    • `lager:warning([{request, RequestId}], "the passwords didn't match for ~s", [UserId])` would allow a filter of `{'request', RequestId}`
    • You can also associate metadata with a `pid()`: `lager:md([{'zone', 'west'}])`

You can also use `'*'` in the second element to match all of that class of filter.

For example, `[{'module', 'kz_json'}, {'function', '*'}]` would filter all calls to the `kz_json` module.


Specify the filename for where to write the trace results.


You can also supply a custom formatter to be used when writing the log line (uses the Kazoo standard format otherwise).


  • Filters: `[{'function', '*'}]`
  • Filename: `/tmp/{32_char_hex}.log`
  • Format: -define(DEFAULT_TRACE_OUTPUT_FORMAT, ['time'," [",'severity',"] |", 'from_app', "|", {'callid', }, "|", 'mod', ":" , 'func', ":", 'line', " (",'pid', ") ", 'message', "\n"]).

Stopping a trace#

Assuming you bound the second element in the resulting tuple:

1> {'ok', Ref} = kz_data_tracing:trace_file().
{'ok', Ref}
2> ...Do some things...or not
3> kz_data_tracing:stop_trace(Ref).