Skip to content

KAZOO Support Channels

This documentation is curated by 2600Hz as part of the KAZOO open source project. Join our community forums here for peer support. Only features in the docs.2600hz.com/supported space are included as part of our 2600Hz Support Services plan.

Flame Graphs

Flame graphs#

It can be nice to visualize where time is spent when executing code.

We have added Scott Fritchie's fork of eflame to help trace and generate flame graphs.

Important: as far as I can tell, you can only have one trace going on at any given time. Be aware!

kz_tracers#

The kz_tracers module (in core/whistle_apps) starts the trace. Currently it just traces the supplied pid() with an optional time to trace (default 100s but this is way too long for most requests).

Tracing Crossbar requests#

You can enable a trace for a crossbar request by adding a header x-trace-request: true.

You will also need to enable tracing as a system config: sup kapps_config set_default crossbar allow_tracing true This has performance implications so I wouldn't leave this enabled on a production environment.

Now try your cURL request:

curl -v -X PUT \
    -H "content-type:application/json" \
    -H "x-trace-request:true" \
    -d '{"data":{"credentials":"{HASH}","account_name":"master"}}' \
    http://localhost:8000/v2/user_auth

In your Erlang VM (if you're running in interactive mode), you should see some logs:

started trace for <0.10699.2> in <0.10700.2>
Tracer <0.10667.2>
self() <0.10700.2>
Starting tracer results: {v_global_and_local_calls,
                             {ok,[{matched,
                                      'whistle_apps@localhost.localdomain',
                                      24456},
                                  {saved,1}]}}
trace for <0.10699.2> done
trace formatted to /tmp/eflame.trace.out
<0.10699.2> Hello, world, I'm <0.10713.2> and I'm running....
<0.10706.2>

Writing to /tmp/eflame.trace.out for 3 processes... finished!

Once its finished, you need to run a script to create the svg file:

kazoo/deps/eflame$ cat /tmp/eflame.trace.out | ./flamegraph.riak-color.pl > /tmp/eflame.svg

Now load that into your browser and click around!

You can also remove the SLEEP time:

kazoo/deps/eflame$ cat /tmp/eflame.trace.out | grep -v 'SLEEP ' | ./flamegraph.riak-color.pl > /tmp/eflame.svg

Issues#

Sometimes nothing gets traced and no svg can be generated. You might see this in your VM:

started trace for <0.10645.2> in <0.10646.2>
Tracer <0.4131.0>
self() <0.10646.2>
Starting tracer results: {v_global_and_local_calls,
                             {ok,[{matched,
                                      'whistle_apps@localhost.localdomain',
                                      24456},
                                  {saved,1}]}}
trace for <0.10645.2> done
trace formatted to /tmp/eflame.trace.out


Writing to /tmp/eflame.trace.out for 0 processes... finished!

I haven't bothered looking into this yet, as rerunning the request to do another trace usually gets some output. I think it depends on how fast the request is processed?