Skip to content

KAZOO Support Channels

This documentation is curated by 2600Hz as part of the KAZOO open source project. Join our community forums here for peer support. Only features in the docs.2600hz.com/supported space are included as part of our 2600Hz Support Services plan.

Property-based testing with KAZOO#

Introduction#

Me#

KAZOO#

  • https://github.com/2600hz/kazoo
  • Started in 2010
  • Telecom platform
    • Clustering layer over FreeSWITCH / Kamailio
  • API-driven
  • Scales from Hobbyists to Enterprise
  • Built on:
    • RabbitMQ
    • CouchDB
  • 275K lines of Erlang in the core project in 1315 modules
  • 106 contributors (~25 that are 2600Hz)

Stateless testing - JSON#

JSON#

  • kz_json.erl
    • Provides lists-esque functionality - maps/folds/filters/etc
    • getters/setters, merging, diffing, and more
    • Hides data structure used - enforced throughout the code
  • kz_json_tests.erl and kz_json_generators.erl
    • Need to generate "deep" objects but not too deep
    • Naive approach: test_object()
      • Generate list of Key/Value pairs
    • Better approach: deep_object()
      • Symbolic calls to build up the object

test_object() generated#

    1> proper_gen:pick(kz_json_generators:test_object()).
    {ok,{[{<<14,141,161>>,-19},
          {<<37,53,158>>,<<>>},
          {<<"&">>,<<>>},
          {<<81,197,72,47,41,80,75,41,19>>,<<>>},
          {<<155,65,38,243,136,74,115>>,<<>>},
          {<<176,5,171,200>>,<<>>},
          {<<"ãO">>,<<>>}]}}

test_object() spread#

    2> proper:quickcheck(kz_json_tests:prop_test_object_gen(), 1000).
    58% {0,2}
    38% {2,4}
    2% {4,6}
    true

deep_object() generated#

    1> {ok, Calls} = proper_gen:pick(kz_json_generators:deep_object()).
    {ok, {'$call',kz_json,set_value,
     [<<"ðg">>,
      [-4,<<>>,<<>>,<<>>,
       [<<>>,<<>>,{[{<<19,184,115,217,157,45,202,135,28>>,<<>>}]}],
       <<>>,<<>>,<<>>],
      {'$call',kz_json,set_value,
       [<<"~áä±õOv·@">>,[],
        {'$call',kz_json,set_value,
         [<<133,141>>,
          [],
          {'$call',kz_json,set_value,
    ...

deep_object() eval'd#

    2> proper_symb:eval([], Calls).
    {[{<<4,133,215,252,0>>,[]},
      {<<"lËÉx&'">>,[]},
      {<<182,179,144,154>>,[]},
      {<<132,250,21,171,119,26,197,90,175,188>>,
       [{[{<<98,36,70,211,105,95,174,109,130>>,<<>>},
          {<<4,35,100,156,67,58,75,203,168,89,107>>,<<>>},
          {<<190,144,146,45,53,85,153,231,166,84,233>>,<<>>},
          {<<35,223,73,21,92,176,167,254,8>>,<<>>},
          {<<214,210,66,21,57,117>>,<<>>},
          {<<226,152,179,17>>,<<>>},
          {<<107,119,61,244,188,157,110,28>>,
           {[{<<133,246,158,227,95,35,251,39,...>>,<<>>},
           ...

deep_object() spread#

    3> proper:quickcheck(kz_json_tests:prop_deep_object_gen(), 1000).
    56% {2,4}
    37% {0,2}
    5% {4,6}
    0% {6,8}
    true

JSON#

  • More control over terms generated
  • More control over depth
  • Create EUnit tests from failing PropEr tests
    • Shrinking is huge!
  • Actually had good EUnit test coverage prior

Stateless testing - Bindings server#

Bindings#

  • AMQP-style bindings
    • Routing key: "a.b.c"
    • Binding key: "*.b.#"
      • "*" - match one segment
      • "#" - match 0 or more segments
  • Credit to Fred Hebert @mononcqc

Bindings - Generator: expanded_path()#

Generates a binding key, a routing key, and whether there should be a match.

    1> proper_gen:pick(kazoo_bindings_tests:expanded_paths()).
    {ok,[{<<"*.1Oj.#.t863f4e3Xu">>,<<"1Oj.t863f4e3Xu">>,false}
        ,{<<"*.1Oj.#.t863f4e3Xu">>,<<"lLTW1.1Oj.t863f4e3Xu">>,true}
    ]}

Bindings - Binding Key#

  • Binding key: "*", "1Oj", "#", "t863f4e3Xu"
    1. Match any first segment
    2. Match "1Oj" as second segment
    3. Match 0 or more segments
    4. Match "t863f4e3Xu" as last segment

Bindings - Matching routing key#

  • Matching Routing key: "lLTW1" "1Oj" "t863f4e3Xu"
    1. "lLTW1" matches "*"
    2. "1Oj" matches "1Oj"
    3. 0 matches for "#"
    4. "t863f4e3Xu" matches "t863f4e3Xu"

Bindings - Failing routing key#

  • Failing Routing Key: "1Oj" "t863f4e3Xu"
    1. "1Oj" matches "*"
    2. "t863f4e3Xu" does not match "1Oj"

Bindings - Found bugs#

  • Had "reasonable" EUnit tests
  • 2011-06-12: Introduced PropEr testing, first 5 bugs found were generic
  • 2011-06-13: First "weird" match
    • {<<"#.6.*.1.4.*">>, <<"6.a.a.6.a.1.4.a">>}
  • 2016-12-19: Found two more failing
    • {<<"*.u.*.7.7.#">>,<<"i.u.e.7.7.7.a">>}
    • {<<"#.c.#.c.#">>, <<"c.c">>}
  • 2017-01-27: Found one more
    • {<<"#.Z.*.9.0.#">>,<<"1.Z.7.9.0.9.a.0">>}
  • 2017-02-27: Found one more
    • {<<"W0.*.m.m.#">>, <<"W0.m.m.m.5">>}

Stateful testing - LRU Cache#

Cache#

  • LRU cache in ETS
  • Maintains "origin pointers" - links to the datastore
    • AMQP events can evict cache entries
  • Maintains "monitors"
    • Wait for a key to be cached or timeout
  • Callbacks on events
    • 'timeout', 'expire', 'flush', 'erase', 'store'
  • Cache stampede mitigation
    • Provides a way to block calling processes while a key's value is being computed

Cache - API commands#

  • store
  • peek
  • fetch
  • erase
  • wait_for_key
  • mitigate_stampede
  • wait_for_stampede
  • timer:sleep/1

Cache - Model#

  • Track cache as a proplist
  • Track "time" as # of milliseconds
    • Increment "time" on each timer:sleep/1
    • Expire entries

Cache - Sample commands#

    1> proper_gen:pick(kz_cache_pqc:command({state, [], 0})).
    {ok,{call,kz_cache,store_local,
              [kz_cache_pqc,113,8,[{expires,1}]]}}
    2> proper_gen:pick(kz_cache_pqc:command({state, [], 0})).
    {ok,{call,kz_cache,erase_local,[kz_cache_pqc,99]}}

Cache - Running the tests#

    1> proper:quickcheck(kz_cache_pqc:correct()).
    ...
    13% {kz_cache,peek_local,2}
    13% {kz_cache,mitigate_stampede_local,3}
    13% {kz_cache,wait_for_stampede_local,3}
    13% {timer,sleep,1}
    12% {kz_cache,fetch_local,2}
    12% {kz_cache,erase_local,2}
    11% {kz_cache,store_local,4}
    10% {kz_cache,wait_for_key_local,3}
    true

Cache - Challenges with time#

  • Model-only pass is "accurate" on expiration
  • SUT pass is subject to the VM and timers may not fire at the precise timeout
  • Mostly works for LRU testing though - rare that the tests fail due to time

Stateful testing - API Server#

Crossbar#

  • REST-ish HTTP server
    • Initially webmachine
    • Cowboy + cowboy_rest
  • Basic CRUD operations
  • Call initiation and control
  • Hacks, hacks everywhere
    • HTTP clients that only GET/POST
    • HTTP clients that can't set Accept
    • HTTP clients that can't set Content-Type
  • No tests, siloed testing, regression central

Crossbar - Model#

  • Map representing high-level concepts
    • Accounts: #{AccountName => #{}=AccountInfo}
    • Phone Numbers: #{PhoneNumber => #{}=NumberProperties}
    • Dedicated IPs: #{IPAddress => #{}=IPInfo}
    • Ratedecks: #{RatedeckName => #{}=Rates}
  • Provides API auth credentials to endpoint modules

Crossbar - Endpoints#

  • Per-endpoint test modules
    • API actions become PropEr commands
    • next_state/3 updates the model (if necessary)
    • postcondition/3 checks the API response against the model
  • Per-endpoint API modules
    • Erlang SDK in hiding

Crossbar - HTTP endpoint#

  • Some Crossbar endpoints query a provided HTTP server
    • Storage
    • Webhooks
  • Generic cowboy server to accept those requests
  • Endpoint modules can test that requests are received and responded to as necessary

Crossbar - Testing#

  • Default pqc_runner mixes all endpoint commands
  • Helpers to run counterexamples
  • Helpers to print counterexamples
  • Helpers to create counterexamples

Crossbar - Counterexample#

    pqc_util:simple_counterexample().
    [{pqc_cb_ips,remove_ips,['{API}',<<"pqc_cb_ips">>]},
     {pqc_cb_accounts,create_account,['{API}',<<"pqc_cb_ips">>]},
     {pqc_cb_accounts,create_account,['{API}',<<"pqc_cb_ips">>]},
     {pqc_cb_ips,assign_ips,
                 ['{API}',<<"pqc_cb_ips">>,
                  [{dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]]},
     {pqc_cb_ips,create_ip,
                 ['{API}',
                  {dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]},
     {pqc_cb_accounts,create_account,['{API}',<<"pqc_cb_ips">>]},
     {pqc_cb_ips,delete_ip,
                 ['{API}',
                  {dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]},
     {pqc_cb_ips,create_ip,
                 ['{API}',
                  {dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]}]

Crossbar - Bugs found (so far!)#

  • Account create/delete/create race condition (sounds like John Hughes' DETS find!)
  • IP create/assign/delete semantic changes
  • Custom ratedeck phone number evaluation
    • Service plan rewrite regression

Crossbar - Future#

  • Generate more sample data using JSON schemas
  • More coverage of APIs
  • Calculate whether changeset is covered by test suite

Crossbar - Advice#

  • next_state/3 API result can be dynamic or concrete
    • Use "concrete" values as indexes; find dynamic values in SUT shims
  • Cleanup properly after testing
  • Write the properties from the docs (you have those, right?)

Wrapping Up#

Advice#

  • KISS for reals!
  • Property testing is a mindset and skillset
    • There will be a learning curve
  • Read 'Property-Based Testing with PropEr, Erlang, and Elixir' by Fred Hebert
  • Read the PropEr code
    • Especially as you get more practice
  • Practice!

Questions?#

Thanks!