Property-based testing with KAZOO#
Introduction#
Me#
KAZOO#
- https://github.com/2600hz/kazoo
- Started in 2010
- Telecom platform
- Clustering layer over FreeSWITCH / Kamailio
- API-driven
- Scales from Hobbyists to Enterprise
- Built on:
- RabbitMQ
- CouchDB
- 275K lines of Erlang in the core project in 1315 modules
- 106 contributors (~25 that are 2600Hz)
Stateless testing - JSON#
JSON#
- kz_json.erl
- Provides lists-esque functionality - maps/folds/filters/etc
- getters/setters, merging, diffing, and more
- Hides data structure used - enforced throughout the code
- kz_json_tests.erl and kz_json_generators.erl
- Need to generate "deep" objects but not too deep
- Naive approach:
test_object()
- Generate list of Key/Value pairs
- Better approach:
deep_object()
- Symbolic calls to build up the object
test_object()
generated#
1> proper_gen:pick(kz_json_generators:test_object()).
{ok,{[{<<14,141,161>>,-19},
{<<37,53,158>>,<<>>},
{<<"&">>,<<>>},
{<<81,197,72,47,41,80,75,41,19>>,<<>>},
{<<155,65,38,243,136,74,115>>,<<>>},
{<<176,5,171,200>>,<<>>},
{<<"ãO">>,<<>>}]}}
test_object()
spread#
2> proper:quickcheck(kz_json_tests:prop_test_object_gen(), 1000).
58% {0,2}
38% {2,4}
2% {4,6}
true
deep_object()
generated#
1> {ok, Calls} = proper_gen:pick(kz_json_generators:deep_object()).
{ok, {'$call',kz_json,set_value,
[<<"ðg">>,
[-4,<<>>,<<>>,<<>>,
[<<>>,<<>>,{[{<<19,184,115,217,157,45,202,135,28>>,<<>>}]}],
<<>>,<<>>,<<>>],
{'$call',kz_json,set_value,
[<<"~áä±õOv·@">>,[],
{'$call',kz_json,set_value,
[<<133,141>>,
[],
{'$call',kz_json,set_value,
...
deep_object()
eval'd#
2> proper_symb:eval([], Calls).
{[{<<4,133,215,252,0>>,[]},
{<<"lËÉx&'">>,[]},
{<<182,179,144,154>>,[]},
{<<132,250,21,171,119,26,197,90,175,188>>,
[{[{<<98,36,70,211,105,95,174,109,130>>,<<>>},
{<<4,35,100,156,67,58,75,203,168,89,107>>,<<>>},
{<<190,144,146,45,53,85,153,231,166,84,233>>,<<>>},
{<<35,223,73,21,92,176,167,254,8>>,<<>>},
{<<214,210,66,21,57,117>>,<<>>},
{<<226,152,179,17>>,<<>>},
{<<107,119,61,244,188,157,110,28>>,
{[{<<133,246,158,227,95,35,251,39,...>>,<<>>},
...
deep_object()
spread#
3> proper:quickcheck(kz_json_tests:prop_deep_object_gen(), 1000).
56% {2,4}
37% {0,2}
5% {4,6}
0% {6,8}
true
JSON#
- More control over terms generated
- More control over depth
- Create EUnit tests from failing PropEr tests
- Shrinking is huge!
- Actually had good EUnit test coverage prior
Stateless testing - Bindings server#
Bindings#
- AMQP-style bindings
- Routing key: "a.b.c"
- Binding key: "*.b.#"
- "*" - match one segment
- "#" - match 0 or more segments
- Credit to Fred Hebert @mononcqc
Bindings - Generator: expanded_path()#
Generates a binding key, a routing key, and whether there should be a match.
1> proper_gen:pick(kazoo_bindings_tests:expanded_paths()).
{ok,[{<<"*.1Oj.#.t863f4e3Xu">>,<<"1Oj.t863f4e3Xu">>,false}
,{<<"*.1Oj.#.t863f4e3Xu">>,<<"lLTW1.1Oj.t863f4e3Xu">>,true}
]}
Bindings - Binding Key#
- Binding key: "*", "1Oj", "#", "t863f4e3Xu"
- Match any first segment
- Match "1Oj" as second segment
- Match 0 or more segments
- Match "t863f4e3Xu" as last segment
Bindings - Matching routing key#
- Matching Routing key: "lLTW1" "1Oj" "t863f4e3Xu"
- "lLTW1" matches "*"
- "1Oj" matches "1Oj"
- 0 matches for "#"
- "t863f4e3Xu" matches "t863f4e3Xu"
Bindings - Failing routing key#
- Failing Routing Key: "1Oj" "t863f4e3Xu"
- "1Oj" matches "*"
- "t863f4e3Xu" does not match "1Oj"
Bindings - Found bugs#
- Had "reasonable" EUnit tests
- 2011-06-12: Introduced PropEr testing, first 5 bugs found were generic
- 2011-06-13: First "weird" match
{<<"#.6.*.1.4.*">>, <<"6.a.a.6.a.1.4.a">>}
- 2016-12-19: Found two more failing
{<<"*.u.*.7.7.#">>,<<"i.u.e.7.7.7.a">>}
{<<"#.c.#.c.#">>, <<"c.c">>}
- 2017-01-27: Found one more
{<<"#.Z.*.9.0.#">>,<<"1.Z.7.9.0.9.a.0">>}
- 2017-02-27: Found one more
{<<"W0.*.m.m.#">>, <<"W0.m.m.m.5">>}
Stateful testing - LRU Cache#
Cache#
- LRU cache in ETS
- Maintains "origin pointers" - links to the datastore
- AMQP events can evict cache entries
- Maintains "monitors"
- Wait for a key to be cached or timeout
- Callbacks on events
- 'timeout', 'expire', 'flush', 'erase', 'store'
- Cache stampede mitigation
- Provides a way to block calling processes while a key's value is being computed
Cache - API commands#
- store
- peek
- fetch
- erase
- wait_for_key
- mitigate_stampede
- wait_for_stampede
- timer:sleep/1
Cache - Model#
- Track cache as a proplist
- Track "time" as # of milliseconds
- Increment "time" on each timer:sleep/1
- Expire entries
Cache - Sample commands#
1> proper_gen:pick(kz_cache_pqc:command({state, [], 0})).
{ok,{call,kz_cache,store_local,
[kz_cache_pqc,113,8,[{expires,1}]]}}
2> proper_gen:pick(kz_cache_pqc:command({state, [], 0})).
{ok,{call,kz_cache,erase_local,[kz_cache_pqc,99]}}
Cache - Running the tests#
1> proper:quickcheck(kz_cache_pqc:correct()).
...
13% {kz_cache,peek_local,2}
13% {kz_cache,mitigate_stampede_local,3}
13% {kz_cache,wait_for_stampede_local,3}
13% {timer,sleep,1}
12% {kz_cache,fetch_local,2}
12% {kz_cache,erase_local,2}
11% {kz_cache,store_local,4}
10% {kz_cache,wait_for_key_local,3}
true
Cache - Challenges with time#
- Model-only pass is "accurate" on expiration
- SUT pass is subject to the VM and timers may not fire at the precise timeout
- Mostly works for LRU testing though - rare that the tests fail due to time
Stateful testing - API Server#
Crossbar#
- REST-ish HTTP server
- Initially webmachine
- Cowboy +
cowboy_rest
- Basic CRUD operations
- Call initiation and control
- Hacks, hacks everywhere
- HTTP clients that only GET/POST
- HTTP clients that can't set
Accept
- HTTP clients that can't set
Content-Type
- No tests, siloed testing, regression central
Crossbar - Model#
- Map representing high-level concepts
- Accounts:
#{AccountName => #{}=AccountInfo}
- Phone Numbers:
#{PhoneNumber => #{}=NumberProperties}
- Dedicated IPs:
#{IPAddress => #{}=IPInfo}
- Ratedecks:
#{RatedeckName => #{}=Rates}
- Accounts:
- Provides API auth credentials to endpoint modules
Crossbar - Endpoints#
- Per-endpoint test modules
- API actions become PropEr commands
- next_state/3 updates the model (if necessary)
- postcondition/3 checks the API response against the model
- Per-endpoint API modules
- Erlang SDK in hiding
Crossbar - HTTP endpoint#
- Some Crossbar endpoints query a provided HTTP server
- Storage
- Webhooks
- Generic cowboy server to accept those requests
- Endpoint modules can test that requests are received and responded to as necessary
Crossbar - Testing#
- Default pqc_runner mixes all endpoint commands
- Helpers to run counterexamples
- Helpers to print counterexamples
- Helpers to create counterexamples
Crossbar - Counterexample#
pqc_util:simple_counterexample().
[{pqc_cb_ips,remove_ips,['{API}',<<"pqc_cb_ips">>]},
{pqc_cb_accounts,create_account,['{API}',<<"pqc_cb_ips">>]},
{pqc_cb_accounts,create_account,['{API}',<<"pqc_cb_ips">>]},
{pqc_cb_ips,assign_ips,
['{API}',<<"pqc_cb_ips">>,
[{dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]]},
{pqc_cb_ips,create_ip,
['{API}',
{dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]},
{pqc_cb_accounts,create_account,['{API}',<<"pqc_cb_ips">>]},
{pqc_cb_ips,delete_ip,
['{API}',
{dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]},
{pqc_cb_ips,create_ip,
['{API}',
{dedicated,<<"1.2.3.4">>,<<"a.host.com">>,<<"zone-1">>}]}]
Crossbar - Bugs found (so far!)#
- Account create/delete/create race condition (sounds like John Hughes' DETS find!)
- IP create/assign/delete semantic changes
- Custom ratedeck phone number evaluation
- Service plan rewrite regression
Crossbar - Future#
- Generate more sample data using JSON schemas
- More coverage of APIs
- Calculate whether changeset is covered by test suite
Crossbar - Advice#
next_state/3
API result can be dynamic or concrete- Use "concrete" values as indexes; find dynamic values in SUT shims
- Cleanup properly after testing
- Write the properties from the docs (you have those, right?)
Wrapping Up#
Advice#
- KISS for reals!
- Property testing is a mindset and skillset
- There will be a learning curve
- Read 'Property-Based Testing with PropEr, Erlang, and Elixir' by Fred Hebert
- Read the PropEr code
- Especially as you get more practice
- Practice!
Questions?#
Thanks!