Mobile Extensions + testtrack-cli

This release is feature-complete for testtrack-cli-driven split migrations that have feature parity with rails activerecord migrations.

Mobile Extensions

A set of backend features to support long-lived clients in the field and platform-agnostic split management (i.e. not just rails migrations).

While these features on their own don't have major visible effects in TestTrack, in conjunction with a forthcoming testtrack CLI and Mobile iOS and Android libraries, they'll enable single-branch development for mobile with remote kill of features that you discover client bugs in after release.

Recreating splits

Adds the ability to recreate a previously existing split from a migration

Application-wide feature flags

To go along with this update to the test_track_rails_client Betterment/test_track_rails_client#60, this adds a default "TestTrack" app and an "app_id" identifier type to the seeds file, so that consumer apps all don't have to create their own "app_id" identifier type.

For new apps, the seeds will run as part of rake db:setup, and for existing apps, you will have to run rake db:seed. It is designed to be idempotent, and accept any already existing apps with the name "TestTrack" or existing identifier types with the name "app_id".

Experience Event Sampling

This release adds the ability to instruct the latest clients to send events for only a random representative sampling of feature gate experience events.

Freewheeling Feature Gate Assignments

This marks the first release that treats feature gates differently from experiments as TestTrack begins to specialize those concepts. Conventionally, feature gates are already splits with names ending in _enabled, and with two variants - true and false. We'll begin adding split migration syntax to enforce that convention in future versions.

As of version 1.2, when the server receives a deterministic pseudorandom assignment event from a client for a feature gate (any split with a name ending in _enabled), it will ignore the event and not record the assignment. This means that unless a visitor's assignment is explicitly set to a variant (either via the chrome extension or TT admin), changes in variant weightings will instantly change the variant seen by clients in the field, even those who have already experienced that split.

Because of the way deterministic pseudorandom assignment works, every visitor in variant X at weighting = N is guaranteed to remain in that variant for weighting > N, but probabilistically will switch to variant !X as N approaches 0, providing as much continuity of user experience as possible while enabling freewheeling.

This has a couple desireable effects:

• we can slow roll features quickly in small percentage chunks as we establish confidence in a feature's correctness, and roll them back instantly because there's no "stickiness" for feature gate assignments
• we record less data, which reduces operational cost and bends the scale curve

Split Detail Endpoint includes Variant Detail lazily-initialized display_name

This resolves a problem with the test_track_rails_client where on the server we were leaning on lazily initialized records to display certain pieces of information. The client would only display variant_details that had been updated, so unless the data was updated manually the rows would be empty. This allows a default value of the Split variant name to be displayed if the administrator display_name has not been set.

Split and Variant Details

This new release allows split test administrators to add details about the split and its variants to help elucidate why a user might be experiencing a particular version of a feature, and what they might be seeing. It includes an API to expose that information outside of the TestTrack experience as well.

Fix DoS vulnerability in SplitConfigs controller

There was a bug in the authentication code that would allow an unauthenticated attacker to mark a running split as complete, breaking clients in the field that depended on the split.

This vulnerability has been assigned CVE identifier CVE-2017-1000068.

Affected versions:

All versions of TestTrack less than 1.0.1

Impact:

AuthenticatedApiController, the base class of all authenticated endpoints, was making incorrect use of authenticate_with_http_basic. authenticate_with_http_basic does not evaluate the provided block if there is no basic auth header in the request, allowing request processing to proceed if no creds at all were provided.

The impact was mitigated by the fact that most authenticated endpoints require current_app in order to fulfill their functions. The exception was the SplitConfigsController, which would deactivate any split provided by an unauthenticated client.

The fix was two-fold:

• Disallow finishing splits not owned by current_app
• use authenticate_or_request_with_http_basic, which will halt the filter chain and return 401 if no HTTP basic auth credentials are provided.

Releases:

The fixed version - v1.0.1 - is available via github.

Credits:

Thanks to @agirlnamedsophia for identifying the underlying authentication vulnerability.

Initial release

First official release of the TestTrack server. TestTrack has been in production use for a while, and master has been a stable, deployable artifact, but we're leveling up our open source release process.