diff --git a/README.md b/README.md index eedf277d..872fbb6a 100644 --- a/README.md +++ b/README.md @@ -154,6 +154,44 @@ One the values that a given visitor will be assigned for a split, e.g. `true` or ### Weighting Variants are assigned pseudo-randomly to visitors based on their visitor IDs and the weightings for the variants. Weightings describe the probability of a visitor being assigned to a given variant in integer percentages. All the variant weightings for a given split must sum to 100, though variants may have a weighting of 0. +### Experiment + +Experiments are the standard flavor of splits in TestTrack. They are +intended to be used for A/B testing, and the TestTrack server records +visitors' experienced variants so that those visitors will continue to +experience the same variant regardless of subsequent changes to the +weightings of those variants via the admin interface. + +Storing the variant a visitor experienced for an experiment also allows +TestTrack to provide a consistent UX to a customer who experienced a +new-to-them experiment before logging in on a new device, only to be +recognized as an existing visitor upon sign-in. TestTrack will merge +all variant assignments from the anonymous visitor into the +authenticated visitor at sign-in as long as the authenticated visitor +doesn't have conflicting assignments. In that case, the authenticated +visitor's previous assignments win. + +### Feature Gate + +As of TestTrack version 1.2, splits with names ending in the `_enabled` +suffix will be treated as feature gates. Feature gates differ from +experiments in that they are not intended to be used for analysis, and +therefore it is not important that the user remain in the same variant +throughout the entire split lifecycle. Feature gates are meant to be +slow-rolled (incrementally increasing the percentage of customers +experiencing the new feature), released en masse, or instantly rolled +back. + +To facilitate these smooth transitions and rapid toggles, the TestTrack +server will not record variant assignments when a visitor experiences a +split. This means that every time a visitor experiences a split, they +will be deterministically (pseudorandomly) assigned to a variant based +on their visitor ID and the name of the split. This will provide the +customer with a stable variant given a constant split weighting, but +probablistically increase the percentage of visitors experiencing the +the `true` variant as the split weightings are increased via the admin +panel, giving an admin full control over the feature's release. + ### IdentifierType A name for a customer identifier that is meaningful in your application, typically things that people sign up as, log in as. They should be expressed in `snake_case` and conventionally are prefixed with the application name that the identifier is for, e.g. `myapp_user_id`, `myapp_lead_id`. diff --git a/app/models/deterministic_assignment_creation.rb b/app/models/deterministic_assignment_creation.rb index 2f02dba5..5731e7c6 100644 --- a/app/models/deterministic_assignment_creation.rb +++ b/app/models/deterministic_assignment_creation.rb @@ -14,13 +14,15 @@ def self.create!(params) end def save! - ArbitraryAssignmentCreation.create!( - visitor_id: visitor_id, - split_name: split_name, - variant: variant, - mixpanel_result: mixpanel_result, - context: context - ) + unless split.feature_gate? + ArbitraryAssignmentCreation.create!( + visitor_id: visitor_id, + split_name: split_name, + variant: variant, + mixpanel_result: mixpanel_result, + context: context + ) + end end def variant diff --git a/app/models/split_creation.rb b/app/models/split_creation.rb index 19d70c27..d9a8dd77 100644 --- a/app/models/split_creation.rb +++ b/app/models/split_creation.rb @@ -25,7 +25,11 @@ def weighting_registry=(registry) end def split - @split ||= app.splits.create_with(registry: merged_registry).find_or_initialize_by(name: name) + @split ||= app.splits.create_with(registry: merged_registry, feature_gate: feature_gate?).find_or_initialize_by(name: name) + end + + def feature_gate? + name.end_with?("_enabled") end private diff --git a/app/views/admin/split_configs/new.html.erb b/app/views/admin/split_configs/new.html.erb index 58d21189..c2bb6f9c 100644 --- a/app/views/admin/split_configs/new.html.erb +++ b/app/views/admin/split_configs/new.html.erb @@ -5,6 +5,22 @@
+ This split is a feature gate. Changing weights will immediately change + behavior of visitors who do not have an explicit assignment, even if they've + already experienced a specific variant of this split. This is usually + desirable for slow-rolling features. +
+<% else %> ++ This split is an experiment. Changing weights will have no immediate effect + on the behavior of visitors who have already experienced a variant of this + split. Experiments rarely benefit from changing weightings unless you are + performing analysis over a date range. +
+<% end %> + <%= simple_form_for(@split_creation, url: admin_split_split_config_path) do |f| %> <% f.simple_fields_for :weighting_registry do |ff| %> @@ -18,6 +34,6 @@ <%= ff.input variant, as: :percent, input_html: { value: weight, class: "weight-input" } %> <% end %> - <%= render "shared/form_footer", f: f, submit_text: "Edit", submit_disable_with_text: "Changing..." %> + <%= render "shared/form_footer", f: f, submit_text: "Save", submit_disable_with_text: "Changing..." %> <% end %> <% end %> diff --git a/app/views/admin/splits/_details.html.erb b/app/views/admin/splits/_details.html.erb index a22e8cac..7a9c497e 100644 --- a/app/views/admin/splits/_details.html.erb +++ b/app/views/admin/splits/_details.html.erb @@ -9,7 +9,9 @@+ + * Feature gates no longer track assignment events and population reflects only visitors assigned to specific variants via the chrome extension or admin tool. +
+ <% end %> diff --git a/app/views/admin/splits/_variants.html.erb b/app/views/admin/splits/_variants.html.erb index 1984b9b2..f878eef8 100644 --- a/app/views/admin/splits/_variants.html.erb +++ b/app/views/admin/splits/_variants.html.erb @@ -1,7 +1,7 @@