/product-experiments

Product experiments

Turn a product question into a measurable decision. A feature flag without trustworthy exposure data and a decision rule is release control, not an experiment.

Choose a mode

• Design: create an experiment brief and analysis plan.
• Implement: add assignment, exposure tracking, metrics, and safeguards.
• Rollout-only: release safely when causal inference is unnecessary.
• Validate: audit instrumentation and assignment before launch.
• Analyze: estimate effects and diagnose data-quality failures.
• Conclude: decide ship, iterate, continue, or rollback and record why.

Keep design vendor-independent. Use an existing analytics/flag provider when present; add a new provider only when selected or explicitly authorized.

Experiment brief

Before implementation, record:

• product decision and causal hypothesis;
• mechanism: why treatment should change behavior;
• eligible population and exclusions;
• assignment unit, exposure unit, and identity transition rules;
• control and variants, experiment key, and immutable version;
• primary outcome with numerator, denominator, window, and direction;
• guardrails and diagnostic metrics;
• baseline, minimum detectable effect or smallest worthwhile effect, and uncertainty method;
• minimum observation/maturity window and stop rules;
• rollout stages, kill conditions, owner, and rollback path;
• action triggered by positive, neutral, harmful, or invalid results.

If inputs are unavailable, state what can be designed now and what must be measured before launch. Do not invent power or duration.

Workflow

1. Inspect the product, event taxonomy, identity model, analytics, flag system, existing experiments, and deployment constraints.
2. Choose assignment and exposure units that match the causal question. Address anonymous-to-authenticated identity, group assignment, repeat exposure, interference, and concurrent experiments.
3. Implement deterministic assignment or the provider's documented mechanism. Preserve assignment across requests and devices as required.
4. Capture one deduplicated exposure record at the point treatment can affect behavior. Include experiment key, version, variant, subject, timestamp, and relevant context. Do not substitute flag evaluation for exposure.
5. Instrument outcomes and guardrails with testable schemas. Verify that exposure joins to outcomes and that control/treatment event semantics match.
6. Launch at a safe initial allocation. Monitor errors, latency, data loss, sample-ratio mismatch, and guardrails before widening.
7. Analyze only after the planned maturity window unless a kill condition fires. Report effect size and uncertainty, not just significance. Check sample-ratio mismatch, missingness, novelty/carryover, peeking, multiple comparisons, censoring, and segment exploration.
8. Conclude against the prewritten decision rule. Separate invalid, inconclusive, practically neutral, beneficial, and harmful results.
9. Remove or graduate flags, document the decision, and verify the post-decision product state.

Provider guidance

• If PostHog is already selected, read PostHog implementation if present. If it is absent, inspect installed versions and use current official documentation rather than copying remembered SDK code.
• If the measurement layer is missing, use an available analytics capability or provide a minimal vendor-neutral event contract and identify the implementation blocker.

Safety

• Do not expose users to security, privacy, billing, or irreversible-risk variants without appropriate review.
• Minimize sensitive properties and define retention/access controls for experiment data.
• Do not recommend shipping from underpowered or invalid data.
• Never silently change the primary metric or exclusions after seeing results.

Output contract

Return the applicable artifact:

• experiment brief and analysis plan;
• implementation changes and event schemas;
• pre-launch validation report;
• staged rollout/rollback plan;
• analysis with data-quality checks, effect sizes, uncertainty, and limitations;
• decision record with follow-up and flag cleanup.

Verify

• Assignment is stable and matches the intended unit.
• Exposure is deduplicated, versioned, and recorded when treatment can act.
• Exposure joins to outcome and guardrail events.
• Control and treatment pass functional tests.
• Kill switch, provider failure behavior, and rollback are tested.
• Sample-ratio mismatch and event completeness are checked.
• Analysis honors the registered population, windows, and decision rule.
• Final decision includes practical significance, uncertainty, limitations, and cleanup.

PostHog implementation adapter

Use this adapter only when PostHog is already installed or explicitly selected. Inspect the installed SDK versions and current official documentation before editing code; API names, hosts, flag behavior, and shutdown requirements can change.

Detect the existing integration

Inspect:

• package manifests and lockfiles;
• client and server initialization;
• configured host/data region;
• identity calls and anonymous-to-known merges;
• event naming/schema conventions;
• existing flag keys, experiments, cohorts, and groups;
• serverless/runtime lifecycle and error handling.

Extend the project's conventions. Do not add a second client or duplicate provider.

Assignment and exposure

Feature-flag evaluation is not automatically a trustworthy experiment exposure.

For an experiment:

1. evaluate with the intended stable distinct ID or group key;
2. obtain the variant and immutable experiment version;
3. render or execute the treatment;
4. capture one exposure when treatment can affect behavior;
5. deduplicate repeated evaluations according to the analysis unit;
6. include variant, experiment version, subject type, timestamp, and eligibility context.

Verify in the provider's current documentation whether its experiment product captures exposure automatically and under which SDK/path. Do not double-capture.

Client versus server evaluation

Prefer server evaluation for routing, permissions, pricing, API behavior, and initial-render content when the runtime supports it. Client evaluation can be appropriate for post-hydration UI behavior.

Test:

• loading/unknown state;
• provider timeout or outage;
• identity unavailable;
• assignment consistency across server and client;
• hydration/layout effects;
• data-region host and key type;
• serverless flush/shutdown behavior.

Feature flags are not authorization. Enforce access control independently.

Local and test overrides

Use the project's existing override mechanism or add a development/test-only provider abstraction. Prevent production activation through environment validation. Tests should cover control, every variant, unknown flag, provider failure, and kill switch.

Rollout checklist

• Start disabled or at the agreed safe allocation.
• Confirm eligibility and exclusions with known test identities.
• Observe exposure and outcome events in the intended project/environment.
• Reconcile variant counts and check sample-ratio mismatch.
• Monitor errors, latency, support signals, and guardrails.
• Widen only when the prewritten rollout gate passes.
• Document graduation/removal and stale-flag cleanup.

Security and privacy

• Use the correct project key or server credential for the installed SDK; never guess from variable names.
• Keep management/personal API credentials out of client bundles.
• Minimize person properties and avoid sensitive experiment context.
• Follow the project's consent, retention, and deletion behavior.
• Treat provider dashboards as external systems with least-privilege access.