Optimizely Web Experimentation: Interpreting Your Results

Throughout our three-part series, we have seen that several critical phases are required to ensure the optimization of digital experiences through Optimizely Web Experimentation. The process starts with strategic planning and hypothesis development, moves through the careful construction and launch of experiments, and culminates in analysis and iteration. Part three in our series focuses on interpreting results within Optimizely, understanding its statistical foundations, handling various outcomes (winning, losing, or inconclusive), and fostering a culture of continuous improvement through long-term iteration.

The Optimizely Stats Engine

At the core of Optimizely’s analytical capabilities is its sophisticated Stats Engine. This engine is designed specifically to calculate experiment results accurately and determine statistical significance efficiently. It moves beyond traditional A/B testing methods, which often rely on fixed horizon calculations requiring a predetermined sample size before analysis can even begin.

Instead, Optimizely primarily employs a sequential testing approach. It analyzes the data cumulatively, repeatedly checking if a statistically significant difference has emerged between variations. The primary benefit of this approach is speed and efficiency; experiments can often be concluded the moment significance is reached, rather than forcing teams to wait unnecessarily for an arbitrary sample size target, allowing for faster learning cycles and quicker implementation of winning variations.

Furthermore, the Stats Engine incorporates methods to control the False Discovery Rate (FDR). This is particularly critical when an experiment includes multiple goals or metrics being tracked simultaneously. Evaluating multiple metrics increases the inherent risk of encountering false positives (detecting a significant effect where none actually exists simply due to chance). FDR control intelligently manages this risk across all tracked metrics, ensuring greater confidence in the overall findings and preventing teams from acting on misleading signals. The robust combination of sequential testing and FDR control provides a foundation for trustworthy results delivered efficiently.

Understanding Statistical Significance

Statistical significance essentially quantifies the probability that an observed difference between your experiment’s variations (like a change in conversion rate) is a genuine effect resulting from the changes you made, rather than being a product of random chance or normal fluctuations in user behavior.

Optimizely expresses this concept through a confidence level, defaulting to the industry standard of 95%. When a result reaches statistical significance at a 95% confidence level, it implies there is only a 5% probability that the observed difference occurred purely due to random variation. Conversely, it suggests a 95% probability that the difference represents a real effect caused by the variation.

It is vital to recognize what statistical significance does and does not indicate:

• It measures the likelihood that the observed effect is real, not just random noise.
• It does not measure the magnitude or business importance of the effect; a statistically significant result could be very small.
• It is crucial for avoiding false positives – acting on results that aren’t actually real.
• It is typically expressed via a confidence level (e.g., 90%, 95%), which is the inverse of the p-value threshold.

Relying on statistical significance ensures that decisions are based on reliable evidence, preventing organizations from chasing random fluctuations or implementing changes based solely on observation or intuition.

Interpreting Confidence Intervals

While statistical significance answers the question “Is there likely a real difference?”, confidence intervals address the equally important question: “How big is that difference likely to be?”. A confidence interval provides a calculated range within which the true effect size (the actual uplift or downlift caused by the variation) likely lies, based on the observed data.

For instance, a winning result might display a 95% confidence interval of [+2%, +8%] for the conversion rate lift. This means we can be 95% confident that the true improvement generated by the variation, compared to the original, falls somewhere between a 2% lift and an 8% lift. This range offers valuable context for decision-making.

Key aspects of interpreting confidence intervals include:

• Provides a likely range: It estimates the boundaries for the true uplift or downlift.
• Direct link to significance: If the confidence interval for the difference between a variation and the baseline does not include zero (e.g., [+2%, +8%] or [-5%, -1%]), the result is statistically significant at that confidence level. If it does include zero (e.g., [-1%, +5%]), the result is inconclusive because a zero or negative effect is statistically plausible.
• Indicates precision: Narrower intervals (e.g., [+4%, +5%]) suggest higher precision and more certainty about the effect size, often resulting from larger sample sizes or lower data variability. Wider intervals (e.g., [+1%, +15%]) indicate less precision.
• Assesses business impact: The range helps gauge whether the potential impact, even at the low end of the interval, is meaningful enough to warrant implementation.

Understanding confidence intervals allows teams to move beyond a simple “significant/not significant” verdict and evaluate the potential business value and certainty associated with an observed effect.

Winning, Losing, and Inconclusive Results

Based on the calculations performed by the Stats Engine, considering both statistical significance and confidence intervals for the experiment’s primary metric, Optimizely categorizes results into one of three types:

• Winning Result: A variation demonstrates a statistically significant positive impact on the primary metric compared to the baseline or other variations. Its confidence interval for the difference will be entirely above zero. This indicates high confidence that the changes genuinely improved performance for the main goal.
• Losing Result: A variation shows a statistically significant negative impact on the primary metric. Its confidence interval for the difference will be entirely below zero. This reliably suggests the changes were detrimental to the key performance indicator.
• Inconclusive Result: The Stats Engine could not detect a statistically significant difference (either positive or negative) for the primary metric at the chosen confidence level. The confidence interval includes zero. This doesn’t definitively mean there’s no difference, but rather that the collected data doesn’t provide enough evidence to confidently distinguish a real effect from random noise. This can occur due to a very small actual effect size, high variability in user behavior, or insufficient sample size.

Iterate Insights into Action

The real work often begins after the Stats Engine renders its verdict. The Iterate phase is about translating these results – whether winning, losing, or inconclusive – into actionable next steps and feeding those learnings back into the optimization cycle. How an organization handles each type of result is critical for maximizing the value of their experimentation program.

Winning Result

When faced with a Winning Result, the initial reaction is often celebration, followed by implementation. However, careful consideration is still required. Before rolling out the change site-wide, it’s wise to examine secondary metrics. Did the win on the primary metric come at the cost of another important indicator? For example, did simplifying a form increase submissions (primary win) but decrease the quality or value of those submissions (secondary loss)? Segmentation is also key. Did the variation win across all major audience segments (e.g., new vs. returning, mobile vs. desktop), or was the lift driven by a specific group? Understanding these nuances ensures the implementation is truly beneficial overall. Once verified, the winning variation should be prioritized for development and deployment. Importantly, a win shouldn’t necessarily be the end of the line for that area of the user experience. It provides a new, improved baseline upon which further hypotheses can be built. Could the winning design be refined further? Could adjacent elements now be optimized to complement the successful change? A win is both a conclusion and a starting point for the next iteration.

Losing Result

Losing Results can be just as valuable, if not sometimes more valuable, than wins. A statistically significant loss provides clear, reliable feedback that a particular hypothesis was incorrect or that the implemented change harmed the user experience or conversion goals. The primary action is typically to not implement the change and ensure the losing variation is archived. However, the crucial step is to analyze why it lost. Did the change introduce usability issues? Did it conflict with established user mental models or expectations? Did it negatively impact clarity or trust? Digging into session recordings, heatmaps, or conducting qualitative user feedback related to the losing variation can uncover deep insights into user behavior and preferences. These learnings are invaluable for refining future hypotheses. Documenting why something didn’t work prevents teams from repeating mistakes and helps build a stronger understanding of the user base. Embracing losses as learning opportunities is a hallmark of a mature experimentation culture; failing fast and learning efficiently is a strategic advantage.

Inconclusive Results

Inconclusive Results often require the most critical thinking and strategic decision-making. An inconclusive outcome doesn’t mean the experiment was a waste; it simply means the data didn’t provide a clear direction with sufficient confidence. There are several ways to proceed. One option is to let the experiment run longer, gathering more data. If the results are hovering near the significance threshold and the potential impact is high, acquiring a larger sample size might increase the precision and potentially push the result into winning or losing territory. However, this must be balanced against the opportunity cost of occupying testing slots and delaying other experiments. Another powerful approach is segmentation. Even if the overall result is inconclusive, specific audience segments (defined by device, traffic source, behavior, etc.) might show a significant win or loss. Analyzing these segments can uncover pockets of opportunity or reveal that a change benefits one group while slightly harming another, leading to personalization strategies rather than a site-wide change. Alternatively, an inconclusive result might prompt a re-evaluation of the hypothesis or execution. Was the change too subtle to realistically produce a detectable effect? Was the hypothesis fundamentally flawed? Were there technical glitches (like slow loading of the variation) that might have muddied the data? If the potential impact was deemed low initially, or if other tests hold higher promise, the team might decide to archive the experiment and move on, documenting the outcome and any tentative observations. Handling inconclusive results effectively involves assessing the potential value, the quality of the data and hypothesis, and the strategic priorities of the optimization program.

Optimizing for the Long Term

The true potential of Optimizely Web Experimentation unfolds when it fuels a continuous, long-term process of iteration and learning, rather than just executing sporadic, disconnected tests.

1. Establish a Knowledge Repository: This is fundamental for cumulative learning. Every experiment, regardless of outcome, generates valuable data points and insights. Maintain a centralized location (like a wiki, spreadsheet, or dedicated platform) to document:
   • The original hypothesis and its rationale.
   • Detailed descriptions or screenshots of the variations tested.
   • Key metrics, final results (including significance and confidence intervals), and segment performance.
   • Learnings derived – the interpretation of why the result occurred.
   • Decisions made (e.g., implemented, archived, follow-up test planned). This repository prevents redundant efforts, informs future hypotheses with past data, accelerates onboarding, and demonstrates the program’s impact.
2. Build on Previous Learnings: Use the knowledge repository to ensure that future tests are informed by past results. Wins create new baselines. Losses define boundaries and highlight user sensitivities. Inconclusive results point towards areas needing different approaches or deeper investigation. Each experiment should ideally build upon the collective understanding gained from prior tests.
3. Expand Scope and Complexity: As the team and program mature, move beyond simple A/B tests. Explore multivariate testing (MVT) to understand the interaction effects of multiple changes simultaneously. Test more significant changes in user flows, information architecture, or core features. Integrate Optimizely with other data sources (Analytics, CRM, CDP) for more sophisticated targeting and measurement, leading towards advanced personalization strategies.
4. Foster Cross-functional Collaboration: Optimization is a team sport. Ensure continuous input from various departments – marketing, product management, UX/UI design, development, analytics, customer support. Each function brings unique perspectives and data points that can inspire powerful hypotheses and help interpret results more holistically.
5. Maintain a Strategic Roadmap: Regularly review and prioritize the experimentation backlog based on potential impact, confidence, ease, and alignment with current business objectives. An experimentation roadmap provides direction and ensures resources are focused on the most promising opportunities.

Embedding this iterative mindset into the organizational culture transforms experimentation from a tactic into a strategic capability for driving continuous improvement.

Driving Continuous Improvement

The Analyze and Iterate phases are the heartbeat of a successful Optimizely Web Experimentation program. They bridge the gap between running tests and achieving real-world improvements. Through rigorous analysis grounded in a solid understanding of Optimizely’s Stats Engine, statistical significance, and confidence intervals, teams can confidently interpret whether a change resulted in a win, a loss, or an inconclusive outcome. More importantly, by thoughtfully handling each result type – implementing and iterating on wins, learning deeply from losses, and strategically navigating inconclusive findings – organizations can unlock immense value. By committing to this cycle and fostering a culture of long-term, data-driven iteration supported by a robust knowledge base and cross-functional collaboration, businesses can leverage experimentation to deliver consistently better user experiences and achieve sustained growth in their digital endeavors.

If you’re ready to start analyzing your experiments or looking for ways to drive continuous optimization with your process, call Relationship One. We are ready to help.