Ten Steps to Building Successful Optimizely Web Experiments

As we saw in Part 1 of our series on Optimizely Web Experimentation, creating and launching an effective A/B or multivariate test involves a meticulous process of strategizing, planning, and defining the structures needed for success. In this post, we will dive deeper into the details of delivery, reviewing all of the steps required to configure an experiment, and the process of fully leveraging the capabilities of Web Experimentation.

In our previous post, we reviewed why the ideation and planning stages for Optimizely Web Experimentation are crucial for ensuring tests are strategic and aligned with business objectives. You begin with defining clear, measurable goals, such as increasing checkout completion rates or improving user engagement on key landing pages. Based on these goals, the team develops specific, testable hypotheses – educated guesses about how changing an element (e.g., headline, button, layout) might impact user behavior to achieve the desired outcome. Since resources are limited, prioritizing experiments using frameworks like ICE (Impact, Confidence, Ease) or PIE (Potential, Importance, Ease) is essential to focus on ideas with the highest expected return. Finally, crafting the right team, typically involving stakeholders from marketing, product, design, development, and analytics, ensures the necessary skills and perspectives are available to design, build, launch, and interpret experiments effectively, fostering a collaborative culture of optimization.

Once you have a solid charter in place, the next step is to convert your plans into concrete experiments within Optimizely. Here are ten steps to get you up and running.

1. Creating a New Experiment

The first action within Optimizely is to create a new experiment. Typically, this involves clicking a button labeled “Create New” or a similar call to action, followed by selecting “Experiment.” You will then be prompted to give your experiment a descriptive and easily understandable name. This name should reflect the core change being tested and perhaps a brief identifier. For instance, “Product Page CTA Test – Buy Now vs. Learn More” is a clear and informative name.

After naming the experiment, you’ll usually be presented with options regarding the experiment type. Optimizely supports various types, including A/B tests (testing one variable with two or more variations), multivariate tests (testing multiple variables simultaneously), and personalization campaigns. For a standard A/B test focusing on a single element like a button, you would select the A/B test option.

2. Defining the Pages

The Pages section is crucial for specifying where your experiment will run. You need to tell Optimizely which URLs or URL patterns should trigger the experiment for your website visitors. This ensures that only the intended users on the designated pages will be included in the test.

Optimizely offers several ways to define Pages:

• Exact URL: This is used when the experiment should run on a single, specific page. You would enter the full URL of that page (e.g., ‘www.example.com/product/123’).
• URL Contains: This option allows you to target a group of pages that share a common string in their URL. For example, entering /blog/ would target all pages within the blog section of your website.
• URL Matches Regex: This provides the most flexibility, allowing you to use regular expressions to define complex URL patterns. This is useful for targeting pages with dynamic URLs or specific structures.
• URL Starts With: This targets all pages whose URLs begin with a specified string (e.g., ‘www.example.com/checkout’).
• URL Ends With: This targets all pages whose URLs end with a specified string (e.g., .html).

When configuring Pages, it’s essential to be precise. Incorrectly defined pages can lead to the experiment running on unintended parts of your website, skewing your results and potentially impacting user experience negatively. You can often add multiple page targeting rules to ensure comprehensive coverage of the desired areas.

3. Creating Variations

The Variations section is where you define the different versions of the element or page you are testing. In an A/B test, you will typically have a “Original” (the control) and one or more variations.

To create a Variation, you click an “Add Variation” button. You will then be prompted to give the variation a name that clearly describes the change being implemented (e.g., “CTA – Buy Now”).

Once the Variation is created, you need to use Optimizely’s visual editor (or code editor, depending on your preference and the complexity of the changes) to make the desired modifications. The visual editor allows you to point and click on elements of the page and change their text, color, size, position, and more, without needing to write code. For more complex changes involving structural modifications or JavaScript interactions, you might need to use the code editor.

For our example hypothesis, you would navigate to the product page within the visual editor (or target the specific element using CSS selectors in the code editor) and change the text of the primary call-to-action button in the “CTA – Buy Now” variation from “Learn More” to “Buy Now.”

You can create multiple Variations to test different approaches simultaneously. For instance, you might have a third Variation with the button text “Add to Cart.” However, remember that increasing the number of variations will require more traffic to achieve statistical significance within a reasonable timeframe.

4. Defining Events

Events in Optimizely are the user interactions you want to track as part of your experiment. These are the actions that indicate whether a user is moving towards your desired outcome. Common Events include button clicks, form submissions, page views (of specific confirmation pages), video plays, and downloads.

To define an e|Event, you typically click an “Add Event” button and give it a descriptive name (e.g., “Purchase Completed”). You then need to specify how Optimizely should track this Event. This can be done in several ways:

• Track Clicks on Element: You can target a specific HTML element (like a button or a link) and track when users click on it. This usually involves using CSS selectors to identify the element.
• Track Page Views: You can track when users view a specific page, often a thank-you or confirmation page that signifies the completion of a goal. You would define this using the same URL matching rules as in the “Pages” section.
• Custom Events: For more complex tracking scenarios, you can implement custom JavaScript code that triggers an Optimizely event when a specific action occurs. This requires some coding knowledge but offers greater flexibility.

For our example, the primary Event would be “Purchase Completed,” which would likely be tracked as a page view of the order confirmation page or through a custom event triggered by the successful completion of the purchase process.

It’s crucial to accurately define and implement your Events, as these are the data points that will be used to measure the performance of your variations. You should test your event tracking thoroughly to ensure it’s firing correctly before launching your experiment.

5. Setting Up Metrics

Metrics are the quantifiable measures you will use to evaluate the success of your experiment. They are derived from the Events you have defined. A Metric typically represents the conversion rate of a specific Event.

To create a Metric, you click an “Add Metric” button in your experiment and give it a clear name (e.g., “Purchase Conversion Rate”). You then need to associate it with one or more of the Events you have already defined. For our example, the “Purchase Conversion Rate” metric would be linked to the “Purchase Completed” event.

Optimizely allows you to define primary and secondary metrics. The primary metric is the main measure of success for your experiment, directly tied to your hypothesis. Secondary metrics are other indicators you want to monitor to understand the broader impact of your changes. For instance, while the primary metric might be purchase conversion rate, you might also track metrics like product page views or the number of users who proceed to the checkout process as secondary indicators.

You can also define how the metric should be calculated (e.g., unique conversions vs. total conversions) and its direction of improvement (increase or decrease).

6. Defining Audiences

Audiences allow you to target your experiment to specific groups of visitors based on various criteria. This ensures that your experiment is only shown to the users you are interested in studying.

Optimizely offers a wide range of audience targeting options:

• Behavioral Targeting: Based on past actions on your website, such as the number of pages viewed, specific products viewed, or previous purchases.
• Demographic Targeting: Based on inferred or known demographic information (if available), such as location, age, or gender.
• Technological Targeting: Based on the visitor’s browser, operating system, device type (desktop, mobile, tablet), or screen resolution.
• Traffic Source Targeting: Based on how the visitor arrived at your website (e.g., direct traffic, search engines, social media, specific campaigns).
• Custom Attributes: You can pass custom data about your users to Optimizely (e.g., logged-in status, customer segment, subscription level) and use this for targeting.

To create an audience, you click an “Create New Audience” button and then define the rules based on the available criteria. You can combine multiple rules to create highly specific audience segments.

In our example, if our hypothesis specifically targets users who have viewed at least three product pages, we would create an Audience based on this behavioral criterion. Applying this audience to our experiment would ensure that only these users are included in the test.

7. Configuring Experiment Settings

Beyond the core components of Pages, Variations, Events, Metrics, and Audiences, there are several other important settings to configure for your experiment:

• Traffic Allocation: You need to decide what percentage of eligible visitors should be included in the experiment. You can start with a smaller percentage (e.g., 25% or 50%) to monitor the initial impact before rolling it out to a larger audience. You also need to determine how the traffic should be split between the original and the variations. For an A/B test with one variation, you would typically split the traffic evenly (e.g., 50% to the original, 50% to the variation).
• Goals: While metrics define how success is measured, goals provide a high-level overview of what you are trying to achieve with the experiment. You can associate your primary metric with a goal (e.g., “Increase Purchase Conversion Rate”).
• Integration Settings: Optimizely integrates with various analytics platforms (like Google Analytics), allowing you to send experiment data to these tools for more in-depth analysis. You should ensure these integrations are properly configured.
• Scheduling: You can set a start and end date and time for your experiment. This is useful for running tests during specific periods, such as marketing campaigns or seasonal events.

8. Quality Assurance and Testing

Before launching your experiment to live traffic, thorough quality assurance (QA) is essential. This involves:

• Verifying that the variations appear correctly on the targeted pages across different browsers and devices.
• Ensuring that the event tracking is firing accurately when the defined user actions are performed.
• Checking that the audience targeting rules are working as expected and that only the intended users are seeing the experiment.
• Testing any custom JavaScript code to ensure it’s functioning without errors.

Optimizely provides preview modes and QA tools to help with this process. You should also involve other team members in the QA process to get different perspectives and catch any potential issues.

9. Launching the Experiment

Once you are confident that everything is configured correctly and the QA process is complete, you can launch your experiment. This involves changing the experiment status from “Draft” or “Inactive” to “Running.”

10. Monitoring and Analysis

After launching, it’s crucial to monitor the performance of your experiment closely. Optimizely provides a results dashboard where you can track the performance of your variations against your defined metrics in real-time. You should pay attention to key metrics like conversion rates, statistical significance, and any unexpected behavior.

Allow your experiment to run for a sufficient duration to gather enough data to reach statistical significance. The required duration will depend on factors like your website traffic, the baseline conversion rate, and the magnitude of the difference between your variations.

Once the experiment has run for an adequate period, you can analyze the results to determine which variation (if any) performed significantly better. Optimizely provides statistical analysis tools to help you understand the likelihood that the observed differences are due to the changes you made rather than random chance.

The final step is to take action based on the results of your experiment. If a variation shows a statistically significant improvement in your primary metric, you would typically implement that change on your website for all users. If no variation performs significantly better than the original, you have learned that this particular change did not have the desired impact, and you can use these learnings to inform future experiments.

Creating an experiment in Optimizely Web Experimentation is a multi-faceted process that requires careful attention to detail across several key areas. By thoughtfully configuring Pages, Variations, Events, Metrics, and Audiences, and by conducting thorough QA, you can ensure that your experiments are well-designed, accurately tracked, and provide valuable insights for optimizing your website and achieving your business goals. Remember that experimentation is an iterative process, and the learnings from each test will help you refine your hypotheses and drive continuous improvement.

If you’re ready to launch experiments with Optimizely Web Experimentation, reach out to Relationship One. We can help you plan, configure, and optimize.