Key A/B testing concepts explained

The Region of Practical Equivalence in statistics delineates a zone around a baseline where differences, though statistically significant, are considered trivial in practical terms. In experimentation, ROPE enables the early discontinuation of variations that are statistically unlikely to outperform the baseline, helping you avoid wasting time and resources on ineffective changes. Further, if you want to test whether a variation is equivalent or better (but not worse) than the baseline,…

A website with responsive design adapts its layout to that of the viewing environment (mobile, desktop or tablet) to provide an optimum viewing experience to visitors.

Retargeting is the practice whereby website visitors are shown ads/banners about the things they earlier browsed but didn't convert. Retargeting is used to push those visitors for a conversion who have already shown some interest in the website.

Revenue per visitor is a business statistic used to calculate the overall revenue that each visitor to your website generates.

A sales funnel displays the customer journey visually and illustrates the sales process from awareness to action.

Sample Ratio Mismatch (SRM) arises when traffic allocation to the groups in an A/B test deviates from the intended distribution. Maintaining intended sample size allocation between control and treatment groups is crucial for accurate test results and sound decision-making based on them. Hence, early detection and correction of SRM are critical to optimizing test resources and achieving reliable outcomes.

A sample size is the number of visitors needed to run a test to produce statistically significant results. Selecting the right sample size ensures optimal utilization of resources and shipping of test ideas that have higher chances of winning.

Scarcity is a psychological tactic that marketers use to push visitors towards a conversion action. It takes advantage of the human fear of 'shortage' or in today's terms, FOMO (fear of missing out) - “Only 5 items left” and “Sale only till 8 PM” are some examples where scarcity marketing is used.

A scrollmap is a type of website heatmap that provides a visual representation of a user’s scrolling behavior. It offers insights into how users scroll across a specific website area, allowing for the identification of UX improvements, the implementation of changes, and a boost in conversions.

Search engine marketing is the use of paid advertising to ensure that the products and services offered by your company are visible on search engine results pages (SERPs).

SEO is the process of optimizing a website or web page to improve its ranking in search engine results pages (SERPs).

SERPs is a search engine result page that displays the best results for a user's search query.

Sequential testing correction consists of techniques used in statistical analysis to mitigate the risk of increased false positives while continuously monitoring test statistics for significance. Sequential testing can increase the chance of declaring a variation to be a winner when it is actually equivalent to the baseline. That is where sequential testing correction helps control this risk by adjusting the level of confidence required before declaring something as significant.

Server-side testing is the process of experimentation that involves rendering the website variation at the server level before sending it to a user’s browser.

Session recording is a powerful tool that provides a recorded presentation of how a user navigates a website or mobile application. It operates by recording visitor actions such as clicks, mouse movements, and page scrolling as they move across the website.

Simpson's paradox is a statistical phenomenon in which a trend or characteristic observed within individual data groups undergoes a reversal or disappearance when these groups are aggregated.

Social proof is a psychological phenomenon based on the idea that individuals are more likely to adopt certain behaviors or make certain decisions if they see others doing the same thing.

Split testing, also known as A/B testing, is a method used to compare two versions of a webpage to determine which one performs better.

In a Split-URL test, you create multiple versions of your website hosted on different URLs. On these versions, traffic is distributed randomly and a randomized controlled experiment is conducted. The performance of each of the variant is monitored and evaluated to identify the most effective variation. A split test enables you to determine which of multiple variations of your web pages has a higher conversion rate for your visitors.

A squeeze page is a web page designed to collect email addresses from potential consumers.

The degree to which the result of an A/B test can be apart from the actual figure is called standard error.

In a hypothesis test, a result is declared statistically significant if one can rule out the possibility to obtain a result by a random chance, and instead attribute it to a truly existing effect.

The test duration comes from a sample size calculator which estimates the number of samples needed to estimate a significant effect within the chosen error bounds correctly. This calculation is performed before running an experiment to plan resources such that a timeline of an experiment is decided.

A test hypothesis is a proposed explanation for a phenomenon that can be tested through experiments or observations. It serves as a tentative answer to a research question and is formulated based on prior observations, theories, or logical deductions.

Testing in production is a standard practice in the modern software development cycle to ensure the release of high-quality products and improve user experience. Using this methodology, marketers can run tests to understand users’ responses to any new feature releases, while product managers can use these data to improve product experiences.

The title tag is an HTML element used to define the title of a web page.

Trust badges are small logos or icons that convince visitors about the safety and credibility of the website.

In an A/B test, if both variations are similar and don’t affect the metric being tested any differently, an error may occur where the null hypothesis is rejected after the test concludes. In such a case, if it’s determined that there is a statistical difference between the variations, the result is a Type I error.

During the hypothesis testing process, when the competing variations affect the experiment's metric differently but the test fails to reject the null hypothesis (representing no effect), then it is called a Type-I error or a False Negative.

Unique visitors are internet users who access websites using a distinct IP address.

Upselling is a sales technique where a merchant encourages a customer to purchase more expensive or upgrade to a higher-end version of products or services that they are already interested in buying.

Usability refers to the ease of use of a website. The usability of a website can be tested by hired experts, who thoroughly analyze the website and provide suggestions for improvement.

Usability testing is a technique for evaluating a product/app/website by testing it on a group of users with no prior exposure to it. It is carried out to enable the creation of a successful, productive, and engaging system based on user feedback.

User intent is the intention or goal users have in mind when making search queries. Creating content that meets user intent is imperative if you want your page to rank highly for the keyword you’ve targeted on the SERP. This is because Google considers user intent when displaying relevant search results to users for specific search queries.

A user journey map is a visual representation of the steps and experiences that a user goes through in interacting with a product or service.