Imagine you run an online business and have just redesigned your product page. You've moved the checkout button above the product details. You're very excited to see if this new layout change will drive more sales. But how do you know if the redesign is making a difference?

This is where A/B testing comes in. It's how companies at scale test their features before rolling them out to all customers. It is a method used to compare two versions of a webpage or app, to determine which one performs better.

So this is how you run an A/B test for your redesigned product page:

You show half the users the original version (A) and the other half the new version (B). Then you check if more people are buying on the new page. By comparing the two groups, you can see if the redesign works.

However, it's surprisingly very easy to conduct a biased or inconsistent test, which in turn skews the result.

For a test to yield meaningful insights, two things are crucial: fairness and consistency.

Fairness ensures that every user has an equal chance of seeing either the original or the redesigned page, while consistency guarantees that the same user always sees the same version, no matter how many times they visit or from which device.

So how do you decide which users see which version? It's definitely not about manually assigning each user a variant, unless you want to become the office legend for the most tedious task ever!

A simple approach is to use the Random Number Strategy.

Random Number Strategy

The concept is as simple as flipping a coin.

The approach suggests randomly generating a number, either 0 or 1. If the number is 0, show them the original layout(Variant A), otherwise show them the new layout(Variant B).

if(randomNumber is 0) -> Variant A
if(randomNumber is 1) -> Variant B

Try out the demo below to understand how this strategy works.

This ensures that every user has an equal chance of seeing either version, making the test fair. But what about consistency?

To ensure users see the same variant on repeat visits, we can store the assigned variant in a cookie or session. However, if the user logs in from a different device, we cannot get the assigned variant because cookies and sessions are device-specific.

Another approach to solve this issue is to store the assigned variant in a database. But this also means we fetch the variant details everytime the user loads the page. This will increase latency and impact the performance of the application, especially at scale.

We need a strategy that consistently assigns the same variant across devices, and the Random Number Strategy falls short in this regard.

The Random Number Strategy is great for fairness but lacks consistency when users switch devices.

Hashing Strategy

We ruled out using cookies, session and the database. So, how do we ensure consistency?

What if we derive the variant from userID? This way, we can be always sure that for a given userID the variant is always the same.

A userID is a unique identifier, often an alphanumeric string, assigned to each user in your system (e.g., user_12345 or abc123xyz). So if we could somehow convert this string into a number and then assign the variant based on this number, we could ensure consistency.

So, how do we convert the userID into a number?

We could use a technique called hashing.

Think of hashing as a black box that takes in an input and consistently produces a fixed-length output for the same input.

So we hash the userID to generate a numeric value.

Finally, we perform the modulo operation to distribute users evenly across the variants.

variant = hashedUserID % totalVariants

In our case, with two variants, this would return either 0 or 1.

If the result is 0, we assign variant A, otherwise variant B.

Try out the demo below to see how the Hashing Strategy works for various hashed userIDs.

This approach seems like it's satisfying both the requirements: fairness & consistency.

You've implemented this on your website and finding the new layout is getting you more sales. You're thrilled!

But it's not as much as you were expecting. You decide to test with another new layout. This time you move the checkout button to the right side of the page making it more prominent.

You decide to modify the test by adding this new layout as Variant C. So now you have three variants to test.

All you have to do is simply increase the total number of variants in the modulo operation.

variant = hashedUserID % 3

Try out the demo below to understand how for a user the variant assignment changes when we introduce more variants into the test.

This way we can introduce more variants without affecting the consistency of the test.

Let's go back to the scenario where we had only two variants. Now imagine you're seeing significant increase in sales with the new layout.

You decide to roll out the new layout to all of your users. However that's a bad idea.

Why?

Because there could be many factors that could have influenced the increase in sales. It could be the holiday season, a marketing campaign, or even a bug in the original layout.

It's recommended to run the test for a longer period to ensure that the new layout is indeed driving more sales.

Now, you might be wondering, "But what about the potential sales I'm losing during this stupid test?"

There's a way to solve this. You can introduce traffic allocation to the test.

By using traffic allocation, you can gradually increase the percentage of users seeing the new layout. This way, you are gaining confidence in the new layout's effectiveness and also minimizing any potentials loss in sales.

So how do we increase traffic to the new layout?

Well, this is where Hashing Strategy takes a step back. It's not possible with this strategy because we don't have control over the percentage of users. It is the modulo operator that is evenly distributing the users to the variants.

The Hashing Strategy is great for simple A/B tests but fails to provide a way to control traffic distribution.

Hash Bucketing Strategy

We learned Hashing Strategy does not provide a way to control traffic distribution.

One way to deal with this is to find out a way to control percentage of users who gets to see the new variant.

For example, if a user was part of the first 50% of users, we let them see Variant A. But how do we determine which percentage of users the user belongs to?

We could use bucketing to acheive this. But what exactly is a bucket?

Think of bucket as a container that holds a group of users.

So, we basically try to distribute users into these buckets. How many? It depends on the user base. Ideally, it's better to have a number of buckets that allows granular control over the percentage of users assigned to each variation. For example, using 100 buckets lets you split traffic as precisely as 1% increments, while 1,000 buckets allow for even finer adjustments, down to 0.1%.

Each user will be assigned to a bucket instead of a variant. Why? Because this bucket will help us determine which percentage of users the user falls into. For example, if we have 10 buckets, each bucket will represent 10% of users. A users who belongs to bucket 1 will be in the first 10% of users.

And then we map each bucket to a specific variant. For example, if we want 50% of our users to see Variant A, we map the first 50% of buckets to Variant A and the remaining 50% to Variant B.

Let's summarize what we learned till now:

• Assign a bucket to a user based on their userID.

• Map each bucket to a specific variant.

The approach begins by hashing userID to generate consistent numeric value. The hashed value is then used to assign user to a specific bucket.

bucket = hashedUserID % totalBuckets

In the following demo, we used 10 buckets to demonstrate how bucket assignment works for a given hashed userID.

The next step is to map each bucket to a specific variant. It's basically like saying the first 50% of buckets will be mapped with variant A and the remaining 50% with variant B.

For example, in our case, if a user falls between buckets 1 to 5, we show them layout A, while users falling between buckets 6 to 10 see layout B.

Hope that explains how hash bucketing works for simple use cases.

Now let's try to solve the traffic distribution problem we faced earlier with Hashing Strategy.

If we want to increase the traffic to the new layout, we can simply increase the number of buckets mapped to the new layout.

For example, if we want 40% of users to see old layout and 60% to see the new layout, we can map the first 40% of buckets to the old layout and the remaining 60% to the new layout.

This approach gives you precise control over how you distribute traffic across variants.

Try the demo below to see how variant assignment changes when we slowly increase/decrease traffic allocation for a variant.

Alas! You are now able to slowly roll out the success variant to all your users.

Let's see how we can introduce more variants into the test using this strategy. You can simply redistribute traffic so that each variant gets a fair share of users.

Try out the below demo to see how bucket mapping changes when more variants are introduced.

With Hash Bucketing Strategy, you have a fine control over the percentage of users looking at each variant. This will help us to make informed decisions on which variant to roll out to all users.

Conclusion

While Hashing Strategy is great for simple A/B tests, it lacks control when you want to roll out the new feature incrementally to all users.

The Hash Bucketing Strategy, however, stands out for its ability to balance both fairness and consistency while giving you fine control over how you distribute the traffic in a test.

In fact, at Contentstack's Personalize, we allow customers to perform A/B tests on their platform using the Hash Bucketing Strategy under the hood.

Acknowledgements

Thank you @Suvish, @Srinadh, @Sairaj, @Amey and @Vini (in no particular order) for your help with proofreading and feedback. Your support made a big difference!

AI was used for grammar corrections.

About me

Sai Santhosh Pentakota

I write code, I click pictures. One needs rational thinking, the other runs on instinct.

Passion for building smooth interfaces. Software Engineer at Contentstack.