Home / Internet Marketing News / Conversion Optimization Testing: Validity threats from running multiple tests at the same time

Conversion Optimization Testing: Validity threats from running multiple tests at the same time

A/B testing is popular among marketers and businesses because it gives you a way to determine what really works between two (or more) options.

However, to truly extract value from your testing program, it requires more than simply throwing some headlines or images into a website testing tool. There are ways you can undermine your testing tool that the tool itself can’t prevent.

It will still spit out results for you. And you’ll think they’re accurate.

These are called validity threats. In other words, they threaten the ability of your test to give you information that accurately reflects what is really happening with your customer. Instead, you’re seeing skewed data from not running the test in a scientifically sound manner.

In the MECLABS Institute Online Testing certification course, we cover validity threats like history effect, selection effect, instrumentation effect and sampling distortion effect. In this article, we’ll zoom in on one example of a selection effect that might cause a validity threat and thus misinterpretation of results — running multiple tests at the same time — which increases the likelihood of a false positive.

Interaction Effect — different variations in the tests can influence each other and thus skew the data

The goal of an experiment is to isolate a scenario that accurately reflects how the customer experiences your sales and marketing path. If you’re running two tests at the same time, the first test could influence how they experience the second test and therefore their likelihood to convert.

This is a psychological phenomenon known as priming. If we talk about the color yellow and then I ask you to mention a fruit, you’re more likely to answer banana. But if we talk about red and I ask you to mention a fruit, you’re more likely to answer apple. 

Another way interaction effect can threaten the validity is with a selection effect. In other words, the way you advertise near the beginning of the funnel impacts the type of customer and the motivations of the customer you’re bringing through your funnel.

Taylor Bartlinski, Senior Manager, Data Analytics, MECLABS Institute, provides this example:

“We run an SEO test where a treatment that uses the word ‘cheap’ has a higher clickthrough rate than the control, which uses the word ‘trustworthy.’ At the same time, we run a landing page test where the treatment also uses the word ‘cheap’ and the control uses ‘trustworthy.’  The treatments in both tests with the ‘cheap’ language work very well together to create a higher conversion rate, and the controls in each test using the ‘trustworthy’ language work together just as well.  Because of this, the landing page test is inconclusive, so we keep the control. Thus, the SEO ad with ‘cheap’ language is implemented and the landing page with ‘trustworthy’ language is kept, resulting in a lower conversion rate due to the lack of continuity in the messaging.”

Running multiple tests and hoping for little to no validity threat

The level of risk depends on the size of the change and the amount of interaction. However, that can be difficult to gauge before, and even after, the tests are run.

“Some people believe (that) unless you suspect extreme interactions and huge overlap between tests, this is going to be OK. But it is difficult to know to what degree you can suspect extreme interactions. We have seen very small changes have very big impacts on sites,” Bartlinski says.

Another example Bartlinski provides is where there this is little interaction between tests. For example, testing PPC landing pages that do not interact with organic landing pages that are part of another test — or testing separate things in mobile and desktop at the same time. “This lowers the risk, but there still may be overlap. It’s still an issue if a percentage gets into both tests; not ideal if we want to isolate findings and be fully confident in customer learnings,” Bartlinski said.

How to overcome the interaction effect when testing at the speed of business

In a perfect scientific experiment, multiple tests would not be run simultaneously. However, science often has the luxury of moving at the speed of academia. In addition, many scientific experiments are seeking to discover knowledge that can have life or death implications.

If you’re reading this article, you likely don’t have the luxury of taking as much time with your tests. You need results — and quick. You also are dealing with business risk, and not the high stakes of, for example, human life or death.

There is a way to run simultaneous tests while limiting validity threats — running multiple tests on (or leading to) the same webpage but splitting traffic so people do not see different variations at the same time.

“Running mutually exclusive tests will eliminate the above validity threats and will allow us to accurately determine which variations truly work best together,” Bartlinski said.

There is a downside though. It will slow down testing since an adequate sample size is needed for each test. If you don’t have a lot of traffic, it may end up taking the same amount of time as running tests one after another.

What’s the big idea?

Another important factor to consider is that the results from grouping the tests should lead to a new understanding of the customer — or what’s the point of running the test?

Bartlinski explains, “Grouping tests makes sense if tests measure the same goal (e.g., reservations), they’re in the same flow (e.g., same page/funnel), and you plan to run them for the same duration.”

The messaging should be parallel as well so you get a lesson. Pointing a treatment ad that focuses on cost to a treatment landing page that focuses on luxury, and then a treatment ad that focuses on luxury pointing to an ad that focuses on cost will not teach you much about your customer’s motivations.

If you’re running multiple tests on different parts of the funnel and aligning them, you should think of each flow as a test of a certain assumption about the customer as part of your overall hypothesis.

It is similar to a radical redesign. Much like testing multiple steps of the funnel can cause an interaction effect, testing multiple elements on a single landing page or in a single email can cause an attribution issue. Which change caused the result we see?

Bartlinski provides this example, “On the same landing page, we run a test where both the call-to-action (CTA) and the headline have been changed in the treatment. The treatment wins, but is it because of the CTA change or the headline? It is possible that the increase comes exclusively from the headline, while the new CTA is actually harming the clickthrough rate. If we tested the headline in isolation, we would be able to determine whether the combination of the new headline and old CTA actually has the best clickthrough, and we are potentially missing out on an even bigger increase.”

While running single-factorial A/B tests is the best way to isolate variables and determine with certainty which change caused a result, if you’re testing at the speed of business you don’t have that luxury. You need results and you need them now!

However, if you align several changes in a single treatment around a common theme that represents something you’re trying to learn about the customer (aka radical redesign), you can get a lift while still attaining a customer discovery. And then, in follow-up single-factorial A/B tests, narrow down which variables had the biggest impact on the customer.

Another cause of attribution effect is running multiple tests on different parts of a landing page because you assume they don’t interact. Perhaps, you run a test on two different ways to display locations on a map in the upper left corner of the page. Then a few days later, while that test is still running, you launch a second test on the same page but in the lower right corner on how star ratings are displayed in the results.

You could assume these two changes won’t have an effect on each other. However, the variables haven’t been isolated from the tests, and they might influence each other. Again, small changes can have big effects. The speed of your testing might necessitate testing like this; just know the risk involved in terms of skewed results.

To avoid that risk, you could run multivariate tests or mutually exclusive tests which would essentially match each combination of multiple variables together into a separate treatment. Again, the “cost” would be that it would take longer for the test to reach a statistically significant sample size since the traffic is split among more treatments.

Test strategically

The big takeaway here is — you can’t simply trust a split testing tool to give you accurate results. And it’s not necessarily the tool’s fault. It’s yours. The tool can’t possibly know ways you are threatening the validity of your results outside that individual split test.

If you take a hypothesis-driven approach to your testing, you can test fast AND smart, getting a result that accurately reflects the real-world situation while discovering more about your customer.

You might also like:

Online Testing certification course — Learn a proven methodology for executing effective and valid experiments

Optimization Testing Tested: Validity threats beyond sample size

Validity Threats: 3 tips for online testing during a promotion (if you can’t avoid it)

B2B Email Testing: Validity threats cause Ferguson to miss out on lift from Black Friday test

Validity Threats: How we could have missed a 31% increase in conversions

The post Conversion Optimization Testing: Validity threats from running multiple tests at the same time appeared first on MarketingExperiments.

Click Here For Original Source Of The Article

Ads by WOW Trk

About Daniel Rodgers

A lot of news that you will not see in the paper. A lot of technology that is coming out that will not see in the paper.

Check Also

Lovestruck: Why We’ve Fallen in Love with B2B Content Marketing

Falling in Love with B2B Content Marketing

Once upon a time, in a land of rainbows, butterflies, and evil content overload, a gripping love story is unfolding.

But this love affair isn’t what you’d expect — it’s no “Romeo and Juliet” or “Sleepless in Seattle.”

Rather, it’s an unlikely courtship with a bond sealed by three little words: Business-to-Business.

That’s right. We’ve said. The secret’s out. For the last decade, we’ve been head-over-heels in love with B2B content marketing.

We know what you’re thinking. B2B can’t hold a candle to B2C. B2B is boring. B2B lacks sex appeal. But you don’t know B2B like we do—she’s a knockout and a showstopper.

via GIPHY

What’s so darn attractive about B2B content marketing? Let us remind you of all there is to fall in love with.

We’ve Fallen in Love With B2B Content Marketing Because ...

… It Forms Genuine (Love) Connections

Audiences are made up of people with specific needs and interests. Part of the challenge — and the thrill — is creating a B2B content marketing strategy that addresses those needs and interests to make our audience fall in love.

via GIPHY

“I love the ever-present challenge of speaking to our audience on topics they genuinely care about and in a way that really connects with them. Seeing big results really pays off all of the audience research and thought behind the content,” Elizabeth Williams, a longtime account managers, says.

And speaking of the real people we’re trying to woo with our content, those people are why Influencer Marketing Strategist Jack Fitzpatrick is enamored B2B content marketing.

“I love B2B content marketing because it is a constant reminder that businesses are made up of actual people,” he says. “When crafting an influencer marketing or social media campaign, we are creating content for people. Small businesses, large corporations, and every business in between, started with people having a vision and striving to reach it. That’s a powerful concept to me.”

[bctt tweet="I love B2B content marketing because it is a constant reminder that businesses are made up of actual people. - @fitzJackrick" username="toprank"]

… It Nurtures Buyers

Nothing is more rewarding than hearing those three little words from prospects and turning them into loyal companions thanks to our data-informed content strategies. Just take it from our Associate Director of Search & Analytics Tiffani Allen.

“What I love most about B2B content marketing is the journey,” she says. “It’s rarely transactional, but instead focuses on providing the right information at the right time. It’s almost like solving a puzzle. You have to analyze the prospect’s journey – not only through your own properties but in the digital sphere at large.”

[bctt tweet="What I love most about #B2B #contentmarketing is the journey ... It's like solving a puzzle. - @Tiffani_Allen" username="toprank"]

… It Combines Analysis With Creativity

Fresh out of graduating high school, I was confronted with the same question most graduates face: “What do you want to do with your life?” After courting professions like biology, psychology, economics, and several others, I struggled to find exactly what I was looking for. No occupation seemed to fulfill my need for a creative outlet and exercise my analytical mind. Until I found marketing.

B2B content marketing allows you to be both analytical and creative. Through analysis, you can uncover powerful content opportunities that produce desired results. And through creativity, you can make sure those content opportunities are executed in a way that excites and entertains audiences.

via GIPHY

This targeted, thoughtful approach is what helps keep the job interesting and drive such substantial results. Debbie Friez, Influencer Marketing Strategist, agrees.

“B2B marketing is targeted, so you can really look at how to move specific audiences to awareness of our client and buy from our client,” she says. “With my job specifically, I am able to work with influencers who may not have millions of followers, but are highly-regarded in their industry.”

[bctt tweet="#B2Bmarketing is targeted, so you can really look at how to move specific audiences to awareness of our client and buy from our client. - @dfriez" username="toprank"]

… It Presents Constant Learning Opportunities

Some of the best dates are where you and your partner experience new things together. And in B2B content marketing, every day can be a new experience, writing about fresh, emerging topics with varying levels of complexity. That is especially true for those of us that do content marketing for large enterprises or a content marketing agency

Our own creative content genius and Senior Content Marketing Manager Joshua Nite says: “I love the opportunity to learn new things. We have clients in finance, cloud-based software, IoT, healthcare... when we work on content with them, I get to learn about it all. It's endlessly fascinating.”

[bctt tweet="We have clients in finance, cloud-based software, IoT, healthcare... when we work on content with them, I get to learn about it all. It's endlessly fascinating. - @NiteWrites on what he loves about B2B #contentmarketing." username="toprank"]

… It Solves Problems

(Great) B2B content marketing isn’t just putting a message out there for the sake of it. It’s about solving a real, tangible problem for your ideal customers. That’s why Account Manager Jane Bartel is so enchanted.

“One of my favorite things about (good) B2B content marketing is that it's so focused on adding value for the audience,” she says. “More than in the B2C space, B2B content marketers are asking themselves, ‘is the content I'm producing going to help the person consuming it solve a problem?’”

3 B2B Content Marketing Programs We Fell In Love With

TopRank and B2B content marketing is a match made in heaven, and the evidence of our love is in the work we create. The below campaigns gave our team plenty of reasons to fall madly in love with B2B content marketing and we believe they’ll provide plenty of inspiration for you as well.

Prophix: AI in Finance

Interactive. Multimedia. Triple-digit increases. What’s not to love?

Several of our team members, including Josh and Elizabeth, are quite fond of our recent content campaigns for Prophix (including our AI in Finance interactive eBook) the most for the reasons listed above and more.

AI Finance Influencer Content Campaign

“We created something absolutely unique, with the audio plus the interactive interface plus our computer personality, Penny. I love that it's so original and, quite frankly, cool, and I'm proud of the results it's getting, too!” Josh said.

Elizabeth adds, “I have just LOVED working on our last few campaigns with Prophix. I love the partnership we have developed where we can bounce ideas off each other, get super creative with our concepts, and test multimedia formats and interactive content.”

Read the full case study to get a look behind the scenes at how our interactive content drove triple-digit results for Prophix.

SAP Purpose: Helping the World Run Better

Today, business is more about doing “good” than being “good.” In fact, purpose-driven businesses that strive to leave the world in a better state than they found it in have more loyal customers and also feel a stronger emotional connection to the brand.

Our team fell in love with sharing inspirational stories from purpose-driven businesses, NGO’s, nonprofits, and thought leaders for SAP this year through their SAP Purpose campaign.

Interactive Influencer Content Program Focused on Purpose

“Sharing the stories of these Forward Thinkers making a difference in our world inspired our team along with the members of SAP’s audience. Seeing these leaders converge on one project sharing their passions and pursuits was so powerful!” says Jack.

Hyper-Targeted Content Strategies

With continued opportunities to refine and make improvements, Jane Bartel loves our ongoing B2B content strategy programs for clients like DivvyHQ, LinkedIn, and more.

“The project I'm most in love with at the moment: Planning and building a collection of resources for one particular segment of a client's audience. It's targeted, valuable, and makes great use of new and existing content,” Jane points out.

A specific favorite among the team is the seo-driven B2B content program we’ve been working on in partnership with Antea Group USA, a leading EHS and sustainability consulting firm, for the past three years. We’ve made a consistent commitment to each other, and results we love are in the air. From 2017 to 2018, total blog traffic has grown a whopping 149%, and organic blog traffic has grown 183%.

One of the pieces we're absolutely gaga over is this piece featuring hot EHS tips from a Demogorgon.

EHS Tips from a Demogorgon

Plenty of Love to Go Around

B2B content marketing is the perfect partner. Our relationship has a solid foundation and we continue to grow together, instead of apart. It affords us the opportunity to learn. It nurtures our buyers. It helps us connect with target audiences. It solves important problems. What’s not to love?

We aren’t the only ones who love everything B2B marketing. For more inspiration, make sure to follow these B2B marketing influencers.

The post Lovestruck: Why We’ve Fallen in Love with B2B Content Marketing appeared first on Online Marketing Blog - TopRank®.

Leave a Reply

Your email address will not be published. Required fields are marked *

css.php