Home / Internet Marketing News / Conversion Optimization Testing: Validity threats from running multiple tests at the same time

Conversion Optimization Testing: Validity threats from running multiple tests at the same time

A/B testing is popular among marketers and businesses because it gives you a way to determine what really works between two (or more) options.

However, to truly extract value from your testing program, it requires more than simply throwing some headlines or images into a website testing tool. There are ways you can undermine your testing tool that the tool itself can’t prevent.

It will still spit out results for you. And you’ll think they’re accurate.

These are called validity threats. In other words, they threaten the ability of your test to give you information that accurately reflects what is really happening with your customer. Instead, you’re seeing skewed data from not running the test in a scientifically sound manner.

In the MECLABS Institute Online Testing certification course, we cover validity threats like history effect, selection effect, instrumentation effect and sampling distortion effect. In this article, we’ll zoom in on one example of a selection effect that might cause a validity threat and thus misinterpretation of results — running multiple tests at the same time — which increases the likelihood of a false positive.

Interaction Effect — different variations in the tests can influence each other and thus skew the data

The goal of an experiment is to isolate a scenario that accurately reflects how the customer experiences your sales and marketing path. If you’re running two tests at the same time, the first test could influence how they experience the second test and therefore their likelihood to convert.

This is a psychological phenomenon known as priming. If we talk about the color yellow and then I ask you to mention a fruit, you’re more likely to answer banana. But if we talk about red and I ask you to mention a fruit, you’re more likely to answer apple. 

Another way interaction effect can threaten the validity is with a selection effect. In other words, the way you advertise near the beginning of the funnel impacts the type of customer and the motivations of the customer you’re bringing through your funnel.

Taylor Bartlinski, Senior Manager, Data Analytics, MECLABS Institute, provides this example:

“We run an SEO test where a treatment that uses the word ‘cheap’ has a higher clickthrough rate than the control, which uses the word ‘trustworthy.’ At the same time, we run a landing page test where the treatment also uses the word ‘cheap’ and the control uses ‘trustworthy.’  The treatments in both tests with the ‘cheap’ language work very well together to create a higher conversion rate, and the controls in each test using the ‘trustworthy’ language work together just as well.  Because of this, the landing page test is inconclusive, so we keep the control. Thus, the SEO ad with ‘cheap’ language is implemented and the landing page with ‘trustworthy’ language is kept, resulting in a lower conversion rate due to the lack of continuity in the messaging.”

Running multiple tests and hoping for little to no validity threat

The level of risk depends on the size of the change and the amount of interaction. However, that can be difficult to gauge before, and even after, the tests are run.

“Some people believe (that) unless you suspect extreme interactions and huge overlap between tests, this is going to be OK. But it is difficult to know to what degree you can suspect extreme interactions. We have seen very small changes have very big impacts on sites,” Bartlinski says.

Another example Bartlinski provides is where there this is little interaction between tests. For example, testing PPC landing pages that do not interact with organic landing pages that are part of another test — or testing separate things in mobile and desktop at the same time. “This lowers the risk, but there still may be overlap. It’s still an issue if a percentage gets into both tests; not ideal if we want to isolate findings and be fully confident in customer learnings,” Bartlinski said.

How to overcome the interaction effect when testing at the speed of business

In a perfect scientific experiment, multiple tests would not be run simultaneously. However, science often has the luxury of moving at the speed of academia. In addition, many scientific experiments are seeking to discover knowledge that can have life or death implications.

If you’re reading this article, you likely don’t have the luxury of taking as much time with your tests. You need results — and quick. You also are dealing with business risk, and not the high stakes of, for example, human life or death.

There is a way to run simultaneous tests while limiting validity threats — running multiple tests on (or leading to) the same webpage but splitting traffic so people do not see different variations at the same time.

“Running mutually exclusive tests will eliminate the above validity threats and will allow us to accurately determine which variations truly work best together,” Bartlinski said.

There is a downside though. It will slow down testing since an adequate sample size is needed for each test. If you don’t have a lot of traffic, it may end up taking the same amount of time as running tests one after another.

What’s the big idea?

Another important factor to consider is that the results from grouping the tests should lead to a new understanding of the customer — or what’s the point of running the test?

Bartlinski explains, “Grouping tests makes sense if tests measure the same goal (e.g., reservations), they’re in the same flow (e.g., same page/funnel), and you plan to run them for the same duration.”

The messaging should be parallel as well so you get a lesson. Pointing a treatment ad that focuses on cost to a treatment landing page that focuses on luxury, and then a treatment ad that focuses on luxury pointing to an ad that focuses on cost will not teach you much about your customer’s motivations.

If you’re running multiple tests on different parts of the funnel and aligning them, you should think of each flow as a test of a certain assumption about the customer as part of your overall hypothesis.

It is similar to a radical redesign. Much like testing multiple steps of the funnel can cause an interaction effect, testing multiple elements on a single landing page or in a single email can cause an attribution issue. Which change caused the result we see?

Bartlinski provides this example, “On the same landing page, we run a test where both the call-to-action (CTA) and the headline have been changed in the treatment. The treatment wins, but is it because of the CTA change or the headline? It is possible that the increase comes exclusively from the headline, while the new CTA is actually harming the clickthrough rate. If we tested the headline in isolation, we would be able to determine whether the combination of the new headline and old CTA actually has the best clickthrough, and we are potentially missing out on an even bigger increase.”

While running single-factorial A/B tests is the best way to isolate variables and determine with certainty which change caused a result, if you’re testing at the speed of business you don’t have that luxury. You need results and you need them now!

However, if you align several changes in a single treatment around a common theme that represents something you’re trying to learn about the customer (aka radical redesign), you can get a lift while still attaining a customer discovery. And then, in follow-up single-factorial A/B tests, narrow down which variables had the biggest impact on the customer.

Another cause of attribution effect is running multiple tests on different parts of a landing page because you assume they don’t interact. Perhaps, you run a test on two different ways to display locations on a map in the upper left corner of the page. Then a few days later, while that test is still running, you launch a second test on the same page but in the lower right corner on how star ratings are displayed in the results.

You could assume these two changes won’t have an effect on each other. However, the variables haven’t been isolated from the tests, and they might influence each other. Again, small changes can have big effects. The speed of your testing might necessitate testing like this; just know the risk involved in terms of skewed results.

To avoid that risk, you could run multivariate tests or mutually exclusive tests which would essentially match each combination of multiple variables together into a separate treatment. Again, the “cost” would be that it would take longer for the test to reach a statistically significant sample size since the traffic is split among more treatments.

Test strategically

The big takeaway here is — you can’t simply trust a split testing tool to give you accurate results. And it’s not necessarily the tool’s fault. It’s yours. The tool can’t possibly know ways you are threatening the validity of your results outside that individual split test.

If you take a hypothesis-driven approach to your testing, you can test fast AND smart, getting a result that accurately reflects the real-world situation while discovering more about your customer.

You might also like:

Online Testing certification course — Learn a proven methodology for executing effective and valid experiments

Optimization Testing Tested: Validity threats beyond sample size

Validity Threats: 3 tips for online testing during a promotion (if you can’t avoid it)

B2B Email Testing: Validity threats cause Ferguson to miss out on lift from Black Friday test

Validity Threats: How we could have missed a 31% increase in conversions

The post Conversion Optimization Testing: Validity threats from running multiple tests at the same time appeared first on MarketingExperiments.

Click Here For Original Source Of The Article

Ads by WOW Trk

About Daniel Rodgers

A lot of news that you will not see in the paper. A lot of technology that is coming out that will not see in the paper.

Check Also

The Latest Evolution of Facebook: The Marketing Low-Down on 5 Recent Changes

Latest Facebook Updates 2018

Latest Facebook Updates 2018

From the Russian ad debacle to the Cambridge Analytica scandal to suffering a major stock price blow just a few short weeks ago, Facebook has been making less-than-stellar headlines the past few months. Users are concerned about their privacy and how their data is being used—and brands and marketers are wondering whether the platform will continue to be a viable advertising and engagement platform.

In an attempt to rebuild trust, ensure better data protection and transparency, the social network is doubling down, again, on their commitment to improving the user experience and creating a fun, respectful community.

As all marketers will remember, the quest to improve user experience started way back in 2015, with the announcement it would be making refinements to its News Feed to strike a better balance between friends, public figures, publishers, businesses, and community organizations. That continued in the summer 2016 with more updates favoring friends and family content—and was still on the move when the first scandal broke later that year.

Of course, these changes didn’t do marketers and brands any favors in the organic reach department. Organic reach had already been declining, and these moves have nearly eliminated its potential. And now, more changes have arrived, presenting new challenges as well as some opportunities.

Below we share the low-down on five of such recent or rolling out changes, what they mean for social media marketers, and some potential next steps to take.

1. A New News Feed

Once again, the News Feed is getting a facelift—a big one. While Mark Zuckerberg announced back in January 2018 that changes would be rolling out throughout the year, a “major update” was announced in April, which Director of Product Management, Mark Hull, details in the video below:

Essentially, meaningful person-to-person interaction is what will carry the News Feed ranking weight, and person-to-page interactions will continue to be second tier. Oh, and Facebook expects people to spend less time on the platform.

This sounds pretty scary for marketers. Most have adapted their strategies to zero-in on fostering engaging discussion, as well as throwing spend behind Facebook’s ad platform (which is also changing and we’ll get to that later).

But before you eliminate Facebook from your marketing mix, there are a few opportunities to consider:

Working with influencers: With Facebook continuing to elevate content from individuals, there may be no better time to start building relationships with industry influencers and thought leaders that you can collaborate with on content.

Read: Death of Organic Reach = New Opportunities for Influencer Marketing

Facebook Groups: As my colleague, Nick Nelson, reported not too long ago, while groups have long been available as a feature on Facebook, the brand-driven “Facebook Groups for Pages” were just rolled out last year. And some brands are seeing traction with them, but this isn’t something you leap into without being thoughtful.

Read: The Question on Many Marketers’ Minds: Should My Brand Start a Facebook Group?

Facebook Stories: Very recently, Facebook insiders asserted that Facebook Stories may very well be the future of connection on Facebook. Once again, as Nick Nelson pointed out: “Facebook Stories are intriguing because they offer a real chance to capture part of a user’s attention — maybe even more than the minimum amount.” And early adopters may secure an advantage.

Read: The Future of Connection on Facebook: How Stories May Change the Marketing Game

2. Stricter Ad Targeting

As of late, most of Facebook’s critiques are a result of their advertising products and practices. By increasing the targeting capabilities of their advertising products, Facebook arguably put users’ privacy at risk. To help correct that perception and protect user privacy, Facebook is making several changes to their advertising platform.

One such update was released in early July, requiring advertisers to state where they acquired people’s information for their custom audiences. Instead of simply uploading a list of emails you want to target as a custom audience, Facebook wants advertisers to take extra steps to ensure those emails came from a reputable source and that the audience has consented to those ads. With this change, Facebook hopes to improve transparency with users about why they see ads from certain brands and how they received their information.

New Facebook Ad Disclaimer

(Credit: Facebook)

In addition, Facebook has also disabled their Partner Categories product, which provided targeting capabilities from third-party data providers to advertisers. This limits advertiser insight into user behavior outside of Facebook (e.g. purchasing activity), making ads appear more natural to users and less “big brother.”

From our perspective, this is good news. Consumers are increasingly wary of marketing and advertising messages and this move can help strengthen the credibility and relevance of your ads and brand. Of course, this is all assuming you can and do confirm your custom audience lists and sources fall within the new guides.

So, if you haven’t already, take the steps to review your custom audience lists and their sources. You need to be able to state if you received the information directly from your audience, a partner, or a combination. If you used a data provider like an advertising or marketing agency, double check that they’ve acquired the data honestly and they agree to Facebook’s Terms of Use.

3. New Data History Tools

Now more than ever, people want control over their data. They want to know what information is stored, who has access to it, and they want to be able to delete it.

Recognizing this need, Facebook announced a new feature called Clear History that will be released “soon.”With this new feature, Facebook is giving users the ability to see the websites and apps that store information with Facebook, and delete that information from their account.

However, Facebook will still retain aggregated analytics, but no personally identifying information will be contained.

“We’ll still provide apps and websites with aggregated analytics – for example, we can build reports when we’re sent this information so we can tell developers if their apps are more popular with men or women in a certain age group,” Facebook says. “We can do this without storing the information in a way that’s associated with your account, and as always, we don’t tell advertisers who you are.

What exactly does this mean for marketers? If you use Facebook plugins on your website (think Facebook Pixel or “Like” buttons on websites), your audience can now see and delete the information that the plugin collects; meaning it won’t be connected to their profile any longer.

Obviously, if users take advantage of this when it rolls out, clearing their history could be problematic for marketers, and maybe even users. For marketers, it will be incredibly difficult to target these folks with ads. For users, that could mean an uptick in irrelevant ads for a time.

However, the eventual upside for advertisers could be the “re-learning” that needs to happen after a history cleanse, which can lead to a more relevant and accurate look of who your audience is.

4. Poor Customer Feedback = Ban

A poor customer experience really sours your impression of a brand. An to ensure that ads on Facebook lead to positive shopping experiences, and not negative ones, Facebook will now ban brands that have low customer satisfaction ratings from advertising on their platform.

For example, if users give you too many frowning faces shown in the rating system below, Facebook will reject your ads.

Facebook Negative Review Example

(Credit: Facebook)

Before you become too concerned, however, Facebook says it will: “Share feedback directly with businesses that receive high volumes of negative feedback and will give them a chance to improve before taking further action. If feedback does not improve over time, we will reduce the amount of ads that particular business can run.”

So, even if you receive negative ratings, you will have an opportunity to make improvements to prevent being banned.

The opportunity here is obvious. The better customer experience you provide, the more successful your marketing efforts will be. For Facebook ads, this is rooted in focusing on clarity and honesty within your ads.

At a minimum, Facebook suggests taking steps to ensure your ads aren’t misleading. In addition, use images or videos to make it very clear what you are selling and what you are selling it for. It’s also a good idea to set clear expectations for how users will receive your product or service. By setting clear guidelines, you’re more likely to meet customer expectations, leading to more positive user ratings. For more insight, read Facebook’s tips on how to improve customer feedback.

5. All of Your Ads, Exposed

Knowing the organization behind an ad is important. Otherwise, users might not trust the content of the ad. So, to increase ad and page transparency, Facebook now allows users to see all of the ads any page is running within their partner network.

This includes ads on Instagram, Messenger, and the rest of the Facebook partner network. In addition, even if pages aren’t advertising, Facebook will provide more information about a page including name history and the date the page was created. Together, these changes aim to give users more information about an organization so they can decide if an ad is credible or not.

Facebook Ad Transparency

(Credit: Facebook)

For marketers, this change isn’t a bad thing — it may even mean more eyeballs on your advertising content. However, marketers should still be thoughtful about how their ads will be perceived by individuals outside of their target audience.

For example, even if an ad isn’t specifically designed for or served to them, users can still review (and report) your ads. As a result, you need to take extra steps to make sure your ads are consistent, clear, and friendly for all.

The Life Force of Facebook

Long before the scandals and latest privacy concerns, Facebook has been rooted in evolution. Just think what the platform started as and has become. So, while the recent and coming changes seem pretty fierce, I think it’s safe to say we were already on that trajectory. And there’s probably more to come.

Since the major changes that impacted advertisers and brands began rolling out in 2015, Facebook has maintained their actions are all in the interest of creating a better user experience. And at the end of the day, that’s what the goal of any marketer is in their quest to nurture their audience and aid them on their customer journey.

So, it’s OK if you’re a panicked, disheartened, or simply irritated. But, at this point, the platform still holds marketing opportunity, marketers just need to adapt their social media marketing strategies, try new features and avenues, and work hard to ensure they’re part of providing that great user experience.

Looking for more social media marketing news, tips, and insights? Check out all of our recent social media-related blog posts.

The post The Latest Evolution of Facebook: The Marketing Low-Down on 5 Recent Changes appeared first on Online Marketing Blog - TopRank®.

Leave a Reply

Your email address will not be published. Required fields are marked *

css.php