Home / Internet Marketing News / Mobile A/B Testing: Quality assurance checklist

Mobile A/B Testing: Quality assurance checklist

Real-world behavioral tests are an effective way to better understand your customers and optimize your conversion rates. But for this testing to be effective, you must make sure it is accurately measuring customer behavior.

One reason these A/B split tests fail to give a correct representation of customer behavior is because of validity threats. This series of checklists is designed to help you overcome Instrumentation Effect. It is based on actual processes used by MECLABS Institute’s designers, developers and analysts when conducting our research services to help companies improve marketing performance.

MECLABS defines Instrumentation Effect as “the effect on the test variable caused by a variable external to an experiment, which is associated with a change in the measurement instrument.” In other words, the results you see do not come from the change you made (say, a different headline or layout), but rather, because some of your technology has affected the results (slowed load time, miscounted analytics, etc.)

Avoiding Instrumentation Effect is even more challenging for any test that will have traffic from mobile devices (which today is almost every test). So, to help you avoid the Instrumentation Effect validity threat, we’re providing the following QA checklist. This is not meant for you to follow verbatim, but to serve as a good jumping-off point to make sure your mobile tests are technically sound. For example, other browsers than the ones listed here may be more important for your site’s mobile functionality. Maybe your landing page doesn’t have a form, or you may use different testing tools, etc.

Of course, effective mobile tests require much more than thorough QA — you also must know what to test to improve results. If you’re looking for ideas for your tests that include mobile traffic, you can register for the free Mobile Conversion micro course from MECLABS Institute based on 25 years of conversion optimization research (with increasing emphases on mobile traffic in the last half decade or so).

There’s a lot of information here, and different people will want to save this checklist in different ways. You can scroll through the article you’re on to see the key steps of the checklist. Or use the form on this page to download a PDF of the checklist.


Roles Defined

The following checklists are broken out by teams serving specific roles in the overall mobile development and A/B testing process. The checklists are designed to help cross-functional teams, with the benefit being that multiple people in multiple roles bring their own viewpoint and expertise to the project and evaluate whether the mobile landing page and A/B testing are functioning properly before launch and once it is live.

For this reason, if you have people serving multiple roles (or you’re a solopreneur and do all the work yourself), these checklists may be repetitive for you.

Here is a quick look at each team’s overall function in the mobile landing page testing process, along with the unique value it brings to QA:

Dev Team These are the people who build your software and websites, which could include both front-end development and back-end development. They use web development skills to create websites, landing pages and web applications.

For many companies, quality assurance (QA) would fall in this department as well, with the QA team completing technical and web testing. While a technical QA person is an important member of the team for ensuring you run valid mobile tests, we have included other functional areas in this QA checklist because different viewpoints from different departments will help decrease the likelihood of error. Each department has its own unique expertise and is more likely to notice specific types of errors.

Value in QA: The developers and technological people are most likely to notice any errors in the code or scripts and make sure that the code is compatible with all necessary devices.


Project Team – Depending on the size of the organization, this may be a dedicated project management team, a single IT or business project manager, or a passionate marketing manager keeping track of and pushing to get everything done.

It is the person or team in your organization that coordinates work and manages timelines across multiple teams, ensures project work is progressing as planned and that project objectives are being met.

Value in QA: In addition to making sure the QA doesn’t take the project off track and threaten the launch dates of the mobile landing page test, the project team are the people most likely to notice when business requirements are not being met.


Data Team The data scientist(s), analyst(s) or statistician(s) helped establish the measure of success (KPI – key performance indicator) and will monitor the results for the test. They will segment and gather the data in the analytics platform and assemble the report explaining the test results after they have been analyzed and interpreted.

Value in QA: They are the people most likely to notice any tracking issues from the mobile landing page not reporting events and results correctly to the analytics platform.


Design Team The data scientist(s), analyst(s) or statistician(s) helped establish the measure of success (KPI – key performance indicator) and will monitor the results for the test. They will segment and gather the data in the analytics platform and assemble the report explaining the test results after they have been analyzed and interpreted.

Value in QA: They are the people most likely to notice any tracking issues from the mobile landing page not reporting events and results correctly to the analytics platform.



Pre-launch, both initial QA and on Production where applicable

Visual Inspection and Conformity to Design of Page Details

  • Verify latest copy in place
  • Preliminary checks in a “reference browser” to verify design matches latest comp for desktop/tablet/mobile layouts
  • Use the Pixel Perfect Overlay function in Firefox Developer Tools – The purpose of this tool is to take an image that was provided by the designer and lay it over the website that was produced by the developer. The image is a transparency which you can use to point out any differences or missing elements between the design images and the webpage.
  • Displaying of images – Make sure that all images are displaying, aligned and up to spec with the design.
  • Forms, List and Input Elements (Radio Buttons, Click Boxes) – Radio buttons (Dots and Circles) and Checkboxes (Checks and Boxes) are to be tested thoroughly as they may trigger secondary actions. For example, selecting a “Pay by Mail” radio button will sometimes automatically hide the credit card form.
  • Margins and Borders – Many times, you will notice that a portion of the body or perhaps a customer review or image is surrounded by a border or maybe even the whole page. It is our duty to inspect them so that there are no breaks and that they’re prominent enough for the user to decipher each bordered section.
  • Copy accuracy – Consistency between typography, capitalization, punctuation, quotations, hyphens, dashes, etc. The copy noted in the webpage should match any documents provided pertaining to copy and text unless otherwise noted or verified by the project manager/project sponsor.
  • Font styling (Font Color, Format, Style and Size) – To ensure consistency with design, make sure to apply the basic rules of hierarchy for headers across different text modules such as titles, headers, body paragraphs and legal copies.
  • Link(s) (Color, Underline, Clickable)

Web Page Functionality: Verify all page functionality works as expected (ensure treatment changes didn’t impact page functionality)

  • Top navigation functionality – Top menu, side menu, breadcrumb, anchor(s)
  • Links and redirects are correct
  • Media – Video, images, slideshow, PDF, audio
  • Form input elements – drop down, text fields, check and radio module, fancy/modal box
  • Form validation – Error notification, client-side errors, server-side errors, action upon form completion (submission confirmation), SQL injection
  • Full Page Functionality – Search function, load time, JavaScript errors
  • W3C Validation – CSS Validator (http://jigsaw.w3.org/css-validator/), markup validator (http://validator.w3.org/)
  • Verify split functional per targeting requirements
  • Verify key conversion scenario (e.g., complete a test order, send test email from email system, etc.) – If not already clear, QA lead should verify with project team how test orders should be placed
  • Where possible, visit the page as a user would to ensure targeting parameters are working properly (e.g., use URL from the PPC ad or email, search result, etc.)

Tracking Metrics

  • Verify tracking metrics are firing in browser, and metric names match requirements – Check de-bugger to see firing as expected
  • Verify reporting within the test/analytics tool where possible – Success metrics and click tracking in Adobe Target, Google Content Experiments, Google Analytics, Optimizely, Floodlight analytics, email data collection, etc.

Back End Admin Panel

Notify Project Team and Data Team it is ready for their QA (via email preferably) – indicate what reference browser is. After Project Team initial review, complete full cross browser/ cross device checks using “reference browser” as a guide:

Browser Functionality – Windows

  • Internet Explorer 7 (IE7)
  • IE8
  • IE9
  • IE10
  • IE11
  • Modern Firefox
  • Modern Chrome

Browser Functionality – macOS

  • Modern Safari
  • Modern Chrome
  • Modern Firefox8

Mobile Functionality – Tablet

  • Android
  • Windows
  • iOS

Mobile Functionality – Mobile phone

  • Android
  • Windows
  • iOS

Post-launch, after the test is live to the public:

  • Notify Project Team & Data Team the test is live and ready for post-launch review (via email preferably)
  • Verify split is open to public Verify split functional per targeting requirements
  • Where possible, visit the page as a user would to ensure targeting parameters are working properly (e.g., use URL from the PPC ad or email, search result, etc.)
  • Test invalid credit cards on a production environment

Pre-Launch and Post-Launch QA:

  • Check that copy and design are correct for control and treatments in the “reference browser”:
  • Ensure all added copy/design elements are there and correct
  • Ensure all removed copy/design elements are gone
  • Ensure all changed copy/design elements are correct
  • Ensure control experience is as intended for the test
  • Check page functionality:
  • Ensure all added/changed functionality is working as expected
  • Ensure all standard/business as usual – BAU_ functionality is working as expected:
  • Go through the typical visitor path (even beyond the testing page/ location) and ensure everything functions as expected
  • Make sure links go where supposed to, fields work as expected, data passes as expected from page to page.
  • Check across multiple browser sizes (desktop, tablet, mobile)
  • If site is responsive, scale the browser from full screen down to mobile and check to ensure all the page breaks look correct
  • Where possible, visit the page the way a typical visitor would hit the page (e.g., through PPC Ad, organic search result, specific link/button on site, through email)

Pre-Launch QA Checklist (complete on Staging and Production as applicable):

  • Verify all metrics listed in the experiment design are present in analytics portal
  • Verify all new tracking metrics’ names match metrics’ names from tracking document
  • Verify all metrics are present in control and treatment(s) (where applicable)
  • Verify conversion(s) are present in control and treatment(s) (where possible)
  • Verify any metrics tracked in a secondary analytics portal (where applicable)
  • Immediately communicate any issues that arise to the dev lead and project team
  • Notify dev lead and project team when Data QA is complete (e-mail preferably)

Post-Launch QA / First Data Pull:

  • Ensure all metrics for control and treatment(s) are receiving traffic
  • Ensure traffic levels are in line with the pre-test levels used for test duration estimation
  • Update Test Duration Estimation if necessary
  • Immediately communicate any issues that arise to the project team
  • Notify dev lead and project team when first data pull is complete (e-mail preferably)

Pre-Launch Review:

  • Verify intended desktop functionality (if applicable)
  • Accordions
  • Error states
  • Fixed Elements (nav, growler, etc.)
  • Form fields
  • Hover states – desktop only
  • Links
  • Modals
  • Sliders
  • Verify intended tablet functionality (if applicable)
  • Accordions
  • Error states
  • Fixed Elements (nav, growler, etc.)
  • Form fields
  • Gestures – touch device only
  • Links
  • Modals
  • Responsive navigation
  • Sliders
  • Verify intended mobile functionality (if applicable)
  • Accordions
  • Error states
  • Fixed Elements (nav, growler, etc.)
  • Form fields
  • Gestures – touch device only
  • Links
  • Modals
  • Responsive navigation
  • Sliders
  • Verify layout, spacing and flow of elements
  • Padding/Margin
  • “In-between” breakpoint layouts (as these are not visible in the comps)
  • Any “of note” screen sizes that may affect test goals (For example: small laptop 1366×768 pixels, 620px of height visibility)
  • Verify imagery accuracy, sizing and placement
  • Images (Usually slices Design provided to Dev)
  • Icons (Could be image, svg or font)
  • Verify Typography
  • Color
  • Font-size
  • Font-weight
  • Font-family
  • Line-height

Qualifying questions, if discrepancies are found:

  • Is there an extremely strict adherence to brand standards?
  • Does it impact the hierarchy of the page information?
  • Does it appear broken/less credible?
  • Immediately communicate any issues that arise to the dev lead and project team
  • Notify dev lead and project team when data QA is complete (e-mail preferably)

To download a free PDF of this checklist, simply complete the below form.


Increase Your Mobile Conversion Rates: New micro course 

Hopefully, this Mobile QA Checklist helps your team successfully launch tests that have mobile traffic. But you still may be left with the question — what should I test to increase conversion?

MECLABS Institute has created five micro classes (each under 12 minutes) based on 25 years of research to help you maximize the impact of your messages in a mobile environment.

In the complimentary Mobile Conversion Micro Course, you will learn:

  • The 4 most important elements to consider when optimizing mobile messaging
  • How a large telecom company increased subscriptions in a mobile cart by 16%
  • How the same change in desktop and mobile environments had opposing effects on conversion

Register Now for Free

The post Mobile A/B Testing: Quality assurance checklist appeared first on MarketingExperiments.

Click Here For Original Source Of The Article

Ads by WOW TRK

About Daniel Rodgers

A lot of news that you will not see in the paper. A lot of technology that is coming out that will not see in the paper.

Check Also

5 SEO Mistakes Killing Your Content Performance and a Fix for Each

Common SEO Mistakes

Common SEO Mistakes

Even the most seasoned content marketers make mistakes. In the world of SEO-driven content, with constant algorithm tweaks and changing search patterns, it’s nearly unavoidable. However, those same mistakes can often lead to discoveries that enable even better content performance.

The key is being able to recognize those easy-to-fix SEO mistakes and address them. As a result, your content will become an optimized, integrated network of metaphorical highways, leading searchers to best-answer content in a strategic and purposeful way.

So, what are the most common SEO mistakes, and how can they be addressed? Below, I’ve singled out the ‘usual suspects’ along with guidance on how to fix them while setting yourself up for long-term SEO success.

SEO Mistake #1 - Choosing Target Keywords Based on Volume vs. Relevance

How Keywords Affect Content Marketers: Great content isn't great unless people see it. But when content marketers overemphasize high-volume keywords, they miss out on meaningful engagement.

It’s tempting to plug into your keyword research tool of choice and select keywords with the highest search volume as your focuses for new content. But if the content you’re creating doesn’t match the search intent for that high-volume keyword, it’s unlikely to perform to your expectations.

The Fix: Google it! All jokes aside, evaluating the first ten search results for your target keywords can help you understand what searchers are trying to find, and what supporting content you should provide to truly be the best answer for that query.

While you’re analyzing those top results, pay attention to key factors that will shape your content creation and promotion strategy:

  1. What type of information is NOT included in top content, but is topically related? This can help you inform how you differentiate your content.
  2. What’s the content demand for that keyword? For example, are mostly top of funnel blog posts ranking, or are you seeing mostly product or service pages?
  3. How many backlinks and referring domains are pointing to the top search results? This can help you understand how competitive the first page of results is, and whether or not ongoing link building should be part of your content promotion strategy.
  4. How long is the top-ranking content for that keyword? This will help you determine ideal content length for your own post.

SEO Mistake #2 - Targeting the Same Keyword with Multiple Pages or Posts

How Same-Topic Targeting Affects Content Marketers: Pressure to create comprehensive content on a topic can actually result in dilution within search.

The conventional wisdom that more is better doesn’t apply universally — especially when it comes to SEO-driven content. Creating multiple pieces of content that target the exact same keyword is a surefire way to stand in your own way of success. There’s enough competition out there for B2B marketers without having to compete with your own content.

For example, a B2B technology company that wants to rank for B2B software consulting should optimize their service page for that term based on what is currently being served in search results. But, if they also create a series of blogs or resources that are targeting that specific term, search engine bots will be confused about which page is the best answer for that query. This could result in none of the content appearing in the top 10 results, in favor of competing sites with a more clear ‘answer’ to that query.

The Fix: Determine which of your pages or posts is the best answer for that particular query by analyzing ranking and analytics data. Which post or page sees the greatest amount of engaged organic traffic for your target keyword, and most closely matches the associated search intent?

Once you’ve determined your target page, it’s time to evaluate the remaining content targeting that keyword. Look for opportunities to:

  1. Remove or prune low-value or outdated content. Is there a blog post full of stats from 2009 that’s hindering your priority page’s chances of ranking? It might be time to consider removing that post and implementing the proper redirects.
  2. Optimize existing content for related, but different, keyword targets. For example, if you have a priority post for Chocolate Chip Cookies, and another post that more closely relates to ‘Crunchy Chocolate Chip Cookies’, consider optimizing that post for the latter and implementing internal links back to your priority cookies post.
  3. Combine closely related content. For example, if you have several blog posts around your targeted keyword(s), consider combining those posts into a longer, more robust piece of content.

SEO Mistake #3 - Ignoring Internal Link Structure

How Internal Linking Affects Content Marketers: Links are like electricity on the web, lighting up content for people and search engines alike.

Content is discovered by links. Your site’s internal linking structure tells bots (and users) which pages are most important, and which pages are most relevant to specific keywords. If you link to several pages from the same anchor text, for example, there will be some confusion about which page is truly ‘about’ that topic. Other times, you could have pages or posts on your site that are orphaned, with no internal links directing users or bots their way. This can confuse your site users, search engine bots, and even your own team. Confusion is not a ranking factor!

The Fix: Make sure you develop and continue to update your site’s keyword map. It can be as simple as a spreadsheet that lists your page’s URL and associated target keyword(s). This keyword map will help you determine what anchor text should be used to link to your target pages.

Next, conduct a site audit to determine:

  1. If there are orphaned pages that need internal links
  2. If you are linking to multiple pages with the same keyword-rich anchor text
  3. Where there are opportunities to create additional supporting content
  4. Where you might have opportunities to reduce and prune existing supporting content

Next, you’re going to want to crawl your site to find any orphaned pages. Then, map those into your overall keyword strategy and implement internal links.

SEO Mistake #4 - Ignoring Data from Other Digital Tactics

How Marketing Data Affects Content Marketers: Inspiration often drives ideation for many content marketers, but data drives optimization for ideal content performance. Marketing performance data can provide both.

Any data you can collect about how your audience engages with your content has the potential to be an SEO gold mine. For example, analyzing the keywords from your paid search campaigns can give you insight into which keywords are your best converters, and what content best suits searchers for those terms. Social posts that get the greatest amount of engagement can tell you which topics your audience is most interested in. Ignoring data from your other marketing and sales channels means missing out on an opportunity to better engage your prospects.

The Fix: Meet with different teams or departments to find out what kind of content performs best on their channels. Look at the data each platform or channel provides and compare that with your site analytics data for a full picture. And, be sure to share your channel performance data with the rest of your marketing team. The more information available related to content and marketing performance, the better equipped you are to optimize.

SEO Mistake #5 - Giving Up

How Persistence Affects Content Marketers: Content performance in search is a long game and persistence is essential for success.

SEO is a marathon, not a sprint. Sometimes a lack of results can feel demoralizing, but giving up is simply not an option. You wouldn’t stop building your house just because the nearest lumber yard ran out of wood, right? You’d find another lumber yard and keep plugging along.

The Fix: Take a step back. Re-evaluate the search landscape, your competitor’s organic presence, and your site’s overall health. Being able to remove yourself from the frustration can help you find opportunities you may have missed and additional whitespace to tackle.

Next, seek out advice from other SEOs. Ask questions on social media, in specific groups or forums, or send a question to your favorite SEO blog. If budget permits, enlist the help of a consultant or SEO agency that can help you break through your roadblocks.

Finally, we have two big SEO bummers that are tougher to fix, but absolutely necessary to address.

Bonus SEO Mistake: Migrating Your Site with No SEO Plan

How Migrating Without a Plan Affects Content Marketers: A bad migration can effectively undo your hard work, reducing content visibility and creating more user friction.

If you listen closely, you can hear the sound of SEOs cringing around the globe. A botched site migration can wreak havoc on your organic positioning and torpedo your results. It can take months, even years to recuperate organic visibility to pre-migration levels.

The Fix: Always, always consult your in-house SEO team or SEO agency when you’re considering a website migration. Before you move forward, it’s imperative you have a plan for technical, on-page, and off-page factors.

If you’ve already migrated your site and have experienced a loss of organic traffic and rankings, start with a site audit. Check for the basics, like whether or not your site is being indexed, first. Then start to evaluate technical factors like broken links, crawl errors, and duplicate content.

When in doubt, don’t hesitate to reach out for help. Recovering from a site migration is a challenge for even the best of SEOs, and sometimes those big challenges call for a little teamwork.

Bonus SEO Mistake: Not Optimizing for Mobile

How Not Optimizing for Mobile Affects Content Marketers: Even the greatest content can’t stand up to a bad mobile experience. Users will bounce, reducing engagement and sending negative signals to search engines.

Mobile accounts for about half of web traffic worldwide. Knowing this, in March 2018 Google started migrating sites to mobile-first indexing. Providing a seamless mobile experience is no longer optional, especially when you’re living in the wild world of search.

Sites that didn’t properly prepare for this can and will likely see some declines in organic search traffic and rankings as a result. And, as more sites follow mobile best practices, more users will notice and become frustrated by poor mobile experiences. This leads to declines in other pivotal ranking factors like on-page engagement. In short, if not properly addressed, a poor mobile experience can wreak havoc on your search visibility.

The Fix: The first thing to do is to conduct a mobile audit on your site. Understanding your site’s mobile performance is step one toward making improvements. Look for things like:

  1. Mobile site speed. A couple great tools for this are Google Page Speed Insights and Pingdom. These tools can tell you where to look for issues like slow-loading code, images that aren’t optimized, and other technical issues.
  2. Mobile experience. Visit your site on your phone. Ask someone who doesn’t use your site regularly to do the same. Record your experience, take notes on where you get stuck and why. Click on everything. Turn your phone into horizontal mode. Try to think of every single way a user could browse your site. And, don’t forget to try a site search on mobile.
  3. Look at mobile analytics. This will tell you key metrics like mobile bounce rate, mobile time on page and pages per session.

These steps will help you build a hypothesis to test against. Is your mobile bounce rate crazy high? Does your site take a long time to load? Is your time on page way out of line with desktop traffic? Then, use A/B testing to root out the discrepancy. Use these same metrics to test if the fix is working. Then, repeat with another element.

So, What Does This All Mean for You?

Ultimately, following SEO best practices as a content marketer can reduce performance-related headaches and set you up for long-term success.

For example, when Innovatech Labs decided it was time to make major changes to their website, they worked with our team at TopRank Marketing to implement a safe website transition strategy, minimizing their risk of reduced content visibility on Google. This assessment involved avoiding many of the big risks mentioned above, including linking, use of data and keyword research which allowed us to act quickly post-migration to combat organic traffic declines. The result? Double- and triple-digit increases in organic traffic (and increased conversions, too!).

A best-answer content strategy focused on creating content with the most relevance to their audience was the ticket to better marketing performance for a martech SaaS company. Working with the team at TopRank Marketing, long-tail and hyper-relevant keywords were researched for a comprehensive content strategy to help the brand content become the best answer for those queries. The “best answer” approach and topics were applied across organic and paid efforts. As a result, the volume of both paid and organic MQLs increased, leading to better content performance and spontaneous proclamations of love from the client’s sales team.

Fixing these big SEO mistakes aren’t only for short-term wins. Our longtime partner Antea Group USA has achieved amazing triple-digit growth over three years by avoiding these big mistakes and implementing an ongoing commitment to SEO-driven, best answer content.

As I mentioned earlier, even the most experienced content marketers can make these common SEO mistakes. But, with the right SEO strategy driven by diligent execution and monitoring of results, you can get back on track. The key is to be intentional about your site’s architecture, as well as the content you create, and to never, ever give up.

Still feeling stuck? Or maybe your team doesn’t have the resources to take on this battle alone? Check out our SEO services, tweet us your thoughts @toprank, or drop me a line in the comments. We are here to help!

The post 5 SEO Mistakes Killing Your Content Performance and a Fix for Each appeared first on Online Marketing Blog - TopRank®.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.