The ethics of experimentation


The learning curve for figuring out how people will behave on your website is a steep one. It’s hard to predict their emotions. It’s hard to see how they would interact with each other. It’s even harder to figure out why user emotion is so contagious on the Internet.

Companies are probably all too familiar with the frustration this situation breeds. Most have felt the sting of their very best ideas being ignored, their ads not being clicked, and their product launches flopping.

While some companies might try to comfort themselves with the notion that everyone is facing the same problems online, major companies are conducting tests to see if they could affect users’ behavior, emotions, and the way they interact. They are doing it in a multitude of ways, from simple A/B testing where they change colors on a web page to see if it changes the rate at which people buy to showing different information to different audiences to gauge their reactions.

Recent User Testing Sparks Debate


Facebook confessed to a test that experimented with news feeds of 700,000 users in 2012 to see how people responded after reading negative or positive stories, and then published a scientific paper on it.

Then OkCupid, a dating site, made a similar confession of conducting an experiment that tried showing two users as a good match when they were actually a mismatch to see how users reacted. OkCupid founder Christian Rudder stated in a blog post: “When we tell people they are a good match, they act as if they are, even when they should be wrong for each other.”

The impact of these two experiments was small, but they sparked debate about the ethics of testing and experimentation on users. Many users were angered by the lack of permission or awareness about these experiments. And they were especially angered by the fact that their emotions (in the case of Facebook) and dating lives (in the case of OkCupid) were toyed with.

Consent or Pre-Approval Wanted by Public


Facebook didn’t submit any proposal to the Institutional Review Board (an independent ethics committee that requires scientific experiments to abide by consent and safety standards) for pre-approval of the study, according to Forbes. Instead, the social giant considered it another A/B test, which most tech companies and startups engage in on a regular basis.

Facebook technically does ask people for consent in its Data Use Policy agreement when people sign up, specifically saying that they may use the information they receive about you for data analysis, testing, research and service improvement. Many consider it to be a weak form of consent, as they don’t know the intent or the risks of the experiment, and believe that Facebook should provide an option to opt out from these experiments.

James Grimmelmann, Professor of Technology and the Law at the University of Maryland, believes Facebook may have crossed the line. He told Slate that exposing users to something that causes physiological status changes is experimentation is the kind of thing that requires user consent.

On the other hand, Dr. Adam Irish, Mellon Postdoctoral Teaching Fellow in Political Science and International Relations at Wheaton College, said that anyone posting on social networks should be aware that online behavior is traceable. If people will use free services online, they should be aware that their data is likely being collected, used and distributed, till it is determined what is ethical. Facebook also made an apology stating the test was poorly communicated to the users.

Unlike Facebook, OkCupid didn’t issue an apology. They simply went on falsifying their results and mismatching users. Though the company’s founder said that they simply did what other companies were already doing, conducting experiments without users consent can quickly erode trust, according to an article on BBC, especially when it involves presenting users with wrong information, like in OkCupid’s case. They simply put the users trust at risk.

The Impact on Conversion Rate Optimization


So how does this affect businesses that implement A/B and multivariate testing to optimize conversion rates on their websites? Washington Post’s Brian Fung asked: “If you’re lying to your users in an attempt to improve your service, what’s the line between A/B testing and fraud?”

When you’re conducting testing to create a better product or improve your service, there should be a clear ethical stance on testing for your company and how the user data will be handled. A New York Times article says these experiments are important, but they often have ethical issues associated with them.

If you are changing button colors or modifying text in a way that has the same meaning, but is presented in a different way, then your experiments are different from those in the examples above. You are not harming your potential customers with A/B and multivariate testing, so long as you are not conducting experiments that lead visitors to believe something false about any aspect of your products and services or themselves.

Nonetheless, as a business, you should offer a disclosure on your website that lets visitors know that their data is being tracked anonymously. You may even want to follow the example of sites like Pinterest that offer the strict “legalese” terms and conditions with a snippet simplifying each section. This way, your visitors can feel confident about how their data is being used, and how their actions are being measured.

Transparency in this manner should have little affect on your ability to optimize conversions, but it will have a major affect on the level of trust your visitors have when on your website. That kind of trust can be the difference between people buying from your versus buying from your competitors.

What are your thoughts?

Join in the discussion below by leaving a comment about your thoughts on the issue.

  • Dominic Hurst

    Great post, and certainly opens up the debate. We (those setting up the test) may think the change is minimal, but changing a word can lead to a false proposition. It’s a very thin line, but shows that a lot of work and thought needs to go into an a/b test before deploying the test.

  • Joanna Wiebe

    I faced this problem when I was doing my research project for my MA. I wanted to run an A/B test on an active, popular ecommerce site with the goal of determining whether people would buy more X if it was positioned in isolation or alongside a product that would facilitate decision-making using the contrast principle. The Ethics Board at my university wouldn’t have it. It took months of discussions with them to get to the point where they approved the A/B test, knowing that consumers would not be told they were participants in an experiment.

    If we know how people make decisions – the basic and advanced ways to nudge them – and we design and test an experience to nudge them to buy a product they might otherwise not have purchased, is that unethical? They’re on your site… they’re browsing for shoes… they see strategically architected catalogue pages that manipulate them to choose the pair of shoes we most want them to buy – is that unethical? They might not have purchased at all, had we simply listed the shoes and their descriptions in massive grids. Did we manipulate them? Is that okay? Haven’t we been doing that since the dawn of commerce and, in particular, the advent of modern advertising?

  • It’s similar to the EU cookie messaging issue really. My take? When you have to tell your visitors that you are tracking them, tell them what’s in it for them and the benefits of it! i.e. that it will help greatly personalize/customize the website so that it shows them what they want/need much better, and quicker. Just like you should also show visitors the benefits of using your site in general, particularly in comparison to your competitor websites.

    Rich Page: Website Optimizer