Mobile platform A/B testing – Art or wizardry?

By Alon Shmueli, Mobile Team Leader, Leverate

Developing a successful trading app requires a lot of research and a little bit of wizardry. Everything form the color of a buy and sell button to the type of charts we make available and the steps required to join the system, play a huge role in the way in which we design our mobile trading apps in order to help our brokers convert and retain traders better.

When it comes to an app’s sign-up process, there is no one-size-fits-all and what may work for one business or industry may not work for another, so a little bit of A/B testing comes into play. A/B testing is simply a method of comparing the effectiveness of two versions against each other with a controlled sample of users.

At Leverate, every feature of our apps we design or change has the ultimate goal of increasing conversion and/or retention of traders, so when we were tasked with redesigning the registration process in order to comply with EU regulation, A/B testing was the only way to solve the dilemma we faced: brokers on one hand, want to gather as much valuable information about a trader as possible and regulatory bodies have a say in what kind of data we must gather.

Traders on the other hand, want to disclose as little information about themselves as possible. Finding the right balance between acquiring the right amount of data from traders and avoiding turning them away because we are asking for too much of it is like walking on a tight rope.

In order to obtain real, quantifiable data to determine the correct and most effective sign-up format, the mobile team at Leverate designed two sign up forms: the first one, will include 5 pieces of data (name, e-mail address, phone number, country and password), while the second version will only include 2 pieces of data (password and e-mail address).

We will distribute both versions amongst users of iOS and Android and collect data using Google Analytics in order to determine:

A) whether the 5 item form is filled out as often as the 2 item form

B) If users quit half way through filling out the 5 item form, how many questions the majority of users filled out (this may help us determine if although 5 items may be too many, perhaps we can ask for more than the two we ask on the shorter form).

Often times, companies will do A/B testing with a sample group of users. In order to obtain the most accurate results however, we have decided it is best to use a wider, uncontrolled sample.

The main challenges when doing a full-scale A/B testing are first and foremost, the fact that we need to develop two full GUIs (Graphical User Interface) of the sign-up form for several operating systems (iOS for the latest two versions and while Android has users in versions going as far back as 3 years ago, we develop for the three most relevant

A second challenge we face is sampling in Asia. When doing A/B testing, timing is a concern and making the Android version of the app available in China, where Google is blocked by the Chinese Firewall, is more time-consuming, as is gathering results using alternate methods of Google Analytics.

A/B testing is a wonderful tool for developers to be able to determine the best course of action when implementing new tools or changing existing ones in an app and although the process is challenging and time-consuming, the rewards far outweigh the effort.