Mashable made a fresh start of the new year by launching a redesign. The intention of this new design was to put more focus on the stories, removing clutter, and dividing the content into sections (Home, Social Media,  Mobile, Web Video, Entertainment, Business, Tech, and Jobs). In the past week more than 150 people commented on the blog post about the new design. Most reactions on Mashable seem to be positive about the new look and feel: ‘Fresh & clean’, ‘I like the sections’, ‘More professional’, and ‘Clean and Simple’. What are the most important changes in this design iteration and what can we learn from feedback? We asked 60 social-media-savy participants for feedback.

Mashable: What draws attention?

Read this case on UX Booth

A/B testing is a popular method to optimize conversion on a website. Basically you set up two or more variants, measure the differences in conversion rates between these variants, and select a winner based on your test results. The winning variant is the one with the highest conversion rates. A/B testing is a great way to improve your webpages one step a time. Unfortunately implementing tests is not always as simple as it sounds, even if you use for example the nifty interface of Google Website Optimizer. To set up a test, you need to have access to the source code and someone who’s able to adapt it, a live website with visitors or a working prototype, etc.. We’ll show you an example of how Usabilla’s One-Click-Tasks can be used as an interesting and low-budget alternative for A/B testing.

The original Firefox A/B test

The website abtests.com shows examples of A/B tests and their results. To demonstrate the use of Usabilla to quickly measure differences between multiple variants we picked a case submitted by Firefox (original writeup at the Mozilla blog). Firefox tested two variants for the button on their homepage that needs to trigger users to download the latest Firefox version. They used two variants of a call-to-action button: “Try Firefox 3″ and “Download Now – Free”.

Download Firefox - Variant A: 'Try Firefox 3'

Download Firefox - Variant B: 'Download now - Free'

In the original A/B test conducted by Mozilla 9.73% of all visitors (+/- 150.000) visiting variant A (’Try Firefox 3′) downloaded Firefox. Using variant B (’Download Now – Free’) 10.07% of the visitors (+/- 150.000) downloaded Firefox. Variant B has a 3.6% higher conversion than variant A.

The Usabilla test

We’ve set up two similar tests in Usabilla in about 5 minutes, using screenshots of variant A and B.  We created a new test (let’s call it ‘A’) and used a screenshot of the Firefox homepage showing the call-to-action button with the text ‘Try Firefox 3′. We’ve created another test (test ‘B’) using the screenshot of variant B (’Download now – Free’). In both tests we asked participants to perform only one task: ‘Where do you click to download Firefox?’. To compare the performance on Test A and B we focused on ‘time per task’, which is a standard Usabilla feature.

Short introduction for participants

Participants read the task before they start

Participants perform the task by clicking on the screen

We sent about half of the participants to Test A and the other half to Test B. We invited participants via Twitter and asked them at the end of the test to help to spread the test. We used the the ‘Redirect URL’ for in Usabilla to redirect users to Twitter to post a message about this case.

Test results

We exported the test results for both test cases as XML and imported this in a Google Docs spreadsheet. We used the test data to calculate the Geometric mean (preferred type of mean for task times).

Usabilla as an alternative for A/B tests

In this demo case we set up two simple Usabilla cases in about 5 minutes as a quick and dirty alternative for A/B testing. We measured the task performance (time) on the task ‘Where do you click to download Firefox’ for two different variants of the Firefox homepage for a total of 388 participants, split into two different groups. It took participants in test A (’Try Firefox 3′) on average 5.3 seconds to click the download button. Participants in test B (’Download Now – Free’) clicked the download button in 4.8 seconds (almost 10% faster).

We demonstrated that simple comparative Usabilla cases like this one can help you to improve your conversion. The results of quick tests can give you guidance when you want to improve your website, without worrying about development, a release protocol, server access, and other difficulties. Use mockups or (edited) screenshots to test multiple variants in a short time span.

The sign up button or link is an important call-to-action on the homepages of most web services. In a recent demo case Usabilla compared the sign up on the homepages of 10 different web services. Users found the sign up button on the Twitter homepage in 1.8 seconds. Animoto was a good runner up with 2.3 seconds. On average it took participants 3.5 seconds to find a way to sign up for these web services.

The differences between the performance of these websites on this important task are big. But what makes Twitter homepage stand out in this test? Why do the sign up buttons at Animoto, Vimeo and MyNameisE catch attention faster than those of Wakoopa, Basecamp, and PayPal? We would love to hear your opinion about these test results.

Results

Twitter – 1.8 seconds

Animoto – 2.3 seconds

Vimeo – 2.9 seconds

MyNameisE – 3 seconds

Usabilla – 3 seconds

Facebook – 3 seconds

Foursquare – 3.3 seconds

Wakoopa – 4.3 seconds

Basecamp – 5.3 seconds

PayPal – 6 seconds

Test data

Tested with Usabilla

This usability test was conducted with the online usability tool Usabilla. This online usability tool allows you to collect direct and valuable feedback from your users in any stage of the design process. Users share their opinion or perform tasks by adding points and notes directly on the webpage, mockup, wireframe, screenshot, or image. Sign up for a free account to try it yourself.

The international travel site Expedia (Alexa Rank 816) gets defeated by its competitors Hotwire, Priceline, and Travelocity on basic usability tasks. Expedia performed the worst in a usability showdown between the four major international travel sites. A total of 148 people participated in this usability test and tried to perform three basic tasks on one of the four websites.

Alexa rank for Expedia, Hotwire, Travelocity, and Priceline

Book a room in Amsterdam

The 148 participants tried to book a hotel room in Amsterdam in a certain price range on one of the four big travel websites. They had to find a way to search for a hotel room (task 1), limit their search on price (task 2), and book a room in the correct price range (task 3). The task performance was measured by the success rate and the time per task.

Expedia, Hotwire, Travelocity, and Priceline

Task 1: Where do you click to search a hotel?

The first task was to find a way to search for a hotel room on the homepage. Clicks on the right search fields or on one of the buttons leading to hotel sections have been measured as successful.

Sample results task 1 (Expedia)

134 participants out of 148 (91%) successfully performed this task. All four travel sites performed quite good on this first task. The diagram below shows the percentage of task success on task 1 for each of the four travel sites.

Another interesting metric is the time users need to answer each task. The following diagram displays the average time for task 1 (in seconds). On average it took users about 10 seconds to find a way to book a hotel room on the homepage of Travelocity. The homepage of Hotwire seems to be well organized for : it took users less than 7 seconds to find a way to search for a hotel.

Task 2: Where do you click to limit your search on price?

The participants were looking for a hotel in a price range of 80 to 120 dollars. The participants have been asked to limit the search results on price.

Sample results task 2 (Priceline)

The average success rate of this task was 76 percent. Only 60 percent of the participants found a way to sort the search results on price at Expedia. The sorting features on Hotwire and Travelocity were found by more than 80 percent of the participants.

It took participants only a little bit more than 7 seconds to find a way to sort their search results on price at Travelocity. Expedia users needed more than 20 seconds to answer this task. Hotwire and Priceline users needed on average around 13 to 14 seconds for this task.

Task 3: Where do you click to book a hotel?

The final task for this test was to book a hotel in the right price range from the page with search results. Only 32 out of 148 participants clicked on a link leading to a booking page of hotel in the right price range. Many participants did not select a hotel in the right price range or clicked on an element that didn’t link to a booking page. Priceline definitely outperformed it’s competitors on this task with a success rate of almost 70 percent.

Participants searching for a way to book a room at Expedia answered the task in about 13 seconds. The success rate on task 3 for Priceline was way better than its competitors and participants answered in less than 6 seconds.

Overall performance

Expedia gets beaten by its smaller competitors in this usability test. Priceline has the highest success rates and Hotwire seems to be the runner up. The differences in the total time for all tasks are small for Priceline, Hotwire, and Travelocity. Users need more time to perform a basic task (booking a hotel in Amsterdam) at Expedia.

Tested with Usabilla

This usability test was conducted with the online usability tool Usabilla. This online usability tool allows you to collect direct and valuable feedback from your users. You can use these lean and mean usability tests in any stage of the design process to collect feedback on a webpage, mockup, wireframe, or image. You create an online test in a few minutes, using any image as source. Users share their opinion or perform tasks by adding points and notes directly on the screenshot, mockup or image. Sign up for a free account to try it yourself.

Have you ever booked a hotel at Expedia, Hotwire, Priceline, TraveloCity or another big travel site? We’re really curious if you found what you were looking for. Let’s compare the usability of these four travel sites in this usability showdown.

In this showdown we’re going to compare the usability of Expedia, Hotwire, Priceline and Travelocity on a basic task: book an affordable hotel room in the beautiful city centre of Amsterdam for a trip in November.

We need your help. Perform three simple tasks in about 2 minutes: test a travel site. We will analyze the performance on three simple tasks and share the results with the world.

Do you want to create your own Usabilla test in a few minutes? Sign up for a free account!
Follow us on Twitter for updates on the results of this test and other interesting cases.