Total Followers 132,503
Welcome back to part two of our email UI faceoff – I recently wrote about the usability testing process, and set up a test for comparing the usability of Yahoo! Mail, Hotmail and Gmail.
Well, the results are in and we’ve got some pretty interesting feedback from the test – you can get directly to the test results yourself, or read on for our analysis and verdict.
Over the days we’ve been running the test, we had north of 300 people take it, and we think the numbers are a pretty good sample. Based on our analytics data, I could see that we had participants from 64 different countries, and it took the average user 2:56 to take the test.
So what did we learn from all that? Read on and find out:
Grids: The first three questions used a grid to discover users expected behaviours. When you overlay the grid over the different UIs you can see how the designs fit in with peoples perceptions.
Learned conventions tend to show through in this kind of area, but conventions are often an important standard for designers to follow (as you will see reflected in the results later on)
As you can see from the results, by far and away the majority of people expect to find the log out button on the top right hand side of the page. This is the most common location for a logout button across numerous websites (have a look for yourself), and so something people are very familiar with. If you want to put your logout button in a different location, you should make sure there is a good reason for doing so.
When overlaid, we can see both Gmail and Hotmail follow this convention, but for some reason, Yahoo! Mail locates their logout button in the middle of the page. What’s worse, it doesn’t render properly in the browser I’ve been using for this test – Safari 5.
Yahoo! Mail 0; Hotmail 1; Gmail 1
The results for this are fairly obvious – as you can see from the heatmap, more all less all users clicked in the top left hand corner to compose a new mail – this is the location for all three services we tested, and the most common location across more or less all webmail services.
Yahoo! Mail 1; Hotmail 2; Gmail 2
Some interesting results for this one – there is really quite a spread all over the screen, which shows perhaps there aren’t quite so many learned behaviours for this. There is a concentration towards the top-middle and top left of the screen however. Lets overlay this over the different services and see how they compare:
Each of the services seems to have ever so slightly different locations for their buttons, and the users of each seem to have learnt these. I don’t see that one or the other is ahead in this test.
Personally though, I do like both the Hotmail and Gmail interface for having reply buttons at the bottom of messages, so you can easily reply after you’ve finished reading. What’s more, the Gmail message reply is the box you can see at the bottom of the messsage. Just put your cursor in there and you can begin typing. Pretty straight forward.
Yahoo! Mail 2; Hotmail 3; Gmail 3
This is a bit of a popularity contest, but evidence shows all things being equal people have a better experience with nicer looking interfaces. Stands to reason. You can see the results here.
The numbers were:
The remaining 5% is people either skipping the question or clicked the side of the interface. Regardless, these numbers are pretty defenitive, and Gmail is well ahead.
Yahoo! Mail 2; Hotmail 3; Gmail 4
Great results for this one, fairly overwhelming:
At the present time, Gmail is the only provider that offers threded messages to all its users. Hotmail is currently rolling out threading on a limited basis, but by no means to all users (our test account doesn’t have threading), and Yahoo! Mail offers no threading at all. Results:
Yahoo! Mail 2; Hotmail 3; Gmail 5
Another comprehensive result as you can see from the image below:
The numbers were as follows:
Again, Gmail is the only provider of the three that enables you to reply on the same page – a behaviour prefered by 81% of users tested.
Yahoo! Mail 2; Hotmail 3; Gmail 6
I was slightly surprised by the results of this test – banner ads are evidently more popular than I thought, and although the results were comprehensively in favour of text ads, it seems there is life for banner ads yet. The results were:
Again though, seeing as Hotmail and Yahoo! Mail both use banner ads, and Gmail text ads, it looks like Gmail is the winner here.
Yahoo! Mail 2; Hotmail 3; Gmail 7
I was surprised when testing Yahoo! Mail that they used a landing page when you log into your mail. This got me thinking as to whether or not this was the prefered behaviour of users. The results are:
Yahoo! Mail is the only one of the three that uses landing pages – and obviously this isn’t very popular with users. Score:
Yahoo! Mail 2; Hotmail 4; Gmail 8
For the next series of questions, the success criteria are less than 10 seconds to complete a task – thus giving the user time to read instructions and to ensure a usable interface – and a success rate of over 75% – to make sure most users are completing the task satisfactorily.
As you can see in the results, the success rate here is 75%, but what we also need to pay attention to is the average time to complete this test – in this case, 27.58 seconds.
This shows us that while users can indeed find the settings area, they are taking too long to do in. In this case Yahoo! Mail could try to make the text more prominent or more visible in order to make this area more usable.
Yahoo! Mail 2; Hotmail 4; Gmail 8
These results are a bit of a fail for Gmail – it fails on both fronts. There is only a 69% success rate, and it took the average user just under 14 seconds to complete the task.
If they wanted to make the calendar more visible, they could try moving it’s location, or featuring it in more than one location. The fact that 16% of people clicked on the left hand side indicates that adding a calendar button in that area would be beneficial.
Yahoo! Mail 2; Hotmail 4; Gmail 8
And at last we have a task that meets our criteria. 76% of users (or 87% if you count the ones marking the messages) successfully completed the task with an average time of 9.16 seconds. This could still be improved upon, but isn’t a bad result.
Yahoo! Mail 2; Hotmail 5; Gmail 8
As you can see here, a pretty poor result. It took an average of 17.73 seconds for people to click, and after that, there was still a success rate of just 54%.
Yahoo obviously needs to put a lot more consideration in to the placement of the calendar button. The ammount of people clicking in the upper left hand side tends to suggest that many people look to this area, so perhaps by offering a link there they would increase their success rate.
Yahoo! Mail 2; Hotmail 5; Gmail 8
As you can see from the results, there was an 85% success rate, and an average click time of 6.38 seconds. This is obviously well inside our success criteria, and a strong result for Gmail.
Yahoo! Mail 2; Hotmail 5; Gmail 9
Another good result for Hotmail here – an 80% success rate, and an average click time of 7.54 seconds. I think this makes a lot of sense as this is a very common location for a settings/options link to be.
Yahoo! Mail 2; Hotmail 6; Gmail 9
A much better showing for Yahoo! Mail in this case, an 87% success rate, and a click time of just 4.13 seconds. This really shows the value of following conventions – bear this in mind when designing.
Yahoo! Mail 3; Hotmail 6; Gmail 9
A 91% success rate, and an average click time of 6.01 seconds show that Gmails setting page location is working very well – these are very good numbers.
Yahoo! Mail 3; Hotmail 6; Gmail 10
And another success to round out our tests. 5.5 seconds average click time, and a 78% success rate. Quite a spread on this page though, so it shows people still aren’t quite sure where to look for calendars.
Yahoo! Mail 3; Hotmail 7; Gmail 10
This was a pretty interesting test, and created some interesting results. It shows the value of following conventions and observing user behaviours when developing or reworking your designs. If you’d like to browse through the whole test results, you can go here.
The final scores were: Yahoo! Mail 3; Hotmail 7; Gmail 10.
While Gmail was ultimately ahead in these tests, all three of the services have areas they could improve upon.
Hotmail was also a reasonably strong contender, and with updates which they are currently rolling out, the usability of their service should continue to improve.
Yahoo! Mail was lagging far behing in third place, and really needs to pick up its game if it wants to remain competitive. They are probably relying on people who have been using their service for many years, and I would be surprised if they are still signing up many users.
How does this compare with your actual experience of these different webmail services? Are there any other sites you’d like see compared in the future? Be sure to let us know in the comments!
Get The Only Freelancer crash course you will ever need to read!
Jacob is a usability geek at IntuitionHQ.com - a new, quick, easy to use usability service from Boost New Media. When he's not indulging his passion for the internet, he is probably bloging about China and Chinese learning (having spent three years there) at Sinomatter.com. You can find him on twitter talking about usability @intuitionhq.
Sunday, February 26th, 2012 00:24
Gmail is the best and would win if they took another test. I was using gmail for more than 4 years and i did not have any problems.
Monday, January 2nd, 2012 21:26
Personally I like the Gmail interface because is simple and simple is always good and ergonomic! On the other hand, I also believe that having your own email server would make things easier for you.
Thursday, November 17th, 2011 18:11
Gmail is the best one in my opinion.
With the change of UI, Google are truly making it more easier to use!
More friendly for older generations.
Thursday, November 3rd, 2011 22:22
for me gmail :
is less complicated
takes no time to load
the new look makes it even more great
identifies emails that are important and marks them with a yellow marker
hasnt got any distracting banners; only text ads in one row so it doesnt take all the space
has got 25mb attachment; more than hotmail and yahoo mail
all the important and common stuff is bigger and good on the eye at the top like reply, forward, report spam etc once you select or open an email. if any of u guys havent seen the new 2011 look yet u better before saying anything else against gmail. by the wy i have used hotmail aswell and am still using it but not much and in my opinion it isnt that attractive or less complicated as gmail. yahoo mail is not an option ..:D
Thursday, October 27th, 2011 23:55
Hey, that’s really the best test i’ve ever seen in my life, congratulations! really D:
Monday, September 5th, 2011 04:17
Do as I did, and stop reading this about half way as it starts to become heavily evident that it is biased solely towards Gmail.
Tuesday, September 20th, 2011 10:09
Sobias towards Gmail. Ruins the point of this exercise
Wednesday, August 24th, 2011 20:57
For me, hotmail clearly wins, though i use both hotmail and yahoo, google is just too selfish for my liking.
first, hotmail has the best spam filter. I three years since i have this adress, i virtually got only 5 spam messages, and their senders (the real ones) were added to block list as soon as i deleted the emails, unlike yahoo, where you must keep the emails in spam folder for about a month.
second, hotmail gives you FREE Microsoft office online, and 2GB of storage for virtually everything (I even keep a website in there). this also makes you able to create .docx documents online without having to buy/install/crack/whatever an office installation. This is HUGE in comparison with gmail.
Third, their web pages have the most powerful encryption available around, you can safely use your hotmail account for things such as bank accounts and stuff like that.
Get The Only Freelancer crash course you will ever need!
If not, then it's time to learn how to:
You can trust 1stWebDesigner to help you become a better web designer!
- Jacob Cass | Just Creative
Just enter your name and email below and click Get Updates!
Jim
Friday, September 3rd, 2010 12:23
Nice analyzing..
TingTingBen
Monday, September 6th, 2010 09:35
Very nice !
But aren’t all these results influenced by years of Gmail using ?
I think all these questions should be asked to new internet users whose have never hear about Gmail, Hotmail and Yahoo UI.
Jacob Creech
Tuesday, September 7th, 2010 04:35
Yeah, it is affected by peoples learned behaviors, but those learned habits are important, and often what we need to design to. I think there is a good spread of users on this site from Yahoo, Hotmail and Gmail so hopefully we avoid most of those biases.
JC Johnston
Monday, September 6th, 2010 10:07
I have accounts with all 3 mail services. I prefer Gmail over all the services, Even my premium and work accounts. I can not stand Yahoo interface and it annoys me every time I have to log in. Hotmail is not so bad, but I rather be using Gmail and I always recommend them to my clients who need a free email for buisness.
Jacob Creech
Tuesday, September 7th, 2010 04:30
Gosh, I feel for you have to deal with all three – especially Yahoo – the whole interface just depresses me. It’s interesting to get your insight based on using all three, glad to see you came to the same conclusion too.
Thanks for your comment.
Fatih
Sunday, September 5th, 2010 02:08
I use GMail all the time. I also got a passport account earlier so I use my email in MSN Messenger too.
- I love Gmail’s interface, Labs, Chat, Options and Efficiency!
- I hate hotmail. I (personally) think, everything that comes from microsoft sucks! Lol. My Opinion!
- Neutral to Yahoo! (I didn’t use it much, so I can’t rate/hate it :P)
Theo
Saturday, September 4th, 2010 21:58
That gmail wins was clear to me, just a bit disappointed by yahoo!, however interesting test.
Jacob Creech
Tuesday, September 7th, 2010 04:54
Thanks for the comment, glad you liked the test.
Mark S
Friday, September 3rd, 2010 17:46
So lets get this completely correct, the most usable site was hotmail, the most popular was gmail. In the tests (part 2) : yahoo 1, hotmail 3, gmail 2.
I know which I’d prefer to use :P
Jacob Creech
Sunday, September 5th, 2010 02:13
Well, except for the fact that replying on the same page, and banner ads vs text ads also contribute to usability, and Gmail was the only service offering those features. Peoples response to a UI is also an important factor here, and again Gmail was in the lead.
brad
Wednesday, October 20th, 2010 21:50
I’m not sure I would consider having any kind of ad, whether it be banner or text , an assett. I consider them a nuisance.. I still use OE because of the simplicity. I hate to give it up. I love hotmail because it actually has spam filters that work.. Verizon certainly has none.. I do not use a phone for play time like internet or e-mails. I am a contractor and am on the phone all day ; so I only read e-mails and surf the internet on a real computer. I would like to see my 17 yr old daughter start using an e-mail, so I was reading this article with interest. She only does texting and facebook, occassionally my space. Thanks for the info.
solcroft
Sunday, September 5th, 2010 21:58
Except that we know that people *cough*fanboys*cough* tend to offer answers skewed by their overall perception when asked a specific question about a specific area of their webmail service.
Jacob Creech
Tuesday, September 7th, 2010 04:59
Most of these questions it’s hard to be a fanboy, it just depends what you’re used to, what you like, and what works well.
solcroft
Wednesday, September 8th, 2010 15:52
Um, it’s exceedingly easy to be a fanboy for that one particular question. You could simply like *insert some other features or aspects here* and think every facet of the service is unmatched.
When you ask a question along the lines of “which do you like best”, you’re obviously throwing objectivity out of the window.
Jay
Friday, September 3rd, 2010 17:15
You were testing the old interface of Yahoo Mail. Why not test the new AJAXy interface?
Jacob Creech
Sunday, September 5th, 2010 02:09
That was the only Yahoo interface I had access too. I checked with a number of people I know who still use Yahoo, and they said that is the screen they see too.
In fact, I know all of these different providers are rolling out updates of one kind or another, but these are what were available, and hence what I tested.