Email UI Faceoff – GMail vs Hotmail vs Yahoo – The results

Welcome back to part two of our email UI faceoff – I recently wrote about the usability testing process, and set up a test for comparing the usability of Yahoo! Mail, Hotmail and Gmail.

Well, the results are in and we’ve got some pretty interesting feedback from the test – you can get directly to the test results yourself, or read on for our analysis and verdict.

Over the days we’ve been running the test, we had north of 300 people take it, and we think the numbers are a pretty good sample. Based on our analytics data, I could see that we had participants from 64 different countries, and it took the average user 2:56 to take the test.

So what did we learn from all that? Read on and find out:

Part one: Learned behaviors and conventions

Grids: The first three questions used a grid to discover users expected behaviours. When you overlay the grid over the different UIs you can see how the designs fit in with peoples perceptions.

Learned conventions tend to show through in this kind of area, but conventions are often an important standard for designers to follow (as you will see reflected in the results later on)

1) Where would you click to logout?

As you can see from the results, by far and away the majority of people expect to find the log out button on the top right hand side of the page. This is the most common location for a logout button across numerous websites (have a look for yourself), and so something people are very familiar with. If you want to put your logout button in a different location, you should make sure there is a good reason for doing so.

When overlaid, we can see both Gmail and Hotmail follow this convention, but for some reason, Yahoo! Mail locates their logout button in the middle of the page. What’s worse, it doesn’t render properly in the browser I’ve been using for this test – Safari 5.

Yahoo! Mail 0; Hotmail 1; Gmail 1

2) Where would you click to write a new email?

The results for this are fairly obvious – as you can see from the heatmap, more all less all users clicked in the top left hand corner to compose a new mail – this is the location for all three services we tested, and the most common location across more or less all webmail services.

Yahoo! Mail 1; Hotmail 2; Gmail 2

3)Where would you click to reply to an email?

Some interesting results for this one – there is really quite a spread all over the screen, which shows perhaps there aren’t quite so many learned behaviours for this. There is a concentration towards the top-middle and top left of the screen however. Lets overlay this over the different services and see how they compare:

Each of the services seems to have ever so slightly different locations for their buttons, and the users of each seem to have learnt these. I don’t see that one or the other is ahead in this test.

Personally though, I do like both the Hotmail and Gmail interface for having reply buttons at the bottom of messages, so you can easily reply after you’ve finished reading. What’s more, the Gmail message reply is the box you can see at the bottom of the messsage. Just put your cursor in there and you can begin typing. Pretty straight forward.

Yahoo! Mail 2; Hotmail 3; Gmail 3

4) Which interface do you prefer?

This is a bit of a popularity contest, but evidence shows all things being equal people have a better experience with nicer looking interfaces. Stands to reason. You can see the results here.

The numbers were:

  • Yahoo! Mail: 20%
  • Hotmail: 16%
  • Gmail: 59%

The remaining 5% is people either skipping the question or clicked the side of the interface. Regardless, these numbers are pretty defenitive, and Gmail is well ahead.

Yahoo! Mail 2; Hotmail 3; Gmail 4

5)Do you prefer your email threaded or non-threaded?

Great results for this one, fairly overwhelming:

  • Threaded: 73%
  • Not Threaded: 12%
  • Don’t know/care: 12%

At the present time, Gmail is the only provider that offers threded messages to all its users. Hotmail is currently rolling out threading on a limited basis, but by no means to all users (our test account doesn’t have threading), and Yahoo! Mail offers no threading at all. Results:

Yahoo! Mail 2; Hotmail 3; Gmail 5

6)Do you prefer to reply on a new page or the same page?

Another comprehensive result as you can see from the image below:

The numbers were as follows:

  • Reply on a new page: 11%
  • Reply on the same page: 81%
  • Don’t know/care: 5%

Again, Gmail is the only provider of the three that enables you to reply on the same page – a behaviour prefered by 81% of users tested.

Yahoo! Mail 2; Hotmail 3; Gmail 6

7) Do you prefer banner ads or text ads?

I was slightly surprised by the results of this test – banner ads are evidently more popular than I thought, and although the results were comprehensively in favour of text ads, it seems there is life for banner ads yet. The results were:

  • Banner ads: 17%
  • Text ads: 68%
  • Don’t know/care: 13%

Again though, seeing as Hotmail and Yahoo! Mail both use banner ads, and Gmail text ads, it looks like Gmail is the winner here.

Yahoo! Mail 2; Hotmail 3; Gmail 7

8) Do you prefer a site with or without a landing page?

I was surprised when testing Yahoo! Mail that they used a landing page when you log into your mail. This got me thinking as to whether or not this was the prefered behaviour of users. The results are:

  • With landing page: 7%
  • Without landing page: 82%
  • Don’t know/care: 3%

Yahoo! Mail is the only one of the three that uses landing pages – and obviously this isn’t very popular with users. Score:

Yahoo! Mail 2; Hotmail 4; Gmail 8

Part two: Usability comparison

For the next series of questions, the success criteria are less than 10 seconds to complete a task – thus giving the user time to read instructions and to ensure a usable interface – and a success rate of over 75% – to make sure most users are completing the task satisfactorily.

9) Where would you click to change your settings? (Yahoo! Mail)

As you can see in the results, the success rate here is 75%, but what we also need to pay attention to is the average time to complete this test – in this case, 27.58 seconds.

This shows us that while users can indeed find the settings area, they are taking too long to do in. In this case Yahoo! Mail could try to make the text more prominent or more visible in order to make this area more usable.

Yahoo! Mail 2; Hotmail 4; Gmail 8

10) Where would you click to access your calendar? (Gmail)

These results are a bit of a fail for Gmail – it fails on both fronts. There is only a 69% success rate, and it took the average user just under 14 seconds to complete the task.

If they wanted to make the calendar more visible, they could try moving it’s location, or featuring it in more than one location. The fact that 16% of people clicked on the left hand side indicates that adding a calendar button in that area would be beneficial.

Yahoo! Mail 2; Hotmail 4; Gmail 8

11) How would you mark a message as spam? (Hotmail)

And at last we have a task that meets our criteria. 76% of users (or 87% if you count the ones marking the messages) successfully completed the task with an average time of 9.16 seconds. This could still be improved upon, but isn’t a bad result.

Yahoo! Mail 2; Hotmail 5; Gmail 8

12) Where would you click to access your calendar? (Yahoo! Mail)

As you can see here, a pretty poor result. It took an average of 17.73 seconds for people to click, and after that, there was still a success rate of just 54%.

Yahoo obviously needs to put a lot more consideration in to the placement of the calendar button. The ammount of people clicking in the upper left hand side tends to suggest that many people look to this area, so perhaps by offering a link there they would increase their success rate.

Yahoo! Mail 2; Hotmail 5; Gmail 8

13) How would you mark a message as spam? (Gmail)

As you can see from the results, there was an 85% success rate, and an average click time of 6.38 seconds. This is obviously well inside our success criteria, and a strong result for Gmail.

Yahoo! Mail 2; Hotmail 5; Gmail 9

14) Where would you click to change your settings? (Hotmail)

Another good result for Hotmail here – an 80% success rate, and an average click time of 7.54 seconds. I think this makes a lot of sense as this is a very common location for a settings/options link to be.

Yahoo! Mail 2; Hotmail 6; Gmail 9

15) How would you mark a message as spam? (Yahoo! Mail)

A much better showing for Yahoo! Mail in this case, an 87% success rate, and a click time of just 4.13 seconds. This really shows the value of following conventions – bear this in mind when designing.

Yahoo! Mail 3; Hotmail 6; Gmail 9

16) Where would you click to change your settings? (Gmail)

A 91% success rate, and an average click time of 6.01 seconds show that Gmails setting page location is working very well – these are very good numbers.

Yahoo! Mail 3; Hotmail 6; Gmail 10

17) Where would you click to access your calendar? (Hotmail)

And another success to round out our tests. 5.5 seconds average click time, and a 78% success rate. Quite a spread on this page though, so it shows people still aren’t quite sure where to look for calendars.

Yahoo! Mail 3; Hotmail 7; Gmail 10


Conclusion

This was a pretty interesting test, and created some interesting results. It shows the value of following conventions and observing user behaviours when developing or reworking your designs. If you’d like to browse through the whole test results, you can go here.

The final scores were: Yahoo! Mail 3; Hotmail 7; Gmail 10.

While Gmail was ultimately ahead in these tests, all three of the services have areas they could improve upon.

Hotmail was also a reasonably strong contender, and with updates which they are currently rolling out, the usability of their service should continue to improve.

Yahoo! Mail was lagging far behing in third place, and really needs to pick up its game if it wants to remain competitive. They are probably relying on people who have been using their service for many years, and I would be surprised if they are still signing up many users.

How does this compare with your actual experience of these different webmail services? Are there any other sites you’d like see compared in the future? Be sure to let us know in the comments!

Jacob Creech

Jacob is a usability geek at IntuitionHQ.com - a new, quick, easy to use usability service from Boost New Media. When he's not indulging his passion for the internet, he is probably bloging about China and Chinese learning (having spent three years there) at Sinomatter.com. You can find him on twitter talking about usability @intuitionhq.

15 Smart Tools To Help You Build Your Freelance Business

Discover the awesome tools we use in making our clients comfortable and happy in learning new things every day.

Download Now

Comments

  1. Munif says

    Gmail is the best and would win if they took another test. I was using gmail for more than 4 years and i did not have any problems.

  2. Cosmin says

    Personally I like the Gmail interface because is simple and simple is always good and ergonomic! On the other hand, I also believe that having your own email server would make things easier for you.

  3. Augusto Malowda says

    Gmail is the best one in my opinion.
    With the change of UI, Google are truly making it more easier to use!
    More friendly for older generations.

  4. Munim says

    for me gmail :
    is less complicated
    takes no time to load
    the new look makes it even more great
    identifies emails that are important and marks them with a yellow marker
    hasnt got any distracting banners; only text ads in one row so it doesnt take all the space
    has got 25mb attachment; more than hotmail and yahoo mail
    all the important and common stuff is bigger and good on the eye at the top like reply, forward, report spam etc once you select or open an email. if any of u guys havent seen the new 2011 look yet u better before saying anything else against gmail. by the wy i have used hotmail aswell and am still using it but not much and in my opinion it isnt that attractive or less complicated as gmail. yahoo mail is not an option ..:D

  5. Rafael Nunes says

    Hey, that’s really the best test i’ve ever seen in my life, congratulations! really D:

  6. Darren says

    Do as I did, and stop reading this about half way as it starts to become heavily evident that it is biased solely towards Gmail.

  7. mihai says

    For me, hotmail clearly wins, though i use both hotmail and yahoo, google is just too selfish for my liking.

    first, hotmail has the best spam filter. I three years since i have this adress, i virtually got only 5 spam messages, and their senders (the real ones) were added to block list as soon as i deleted the emails, unlike yahoo, where you must keep the emails in spam folder for about a month.

    second, hotmail gives you FREE Microsoft office online, and 2GB of storage for virtually everything (I even keep a website in there). this also makes you able to create .docx documents online without having to buy/install/crack/whatever an office installation. This is HUGE in comparison with gmail.

    Third, their web pages have the most powerful encryption available around, you can safely use your hotmail account for things such as bank accounts and stuff like that.

  8. Ashay says

    For me, GMail clearly wins for a number of reasons ranging from prompt email delivery to superb spam filter to straight-to-inbox view to simple, yet elegant interface. But the biggest of them all is the fact that GMail is the only one that allows me to use my email client (I use Apple Mail) for free. In other words, of the three, only GMail does not charge fort IMAP or POP! But then, I believe web based email browsing is outdated, and noobish!

    • Jade says

      The problem with Gmail, is that, if anyone were to hack it, they could easily delete all of your recovery information without needing a verification link for removal from your cell #, or recovery email. With Yahoo, it is the same thing. And if you were to contact customer care, they would not do anything as they are very hard to work with. Your chances are very slim when recovering a yahoo account. For Hotmail, it generates a lot of random spam. Some aren’t even spam. Though, it requires a verification to remove your recovery information so if a hacker were trying to remove your recovery information so you couldn’t get in, you would be able to anyways. Also on Hotmail, you can always empty the spam box by clicking “empty”. I’m siding with Hotmail. Don’t try mail.com or AOL mail. They have many glitches in their system and often you get mail very late. Also past mail fails to show up even though you have saved it in your inbox. Hope this helps!

  9. me says

    Gmail is fully SSL so much more secure when you’re in airports, etc. but I can’t believe it doesn’t serach thruogh attachments. On yahoo email. you can search for any word(s) and it will look through all your emails and the contents of your attachements.

  10. G33K says

    I personally think that gmail is better then all the rest. Mainly because it leads directly to your email inbox

  11. kevin says

    You missed a couple important points that are reasons that I use yahoo.
    First of all, there is no real preview pane for gmail. This annoys me and is my main reason for not using it. The preview pane in hotmail is even better than the yahoo since I prefer a vertical pane rather than the horizontal pane offered by yahoo, but with hotmail, you can only see about 25 messages in the inbox before having to click to see the next page of messages. So if you have 200 facebook updates you literally have to select a page worth, delete, then the next page, select, delete. It’s a pain in the arse. If you have a bunch of messages you want to keep in between these, it gets even uglier. I’m not sure if this is the same on gmail or not. On yahoo, you can scroll and see every message in your inbox. If you have a thousand facebook messages, you can sort by sender, highlight the top one, CTRL and select the bottom one, then press the delete key and voila, they are all gone. This is so much more important than where I expect to find something when I click. After you use it a few times, you get used to that. If you have to manage a busy inbox, Yahoo is the way to go.

    • pipi008 says

      kevin, apparently you are not a very advanced user. In Gmail you can VERY EASILY create a rule which automatically puts a label on all your Facebook messages and they don’t clutter your inbox. If you want to read them – go to your FB label. If you want to delete them – delete everything in your FB label. Really – it take 10 seconds to set up. If you have to manage a busy inbox – Gmail is the way to go! ;)

    • Andrew says

      I totally agree, the Preview Pane is one of the best features of Yahoo and Hotmail. However, now that Yahoo has stuffed up their awesome Mail Client with this new crappy bloated unreliable Beta version introduced in 2011, I am now looking for another Mail client. Yahoo was good, now it is total rubbish.

  12. Mark says

    I prefer Gmail by far over the others. Yahoo! is a distant 3rd on my list. Why they can’t figure it out after all this time I’m not sure.

    • zbob says

      for me the yahoo interface is waaaay better …
      especially for business … i have been using google apps email for about a month now .. still frustrated. things that were easy in yahoo are painful in google
      – – tabbed inteface … great in yahoo .. does not exist in google. with tabs, i could have 3 emails in process, inbox, sent, and whatever i need at a single click. google can done one thing at a time.

      preview is nice in yahooo .. i could click through multiple emails in the in box .. know what they are and never loose track of where i am. google – one thing at a time.

      the conversation email is occasionally useful.. mostly its frustrating ..kinda tough to keep up with where i am .. especially when i may have 3 or 4 different conversations with the same contact .. all about the same event .. but since the subjects are different they are all different conversations.

      so .. maybe this will get better… maybe i will learn what everyone else is so excited about .. for now it seems like a big step back .. yahoo way better

      on spam.. google much better .. on calendar.. google is light years ahead.

      but on business email .. give me yahoo any day !

      my .02

      mbj

  13. Jeff says

    I wholeheartedly agree. Gmail, by far is the most user friendly, and I have used all 3. At present, I use Hotmail (I belong to MS Technet) and Gmail (because I want to).

  14. Ben says

    Another intersting test may have been to evaluate each of the sign up processes. I use google for everything at the moment but set up a new gmail account for my new website, the sign up process was a ruddy nightmare simply because of the horrible Captcha spam filter they use, it took me a good 15 attempts to get past it, maybe the sign up on Yahoo and Live is a lot easier.

    Really liked this post though, great insights into some major email clients.

  15. Marcus Tucker says

    Interesting reading, but I find the analysis rather incomplete because the most obvious questions is left unasked, let alone unanswered – i.e. what correlation is there between the email service(s) that the respondents actually use and the usability/location preferences that you have inferred from their clicks?

    This is very important because I would expect users of each service to prefer things to be where they usually are (i.e. when they use their webmail service), so your results may simply be reflecting these habits, and the relative market share of the email providers amongst your test subjects, rather than offering real insight into the best place for features to be located.

    • Don says

      100% agree, HUGE flaw in the study. people’s preferences are obviously confounded by their current habits which results from the service they use. So this isn’t really testing people’s true a priori preferences but rather, just a popularity contest.

  16. S.Khan says

    I clearly am in the minority but I hate hotmail, its user interface and most of all the spam filter (non-existent?). I actually really like Yahoo because its got a very explorer kind of feel to it which makes it very easy. But yes, I agree, Gmail is the best.

  17. JC Johnston says

    I have accounts with all 3 mail services. I prefer Gmail over all the services, Even my premium and work accounts. I can not stand Yahoo interface and it annoys me every time I have to log in. Hotmail is not so bad, but I rather be using Gmail and I always recommend them to my clients who need a free email for buisness.

    • Jacob Creech says

      Gosh, I feel for you have to deal with all three – especially Yahoo – the whole interface just depresses me. It’s interesting to get your insight based on using all three, glad to see you came to the same conclusion too.

      Thanks for your comment.

  18. TingTingBen says

    Very nice !
    But aren’t all these results influenced by years of Gmail using ?
    I think all these questions should be asked to new internet users whose have never hear about Gmail, Hotmail and Yahoo UI.

    • Jacob Creech says

      Yeah, it is affected by peoples learned behaviors, but those learned habits are important, and often what we need to design to. I think there is a good spread of users on this site from Yahoo, Hotmail and Gmail so hopefully we avoid most of those biases.

  19. Fatih says

    I use GMail all the time. I also got a passport account earlier so I use my email in MSN Messenger too.

    – I love Gmail’s interface, Labs, Chat, Options and Efficiency!
    – I hate hotmail. I (personally) think, everything that comes from microsoft sucks! Lol. My Opinion!
    – Neutral to Yahoo! (I didn’t use it much, so I can’t rate/hate it :P)

  20. Mariano says

    Jacob, Hotmail is now sending you to a landing page too. Not just Yahoo. When I log into my account, it sends me to the Windows Live Home and then I have to click “Go to Inbox” or “Hotmail”.

    • Jacob Creech says

      Thanks for letting me know. It seems the roll out lots of different ‘features’ to different segments at different times. I still don’t have a landing page – interesting to know that they’ve put one in for you. Based on these results I don’t think it will be a popular choice.

  21. Terri Lockerridge says

    1st Webdesigner asks:
    How does this compare with your actual experience of these different webmail services?
    When I started trying to learn how to access and use the internet and how to utilize email.. Hotmail was the first I tried to use. As with any Microsoft product offering.. it is the hardest and most difficult place for a newbie to try!
    They assume that the user is already knowledgeable about things and just do not try to be user friendly in a simple and straightforward way. I would try to send a simple email and I could not get through all the steps I had to work out before I was totally confused and would give up.
    Then I tried Yahoo mail and found it to be simple and so very easy to use with only a very few steps to be successful. I used their services until they did the change over to the new platform and I left them flat after they invaded my privacy in a lot of small ways, and they changed the mail site to be so complicated with so many links on it that you could not move your cursor an inch without a popup proclaiming to be my calendar or the ever present and annoyances of trying to get me to buy their premium service which I do not need. Too much garbage and you can’t find a darn thing that you use everyday easily. Too much in too small a space, invading privacy, and not listening when we give them feedback on how to improve things really turned me off of them.
    So I went to Gmail. I find it to be simple, with a chat built right in directly to it, easy to understand and access all my information, plenty of storage, GoogleTalk for free telephone services, easy to connect to and download a stored item from their website for hosting and editing documents, and they do not try to cram so many ads onto the page that I have to be really intense on where I click to avoid inadvertently clicking onto an ad.

    I have a couple of tips for any free email service out there.

    1. Keep in mind that not all people know how to speak computerlish and to keep it simple without many steps to accomplish a task.
    2. Keep your customer service well staffed and respond to any problems that are reported to them quickly… and no matter how hard it is to help someone, help them till a technical problem is resolved, stay with someone until they learn how to use the service and do not make them feel as if they are a pest, and for the love of pete.. if your going to offer the html option to create an email.. then give us some help on using it to make a great email for any reason! I have yet to find any email service who will give you any real help on this issue. I have the html capability in Gmail.. but there is no, and I mean no really simple, useful help on using it!

    Are there any other sites you’d like see compared in the future?
    I would like to see how many people are using Outlook Express and if they find it to be difficult to use as in setting up other email services as well as Hotmail into it. I do not use it because I can’t get it to accept forwarded mail or any mail other than Hotmail.

    • Jacob Creech says

      Hi Terri, thanks for your comment! It’s really interesting to get input from users such as yourself, and you are right, most companies forget who their core audience is, and just design for themselves.

      That is the point of usability testing, so you can get real information from real users.

      Interesting you talk about Outlook Express. I had kind of forgotten it still existed – is it even in Windows 7? I guess I saw this from my own perspective, as I use Macs 99% of the time, and use webmail 99% of the time. I use mail.app on my iPhone, which seems to do everything fairly well, and Thunderbird occasionally when I use desktop mail.

      Not sure why anyone would even touch outlook with these options around.

    • Jacob Creech says

      I think you sum up Google pretty well with that comment! Hopefully they will learn that lesson soon.

  22. says

    Amazing results, Jacob! However we must consider that the people who took the test are probably web designers, and Gmail is the preferred mail for us all, so we already know it.

    It would be interesting a similar test between all kind of people.

    Anyway, congrats!
    Marcelo.

    • Jacob Creech says

      Usually when we publish a test we aim it at the specific market for the site. In this case I think it’s OK to test a technical audience, because we are all email users anyway. I’m not sure that the majority of readers here are Gmail users though – it seems there is a fairly even distribution of Hotmail, Gmail and Yahoo users.

      Anyway, thanks for your comment!

  23. Mark says

    It’s pretty sad that Yahoo! has had YEARS to get webmail right (and a big head start on Gmail) and they STILL haven’t really gotten it figured out. What the heck are they working on over there?

  24. says

    So lets get this completely correct, the most usable site was hotmail, the most popular was gmail. In the tests (part 2) : yahoo 1, hotmail 3, gmail 2.

    I know which I’d prefer to use :P

    • Jacob Creech says

      Well, except for the fact that replying on the same page, and banner ads vs text ads also contribute to usability, and Gmail was the only service offering those features. Peoples response to a UI is also an important factor here, and again Gmail was in the lead.

      • solcroft says

        Except that we know that people *cough*fanboys*cough* tend to offer answers skewed by their overall perception when asked a specific question about a specific area of their webmail service.

        • Jacob Creech says

          Most of these questions it’s hard to be a fanboy, it just depends what you’re used to, what you like, and what works well.

          • solcroft says

            Um, it’s exceedingly easy to be a fanboy for that one particular question. You could simply like *insert some other features or aspects here* and think every facet of the service is unmatched.

            When you ask a question along the lines of “which do you like best”, you’re obviously throwing objectivity out of the window.

      • brad says

        I’m not sure I would consider having any kind of ad, whether it be banner or text , an assett. I consider them a nuisance.. I still use OE because of the simplicity. I hate to give it up. I love hotmail because it actually has spam filters that work.. Verizon certainly has none.. I do not use a phone for play time like internet or e-mails. I am a contractor and am on the phone all day ; so I only read e-mails and surf the internet on a real computer. I would like to see my 17 yr old daughter start using an e-mail, so I was reading this article with interest. She only does texting and facebook, occassionally my space. Thanks for the info.

    • Jacob Creech says

      That was the only Yahoo interface I had access too. I checked with a number of people I know who still use Yahoo, and they said that is the screen they see too.

      In fact, I know all of these different providers are rolling out updates of one kind or another, but these are what were available, and hence what I tested.

  25. arun says

    Interesting results, but we all know Gmail would win, I also took the test, and now results are similar to what i expected
    Gmail is way ahead in terms of usability that other email services
    :)

      • Ram says

        Software always needs improvements. Point is, it’s a lot more intuitive than the competitors. Yahoo! lost it when they gave us 10 MB mailboxes and wanted people to pay a ridiculous fees for 100MB mailboxes. Once GMail came they got blown out of the water, and deservedly so. It’s a surprise Yahoo are still around. I’d hope they went bankrupt soon.

        I don’t care for the text ads, so GMail works for me, but Yahoo were just greedy beyond anything and they deserved to no longer be the top email provider.

    • Jacob Creech says

      Thanks guys; I must say I wasn’t surprised that Gmail did well (and Hotmail did reasonably too), but I was surprised how far behind Yahoo! is, especially considering how much of the market they control still.

      It was really interesting to observe different behaviors from people as well; I’m wondering if anyone has any suggestions for what we should test next?