Email UI Faceoff – GMail vs Hotmail vs Yahoo – The results

Posted in Web Design • Posted on 56 Comments

Welcome back to part two of our email UI faceoff – I recently wrote about the usability testing process, and set up a test for comparing the usability of Yahoo! Mail, Hotmail and Gmail.

Well, the results are in and we’ve got some pretty interesting feedback from the test – you can get directly to the test results yourself, or read on for our analysis and verdict.

Over the days we’ve been running the test, we had north of 300 people take it, and we think the numbers are a pretty good sample. Based on our analytics data, I could see that we had participants from 64 different countries, and it took the average user 2:56 to take the test.

So what did we learn from all that? Read on and find out:

Part one: Learned behaviors and conventions

Grids: The first three questions used a grid to discover users expected behaviours. When you overlay the grid over the different UIs you can see how the designs fit in with peoples perceptions.

Learned conventions tend to show through in this kind of area, but conventions are often an important standard for designers to follow (as you will see reflected in the results later on)

1) Where would you click to logout?

As you can see from the results, by far and away the majority of people expect to find the log out button on the top right hand side of the page. This is the most common location for a logout button across numerous websites (have a look for yourself), and so something people are very familiar with. If you want to put your logout button in a different location, you should make sure there is a good reason for doing so.

When overlaid, we can see both Gmail and Hotmail follow this convention, but for some reason, Yahoo! Mail locates their logout button in the middle of the page. What’s worse, it doesn’t render properly in the browser I’ve been using for this test – Safari 5.

Yahoo! Mail 0; Hotmail 1; Gmail 1

2) Where would you click to write a new email?

The results for this are fairly obvious – as you can see from the heatmap, more all less all users clicked in the top left hand corner to compose a new mail – this is the location for all three services we tested, and the most common location across more or less all webmail services.

Yahoo! Mail 1; Hotmail 2; Gmail 2

3)Where would you click to reply to an email?

Some interesting results for this one – there is really quite a spread all over the screen, which shows perhaps there aren’t quite so many learned behaviours for this. There is a concentration towards the top-middle and top left of the screen however. Lets overlay this over the different services and see how they compare:

Each of the services seems to have ever so slightly different locations for their buttons, and the users of each seem to have learnt these. I don’t see that one or the other is ahead in this test.

Personally though, I do like both the Hotmail and Gmail interface for having reply buttons at the bottom of messages, so you can easily reply after you’ve finished reading. What’s more, the Gmail message reply is the box you can see at the bottom of the messsage. Just put your cursor in there and you can begin typing. Pretty straight forward.

Yahoo! Mail 2; Hotmail 3; Gmail 3

4) Which interface do you prefer?

This is a bit of a popularity contest, but evidence shows all things being equal people have a better experience with nicer looking interfaces. Stands to reason. You can see the results here.

The numbers were:

  • Yahoo! Mail: 20%
  • Hotmail: 16%
  • Gmail: 59%

The remaining 5% is people either skipping the question or clicked the side of the interface. Regardless, these numbers are pretty defenitive, and Gmail is well ahead.

Yahoo! Mail 2; Hotmail 3; Gmail 4

5)Do you prefer your email threaded or non-threaded?

Great results for this one, fairly overwhelming:

  • Threaded: 73%
  • Not Threaded: 12%
  • Don’t know/care: 12%

At the present time, Gmail is the only provider that offers threded messages to all its users. Hotmail is currently rolling out threading on a limited basis, but by no means to all users (our test account doesn’t have threading), and Yahoo! Mail offers no threading at all. Results:

Yahoo! Mail 2; Hotmail 3; Gmail 5

6)Do you prefer to reply on a new page or the same page?

Another comprehensive result as you can see from the image below:

The numbers were as follows:

  • Reply on a new page: 11%
  • Reply on the same page: 81%
  • Don’t know/care: 5%

Again, Gmail is the only provider of the three that enables you to reply on the same page – a behaviour prefered by 81% of users tested.

Yahoo! Mail 2; Hotmail 3; Gmail 6

7) Do you prefer banner ads or text ads?

I was slightly surprised by the results of this test – banner ads are evidently more popular than I thought, and although the results were comprehensively in favour of text ads, it seems there is life for banner ads yet. The results were:

  • Banner ads: 17%
  • Text ads: 68%
  • Don’t know/care: 13%

Again though, seeing as Hotmail and Yahoo! Mail both use banner ads, and Gmail text ads, it looks like Gmail is the winner here.

Yahoo! Mail 2; Hotmail 3; Gmail 7

8) Do you prefer a site with or without a landing page?

I was surprised when testing Yahoo! Mail that they used a landing page when you log into your mail. This got me thinking as to whether or not this was the prefered behaviour of users. The results are:

  • With landing page: 7%
  • Without landing page: 82%
  • Don’t know/care: 3%

Yahoo! Mail is the only one of the three that uses landing pages – and obviously this isn’t very popular with users. Score:

Yahoo! Mail 2; Hotmail 4; Gmail 8

Part two: Usability comparison

For the next series of questions, the success criteria are less than 10 seconds to complete a task – thus giving the user time to read instructions and to ensure a usable interface – and a success rate of over 75% – to make sure most users are completing the task satisfactorily.

9) Where would you click to change your settings? (Yahoo! Mail)

As you can see in the results, the success rate here is 75%, but what we also need to pay attention to is the average time to complete this test – in this case, 27.58 seconds.

This shows us that while users can indeed find the settings area, they are taking too long to do in. In this case Yahoo! Mail could try to make the text more prominent or more visible in order to make this area more usable.

Yahoo! Mail 2; Hotmail 4; Gmail 8

10) Where would you click to access your calendar? (Gmail)

These results are a bit of a fail for Gmail – it fails on both fronts. There is only a 69% success rate, and it took the average user just under 14 seconds to complete the task.

If they wanted to make the calendar more visible, they could try moving it’s location, or featuring it in more than one location. The fact that 16% of people clicked on the left hand side indicates that adding a calendar button in that area would be beneficial.

Yahoo! Mail 2; Hotmail 4; Gmail 8

11) How would you mark a message as spam? (Hotmail)

And at last we have a task that meets our criteria. 76% of users (or 87% if you count the ones marking the messages) successfully completed the task with an average time of 9.16 seconds. This could still be improved upon, but isn’t a bad result.

Yahoo! Mail 2; Hotmail 5; Gmail 8

12) Where would you click to access your calendar? (Yahoo! Mail)

As you can see here, a pretty poor result. It took an average of 17.73 seconds for people to click, and after that, there was still a success rate of just 54%.

Yahoo obviously needs to put a lot more consideration in to the placement of the calendar button. The ammount of people clicking in the upper left hand side tends to suggest that many people look to this area, so perhaps by offering a link there they would increase their success rate.

Yahoo! Mail 2; Hotmail 5; Gmail 8

13) How would you mark a message as spam? (Gmail)

As you can see from the results, there was an 85% success rate, and an average click time of 6.38 seconds. This is obviously well inside our success criteria, and a strong result for Gmail.

Yahoo! Mail 2; Hotmail 5; Gmail 9

14) Where would you click to change your settings? (Hotmail)

Another good result for Hotmail here – an 80% success rate, and an average click time of 7.54 seconds. I think this makes a lot of sense as this is a very common location for a settings/options link to be.

Yahoo! Mail 2; Hotmail 6; Gmail 9

15) How would you mark a message as spam? (Yahoo! Mail)

A much better showing for Yahoo! Mail in this case, an 87% success rate, and a click time of just 4.13 seconds. This really shows the value of following conventions – bear this in mind when designing.

Yahoo! Mail 3; Hotmail 6; Gmail 9

16) Where would you click to change your settings? (Gmail)

A 91% success rate, and an average click time of 6.01 seconds show that Gmails setting page location is working very well – these are very good numbers.

Yahoo! Mail 3; Hotmail 6; Gmail 10

17) Where would you click to access your calendar? (Hotmail)

And another success to round out our tests. 5.5 seconds average click time, and a 78% success rate. Quite a spread on this page though, so it shows people still aren’t quite sure where to look for calendars.

Yahoo! Mail 3; Hotmail 7; Gmail 10


Conclusion

This was a pretty interesting test, and created some interesting results. It shows the value of following conventions and observing user behaviours when developing or reworking your designs. If you’d like to browse through the whole test results, you can go here.

The final scores were: Yahoo! Mail 3; Hotmail 7; Gmail 10.

While Gmail was ultimately ahead in these tests, all three of the services have areas they could improve upon.

Hotmail was also a reasonably strong contender, and with updates which they are currently rolling out, the usability of their service should continue to improve.

Yahoo! Mail was lagging far behing in third place, and really needs to pick up its game if it wants to remain competitive. They are probably relying on people who have been using their service for many years, and I would be surprised if they are still signing up many users.

How does this compare with your actual experience of these different webmail services? Are there any other sites you’d like see compared in the future? Be sure to let us know in the comments!

5 Written ArticlesWebsite

Jacob is a usability geek at IntuitionHQ.com - a new, quick, easy to use usability service from Boost New Media. When he's not indulging his passion for the internet, he is probably bloging about China and Chinese learning (having spent three years there) at Sinomatter.com. You can find him on twitter talking about usability @intuitionhq.

56 Comments Best Comments First
  • Jim

    Friday, September 3rd, 2010 12:23

    1

    Nice analyzing..

    0
  • TingTingBen

    Monday, September 6th, 2010 09:35

    20

    Very nice !
    But aren’t all these results influenced by years of Gmail using ?
    I think all these questions should be asked to new internet users whose have never hear about Gmail, Hotmail and Yahoo UI.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:35

      23

      Yeah, it is affected by peoples learned behaviors, but those learned habits are important, and often what we need to design to. I think there is a good spread of users on this site from Yahoo, Hotmail and Gmail so hopefully we avoid most of those biases.

      0
  • JC Johnston

    Monday, September 6th, 2010 10:07

    21

    I have accounts with all 3 mail services. I prefer Gmail over all the services, Even my premium and work accounts. I can not stand Yahoo interface and it annoys me every time I have to log in. Hotmail is not so bad, but I rather be using Gmail and I always recommend them to my clients who need a free email for buisness.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:30

      22

      Gosh, I feel for you have to deal with all three – especially Yahoo – the whole interface just depresses me. It’s interesting to get your insight based on using all three, glad to see you came to the same conclusion too.

      Thanks for your comment.

      0
  • Fatih

    Sunday, September 5th, 2010 02:08

    15

    I use GMail all the time. I also got a passport account earlier so I use my email in MSN Messenger too.

    – I love Gmail’s interface, Labs, Chat, Options and Efficiency!
    – I hate hotmail. I (personally) think, everything that comes from microsoft sucks! Lol. My Opinion!
    – Neutral to Yahoo! (I didn’t use it much, so I can’t rate/hate it :P)

    0
  • Theo

    Saturday, September 4th, 2010 21:58

    14

    That gmail wins was clear to me, just a bit disappointed by yahoo!, however interesting test.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:54

      24

      Thanks for the comment, glad you liked the test.

      0
  • Mark S

    Friday, September 3rd, 2010 17:46

    6

    So lets get this completely correct, the most usable site was hotmail, the most popular was gmail. In the tests (part 2) : yahoo 1, hotmail 3, gmail 2.

    I know which I’d prefer to use :P

    0
    • Jacob Creech

      Sunday, September 5th, 2010 02:13

      17

      Well, except for the fact that replying on the same page, and banner ads vs text ads also contribute to usability, and Gmail was the only service offering those features. Peoples response to a UI is also an important factor here, and again Gmail was in the lead.

      0
      • brad

        Wednesday, October 20th, 2010 21:50

        39

        I’m not sure I would consider having any kind of ad, whether it be banner or text , an assett. I consider them a nuisance.. I still use OE because of the simplicity. I hate to give it up. I love hotmail because it actually has spam filters that work.. Verizon certainly has none.. I do not use a phone for play time like internet or e-mails. I am a contractor and am on the phone all day ; so I only read e-mails and surf the internet on a real computer. I would like to see my 17 yr old daughter start using an e-mail, so I was reading this article with interest. She only does texting and facebook, occassionally my space. Thanks for the info.

        0
      • solcroft

        Sunday, September 5th, 2010 21:58

        19

        Except that we know that people *cough*fanboys*cough* tend to offer answers skewed by their overall perception when asked a specific question about a specific area of their webmail service.

        0
        • Jacob Creech

          Tuesday, September 7th, 2010 04:59

          26

          Most of these questions it’s hard to be a fanboy, it just depends what you’re used to, what you like, and what works well.

          0
          • solcroft

            Wednesday, September 8th, 2010 15:52

            35

            Um, it’s exceedingly easy to be a fanboy for that one particular question. You could simply like *insert some other features or aspects here* and think every facet of the service is unmatched.

            When you ask a question along the lines of “which do you like best”, you’re obviously throwing objectivity out of the window.

            0
  • Jay

    Friday, September 3rd, 2010 17:15

    5

    You were testing the old interface of Yahoo Mail. Why not test the new AJAXy interface?

    0
    • Jacob Creech

      Sunday, September 5th, 2010 02:09

      16

      That was the only Yahoo interface I had access too. I checked with a number of people I know who still use Yahoo, and they said that is the screen they see too.

      In fact, I know all of these different providers are rolling out updates of one kind or another, but these are what were available, and hence what I tested.

      0
  • arun

    Friday, September 3rd, 2010 12:51

    2

    Interesting results, but we all know Gmail would win, I also took the test, and now results are similar to what i expected
    Gmail is way ahead in terms of usability that other email services
    :)

    0
    • Jacob Creech

      Friday, September 3rd, 2010 15:34

      9

      Thanks guys; I must say I wasn’t surprised that Gmail did well (and Hotmail did reasonably too), but I was surprised how far behind Yahoo! is, especially considering how much of the market they control still.

      It was really interesting to observe different behaviors from people as well; I’m wondering if anyone has any suggestions for what we should test next?

      0
    • Swamykant

      Friday, September 3rd, 2010 14:57

      3

      You are right .. Gmail is the email services – It is Simple & Smart.. However needs some more improvements :)

      0
      • Ram

        Friday, September 3rd, 2010 15:51

        4

        Software always needs improvements. Point is, it’s a lot more intuitive than the competitors. Yahoo! lost it when they gave us 10 MB mailboxes and wanted people to pay a ridiculous fees for 100MB mailboxes. Once GMail came they got blown out of the water, and deservedly so. It’s a surprise Yahoo are still around. I’d hope they went bankrupt soon.

        I don’t care for the text ads, so GMail works for me, but Yahoo were just greedy beyond anything and they deserved to no longer be the top email provider.

        0
  • Mark

    Friday, September 3rd, 2010 17:57

    7

    It’s pretty sad that Yahoo! has had YEARS to get webmail right (and a big head start on Gmail) and they STILL haven’t really gotten it figured out. What the heck are they working on over there?

    0
    • Jacob Creech

      Sunday, September 5th, 2010 02:14

      18

      Haha, I’ve been asking myself the same question for a while now!

      0
  • Marcelo

    Friday, September 3rd, 2010 19:15

    8

    Amazing results, Jacob! However we must consider that the people who took the test are probably web designers, and Gmail is the preferred mail for us all, so we already know it.

    It would be interesting a similar test between all kind of people.

    Anyway, congrats!
    Marcelo.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:10

      30

      Usually when we publish a test we aim it at the specific market for the site. In this case I think it’s OK to test a technical audience, because we are all email users anyway. I’m not sure that the majority of readers here are Gmail users though – it seems there is a fairly even distribution of Hotmail, Gmail and Yahoo users.

      Anyway, thanks for your comment!

      0
  • Mariano

    Saturday, September 4th, 2010 19:55

    13

    Jacob, Hotmail is now sending you to a landing page too. Not just Yahoo. When I log into my account, it sends me to the Windows Live Home and then I have to click “Go to Inbox” or “Hotmail”.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:55

      25

      Thanks for letting me know. It seems the roll out lots of different ‘features’ to different segments at different times. I still don’t have a landing page – interesting to know that they’ve put one in for you. Based on these results I don’t think it will be a popular choice.

      0
  • chaitanya varma

    Saturday, September 4th, 2010 19:43

    12

    wonderful analysis!

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:08

      29

      Thanks very much! Glad you liked it.

      0
  • Terri Lockerridge

    Saturday, September 4th, 2010 07:48

    11

    1st Webdesigner asks:
    How does this compare with your actual experience of these different webmail services?
    When I started trying to learn how to access and use the internet and how to utilize email.. Hotmail was the first I tried to use. As with any Microsoft product offering.. it is the hardest and most difficult place for a newbie to try!
    They assume that the user is already knowledgeable about things and just do not try to be user friendly in a simple and straightforward way. I would try to send a simple email and I could not get through all the steps I had to work out before I was totally confused and would give up.
    Then I tried Yahoo mail and found it to be simple and so very easy to use with only a very few steps to be successful. I used their services until they did the change over to the new platform and I left them flat after they invaded my privacy in a lot of small ways, and they changed the mail site to be so complicated with so many links on it that you could not move your cursor an inch without a popup proclaiming to be my calendar or the ever present and annoyances of trying to get me to buy their premium service which I do not need. Too much garbage and you can’t find a darn thing that you use everyday easily. Too much in too small a space, invading privacy, and not listening when we give them feedback on how to improve things really turned me off of them.
    So I went to Gmail. I find it to be simple, with a chat built right in directly to it, easy to understand and access all my information, plenty of storage, GoogleTalk for free telephone services, easy to connect to and download a stored item from their website for hosting and editing documents, and they do not try to cram so many ads onto the page that I have to be really intense on where I click to avoid inadvertently clicking onto an ad.

    I have a couple of tips for any free email service out there.

    1. Keep in mind that not all people know how to speak computerlish and to keep it simple without many steps to accomplish a task.
    2. Keep your customer service well staffed and respond to any problems that are reported to them quickly… and no matter how hard it is to help someone, help them till a technical problem is resolved, stay with someone until they learn how to use the service and do not make them feel as if they are a pest, and for the love of pete.. if your going to offer the html option to create an email.. then give us some help on using it to make a great email for any reason! I have yet to find any email service who will give you any real help on this issue. I have the html capability in Gmail.. but there is no, and I mean no really simple, useful help on using it!

    Are there any other sites you’d like see compared in the future?
    I would like to see how many people are using Outlook Express and if they find it to be difficult to use as in setting up other email services as well as Hotmail into it. I do not use it because I can’t get it to accept forwarded mail or any mail other than Hotmail.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:06

      28

      Hi Terri, thanks for your comment! It’s really interesting to get input from users such as yourself, and you are right, most companies forget who their core audience is, and just design for themselves.

      That is the point of usability testing, so you can get real information from real users.

      Interesting you talk about Outlook Express. I had kind of forgotten it still existed – is it even in Windows 7? I guess I saw this from my own perspective, as I use Macs 99% of the time, and use webmail 99% of the time. I use mail.app on my iPhone, which seems to do everything fairly well, and Thunderbird occasionally when I use desktop mail.

      Not sure why anyone would even touch outlook with these options around.

      0
  • Warren

    Friday, September 3rd, 2010 18:31

    10

    Gmail is by far the best, however it does lack a beautiful interface.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:00

      27

      I think you sum up Google pretty well with that comment! Hopefully they will learn that lesson soon.

      0
  • Darren

    Monday, September 5th, 2011 04:17

    49

    Do as I did, and stop reading this about half way as it starts to become heavily evident that it is biased solely towards Gmail.

    0
    • ^agree

      Tuesday, September 20th, 2011 10:09

      50

      Sobias towards Gmail. Ruins the point of this exercise

      0
  • mihai

    Wednesday, August 24th, 2011 20:57

    48

    For me, hotmail clearly wins, though i use both hotmail and yahoo, google is just too selfish for my liking.

    first, hotmail has the best spam filter. I three years since i have this adress, i virtually got only 5 spam messages, and their senders (the real ones) were added to block list as soon as i deleted the emails, unlike yahoo, where you must keep the emails in spam folder for about a month.

    second, hotmail gives you FREE Microsoft office online, and 2GB of storage for virtually everything (I even keep a website in there). this also makes you able to create .docx documents online without having to buy/install/crack/whatever an office installation. This is HUGE in comparison with gmail.

    Third, their web pages have the most powerful encryption available around, you can safely use your hotmail account for things such as bank accounts and stuff like that.

    0
  • Ashay

    Tuesday, May 31st, 2011 17:46

    46

    For me, GMail clearly wins for a number of reasons ranging from prompt email delivery to superb spam filter to straight-to-inbox view to simple, yet elegant interface. But the biggest of them all is the fact that GMail is the only one that allows me to use my email client (I use Apple Mail) for free. In other words, of the three, only GMail does not charge fort IMAP or POP! But then, I believe web based email browsing is outdated, and noobish!

    0
    • Jade

      Sunday, November 27th, 2011 21:21

      54

      The problem with Gmail, is that, if anyone were to hack it, they could easily delete all of your recovery information without needing a verification link for removal from your cell #, or recovery email. With Yahoo, it is the same thing. And if you were to contact customer care, they would not do anything as they are very hard to work with. Your chances are very slim when recovering a yahoo account. For Hotmail, it generates a lot of random spam. Some aren’t even spam. Though, it requires a verification to remove your recovery information so if a hacker were trying to remove your recovery information so you couldn’t get in, you would be able to anyways. Also on Hotmail, you can always empty the spam box by clicking “empty”. I’m siding with Hotmail. Don’t try mail.com or AOL mail. They have many glitches in their system and often you get mail very late. Also past mail fails to show up even though you have saved it in your inbox. Hope this helps!

      0
  • Rafael Nunes

    Thursday, October 27th, 2011 23:55

    51

    Hey, that’s really the best test i’ve ever seen in my life, congratulations! really D:

    0
  • Munif

    Sunday, February 26th, 2012 00:24

    56

    Gmail is the best and would win if they took another test. I was using gmail for more than 4 years and i did not have any problems.

    0
  • Cosmin

    Monday, January 2nd, 2012 21:26

    55

    Personally I like the Gmail interface because is simple and simple is always good and ergonomic! On the other hand, I also believe that having your own email server would make things easier for you.

    0
  • Munim

    Thursday, November 3rd, 2011 22:22

    52

    for me gmail :
    is less complicated
    takes no time to load
    the new look makes it even more great
    identifies emails that are important and marks them with a yellow marker
    hasnt got any distracting banners; only text ads in one row so it doesnt take all the space
    has got 25mb attachment; more than hotmail and yahoo mail
    all the important and common stuff is bigger and good on the eye at the top like reply, forward, report spam etc once you select or open an email. if any of u guys havent seen the new 2011 look yet u better before saying anything else against gmail. by the wy i have used hotmail aswell and am still using it but not much and in my opinion it isnt that attractive or less complicated as gmail. yahoo mail is not an option ..:D

    0
  • Augusto Malowda

    Thursday, November 17th, 2011 18:11

    53

    Gmail is the best one in my opinion.
    With the change of UI, Google are truly making it more easier to use!
    More friendly for older generations.

    0
  • me

    Wednesday, May 18th, 2011 20:34

    44

    Gmail is fully SSL so much more secure when you’re in airports, etc. but I can’t believe it doesn’t serach thruogh attachments. On yahoo email. you can search for any word(s) and it will look through all your emails and the contents of your attachements.

    0
  • Joanne

    Wednesday, September 8th, 2010 10:08

    34

    It would be interesting to see how users in China respond to a similar poll.

    0
  • Ben

    Tuesday, September 7th, 2010 17:52

    33

    Another intersting test may have been to evaluate each of the sign up processes. I use google for everything at the moment but set up a new gmail account for my new website, the sign up process was a ruddy nightmare simply because of the horrible Captcha spam filter they use, it took me a good 15 attempts to get past it, maybe the sign up on Yahoo and Live is a lot easier.

    Really liked this post though, great insights into some major email clients.

    0
  • Marcus Tucker

    Tuesday, September 7th, 2010 17:35

    32

    Interesting reading, but I find the analysis rather incomplete because the most obvious questions is left unasked, let alone unanswered – i.e. what correlation is there between the email service(s) that the respondents actually use and the usability/location preferences that you have inferred from their clicks?

    This is very important because I would expect users of each service to prefer things to be where they usually are (i.e. when they use their webmail service), so your results may simply be reflecting these habits, and the relative market share of the email providers amongst your test subjects, rather than offering real insight into the best place for features to be located.

    0
    • Don

      Tuesday, June 14th, 2011 17:14

      47

      100% agree, HUGE flaw in the study. people’s preferences are obviously confounded by their current habits which results from the service they use. So this isn’t really testing people’s true a priori preferences but rather, just a popularity contest.

      0
  • S.Khan

    Tuesday, September 7th, 2010 12:04

    31

    I clearly am in the minority but I hate hotmail, its user interface and most of all the spam filter (non-existent?). I actually really like Yahoo because its got a very explorer kind of feel to it which makes it very easy. But yes, I agree, Gmail is the best.

    0
  • Jeff

    Friday, September 10th, 2010 18:17

    36

    I wholeheartedly agree. Gmail, by far is the most user friendly, and I have used all 3. At present, I use Hotmail (I belong to MS Technet) and Gmail (because I want to).

    0
  • kevin

    Monday, February 14th, 2011 21:54

    41

    You missed a couple important points that are reasons that I use yahoo.
    First of all, there is no real preview pane for gmail. This annoys me and is my main reason for not using it. The preview pane in hotmail is even better than the yahoo since I prefer a vertical pane rather than the horizontal pane offered by yahoo, but with hotmail, you can only see about 25 messages in the inbox before having to click to see the next page of messages. So if you have 200 facebook updates you literally have to select a page worth, delete, then the next page, select, delete. It’s a pain in the arse. If you have a bunch of messages you want to keep in between these, it gets even uglier. I’m not sure if this is the same on gmail or not. On yahoo, you can scroll and see every message in your inbox. If you have a thousand facebook messages, you can sort by sender, highlight the top one, CTRL and select the bottom one, then press the delete key and voila, they are all gone. This is so much more important than where I expect to find something when I click. After you use it a few times, you get used to that. If you have to manage a busy inbox, Yahoo is the way to go.

    0
    • pipi008

      Thursday, April 28th, 2011 10:02

      43

      kevin, apparently you are not a very advanced user. In Gmail you can VERY EASILY create a rule which automatically puts a label on all your Facebook messages and they don’t clutter your inbox. If you want to read them – go to your FB label. If you want to delete them – delete everything in your FB label. Really – it take 10 seconds to set up. If you have to manage a busy inbox – Gmail is the way to go! ;)

      0
    • Andrew

      Thursday, May 26th, 2011 11:04

      45

      I totally agree, the Preview Pane is one of the best features of Yahoo and Hotmail. However, now that Yahoo has stuffed up their awesome Mail Client with this new crappy bloated unreliable Beta version introduced in 2011, I am now looking for another Mail client. Yahoo was good, now it is total rubbish.

      0
  • G33K

    Saturday, April 9th, 2011 22:55

    42

    I personally think that gmail is better then all the rest. Mainly because it leads directly to your email inbox

    0
  • hi

    Wednesday, October 27th, 2010 02:16

    40

    I like hotmail it’s better than yahoo and gmail

    0
  • Mark

    Monday, September 20th, 2010 22:07

    37

    I prefer Gmail by far over the others. Yahoo! is a distant 3rd on my list. Why they can’t figure it out after all this time I’m not sure.

    0
    • zbob

      Thursday, October 14th, 2010 20:08

      38

      for me the yahoo interface is waaaay better …
      especially for business … i have been using google apps email for about a month now .. still frustrated. things that were easy in yahoo are painful in google
      – – tabbed inteface … great in yahoo .. does not exist in google. with tabs, i could have 3 emails in process, inbox, sent, and whatever i need at a single click. google can done one thing at a time.

      preview is nice in yahooo .. i could click through multiple emails in the in box .. know what they are and never loose track of where i am. google – one thing at a time.

      the conversation email is occasionally useful.. mostly its frustrating ..kinda tough to keep up with where i am .. especially when i may have 3 or 4 different conversations with the same contact .. all about the same event .. but since the subjects are different they are all different conversations.

      so .. maybe this will get better… maybe i will learn what everyone else is so excited about .. for now it seems like a big step back .. yahoo way better

      on spam.. google much better .. on calendar.. google is light years ahead.

      but on business email .. give me yahoo any day !

      my .02

      mbj

      0
  • Munif

    Sunday, February 26th, 2012 00:24

    56

    Gmail is the best and would win if they took another test. I was using gmail for more than 4 years and i did not have any problems.

    0
  • Cosmin

    Monday, January 2nd, 2012 21:26

    55

    Personally I like the Gmail interface because is simple and simple is always good and ergonomic! On the other hand, I also believe that having your own email server would make things easier for you.

    0
  • Augusto Malowda

    Thursday, November 17th, 2011 18:11

    53

    Gmail is the best one in my opinion.
    With the change of UI, Google are truly making it more easier to use!
    More friendly for older generations.

    0
  • Munim

    Thursday, November 3rd, 2011 22:22

    52

    for me gmail :
    is less complicated
    takes no time to load
    the new look makes it even more great
    identifies emails that are important and marks them with a yellow marker
    hasnt got any distracting banners; only text ads in one row so it doesnt take all the space
    has got 25mb attachment; more than hotmail and yahoo mail
    all the important and common stuff is bigger and good on the eye at the top like reply, forward, report spam etc once you select or open an email. if any of u guys havent seen the new 2011 look yet u better before saying anything else against gmail. by the wy i have used hotmail aswell and am still using it but not much and in my opinion it isnt that attractive or less complicated as gmail. yahoo mail is not an option ..:D

    0
  • Rafael Nunes

    Thursday, October 27th, 2011 23:55

    51

    Hey, that’s really the best test i’ve ever seen in my life, congratulations! really D:

    0
  • Darren

    Monday, September 5th, 2011 04:17

    49

    Do as I did, and stop reading this about half way as it starts to become heavily evident that it is biased solely towards Gmail.

    0
    • ^agree

      Tuesday, September 20th, 2011 10:09

      50

      Sobias towards Gmail. Ruins the point of this exercise

      0
  • mihai

    Wednesday, August 24th, 2011 20:57

    48

    For me, hotmail clearly wins, though i use both hotmail and yahoo, google is just too selfish for my liking.

    first, hotmail has the best spam filter. I three years since i have this adress, i virtually got only 5 spam messages, and their senders (the real ones) were added to block list as soon as i deleted the emails, unlike yahoo, where you must keep the emails in spam folder for about a month.

    second, hotmail gives you FREE Microsoft office online, and 2GB of storage for virtually everything (I even keep a website in there). this also makes you able to create .docx documents online without having to buy/install/crack/whatever an office installation. This is HUGE in comparison with gmail.

    Third, their web pages have the most powerful encryption available around, you can safely use your hotmail account for things such as bank accounts and stuff like that.

    0
  • Ashay

    Tuesday, May 31st, 2011 17:46

    46

    For me, GMail clearly wins for a number of reasons ranging from prompt email delivery to superb spam filter to straight-to-inbox view to simple, yet elegant interface. But the biggest of them all is the fact that GMail is the only one that allows me to use my email client (I use Apple Mail) for free. In other words, of the three, only GMail does not charge fort IMAP or POP! But then, I believe web based email browsing is outdated, and noobish!

    0
    • Jade

      Sunday, November 27th, 2011 21:21

      54

      The problem with Gmail, is that, if anyone were to hack it, they could easily delete all of your recovery information without needing a verification link for removal from your cell #, or recovery email. With Yahoo, it is the same thing. And if you were to contact customer care, they would not do anything as they are very hard to work with. Your chances are very slim when recovering a yahoo account. For Hotmail, it generates a lot of random spam. Some aren’t even spam. Though, it requires a verification to remove your recovery information so if a hacker were trying to remove your recovery information so you couldn’t get in, you would be able to anyways. Also on Hotmail, you can always empty the spam box by clicking “empty”. I’m siding with Hotmail. Don’t try mail.com or AOL mail. They have many glitches in their system and often you get mail very late. Also past mail fails to show up even though you have saved it in your inbox. Hope this helps!

      0
  • me

    Wednesday, May 18th, 2011 20:34

    44

    Gmail is fully SSL so much more secure when you’re in airports, etc. but I can’t believe it doesn’t serach thruogh attachments. On yahoo email. you can search for any word(s) and it will look through all your emails and the contents of your attachements.

    0
  • G33K

    Saturday, April 9th, 2011 22:55

    42

    I personally think that gmail is better then all the rest. Mainly because it leads directly to your email inbox

    0
  • kevin

    Monday, February 14th, 2011 21:54

    41

    You missed a couple important points that are reasons that I use yahoo.
    First of all, there is no real preview pane for gmail. This annoys me and is my main reason for not using it. The preview pane in hotmail is even better than the yahoo since I prefer a vertical pane rather than the horizontal pane offered by yahoo, but with hotmail, you can only see about 25 messages in the inbox before having to click to see the next page of messages. So if you have 200 facebook updates you literally have to select a page worth, delete, then the next page, select, delete. It’s a pain in the arse. If you have a bunch of messages you want to keep in between these, it gets even uglier. I’m not sure if this is the same on gmail or not. On yahoo, you can scroll and see every message in your inbox. If you have a thousand facebook messages, you can sort by sender, highlight the top one, CTRL and select the bottom one, then press the delete key and voila, they are all gone. This is so much more important than where I expect to find something when I click. After you use it a few times, you get used to that. If you have to manage a busy inbox, Yahoo is the way to go.

    0
    • pipi008

      Thursday, April 28th, 2011 10:02

      43

      kevin, apparently you are not a very advanced user. In Gmail you can VERY EASILY create a rule which automatically puts a label on all your Facebook messages and they don’t clutter your inbox. If you want to read them – go to your FB label. If you want to delete them – delete everything in your FB label. Really – it take 10 seconds to set up. If you have to manage a busy inbox – Gmail is the way to go! ;)

      0
    • Andrew

      Thursday, May 26th, 2011 11:04

      45

      I totally agree, the Preview Pane is one of the best features of Yahoo and Hotmail. However, now that Yahoo has stuffed up their awesome Mail Client with this new crappy bloated unreliable Beta version introduced in 2011, I am now looking for another Mail client. Yahoo was good, now it is total rubbish.

      0
  • hi

    Wednesday, October 27th, 2010 02:16

    40

    I like hotmail it’s better than yahoo and gmail

    0
  • Mark

    Monday, September 20th, 2010 22:07

    37

    I prefer Gmail by far over the others. Yahoo! is a distant 3rd on my list. Why they can’t figure it out after all this time I’m not sure.

    0
    • zbob

      Thursday, October 14th, 2010 20:08

      38

      for me the yahoo interface is waaaay better …
      especially for business … i have been using google apps email for about a month now .. still frustrated. things that were easy in yahoo are painful in google
      – – tabbed inteface … great in yahoo .. does not exist in google. with tabs, i could have 3 emails in process, inbox, sent, and whatever i need at a single click. google can done one thing at a time.

      preview is nice in yahooo .. i could click through multiple emails in the in box .. know what they are and never loose track of where i am. google – one thing at a time.

      the conversation email is occasionally useful.. mostly its frustrating ..kinda tough to keep up with where i am .. especially when i may have 3 or 4 different conversations with the same contact .. all about the same event .. but since the subjects are different they are all different conversations.

      so .. maybe this will get better… maybe i will learn what everyone else is so excited about .. for now it seems like a big step back .. yahoo way better

      on spam.. google much better .. on calendar.. google is light years ahead.

      but on business email .. give me yahoo any day !

      my .02

      mbj

      0
  • Jeff

    Friday, September 10th, 2010 18:17

    36

    I wholeheartedly agree. Gmail, by far is the most user friendly, and I have used all 3. At present, I use Hotmail (I belong to MS Technet) and Gmail (because I want to).

    0
  • Joanne

    Wednesday, September 8th, 2010 10:08

    34

    It would be interesting to see how users in China respond to a similar poll.

    0
  • Ben

    Tuesday, September 7th, 2010 17:52

    33

    Another intersting test may have been to evaluate each of the sign up processes. I use google for everything at the moment but set up a new gmail account for my new website, the sign up process was a ruddy nightmare simply because of the horrible Captcha spam filter they use, it took me a good 15 attempts to get past it, maybe the sign up on Yahoo and Live is a lot easier.

    Really liked this post though, great insights into some major email clients.

    0
  • Marcus Tucker

    Tuesday, September 7th, 2010 17:35

    32

    Interesting reading, but I find the analysis rather incomplete because the most obvious questions is left unasked, let alone unanswered – i.e. what correlation is there between the email service(s) that the respondents actually use and the usability/location preferences that you have inferred from their clicks?

    This is very important because I would expect users of each service to prefer things to be where they usually are (i.e. when they use their webmail service), so your results may simply be reflecting these habits, and the relative market share of the email providers amongst your test subjects, rather than offering real insight into the best place for features to be located.

    0
    • Don

      Tuesday, June 14th, 2011 17:14

      47

      100% agree, HUGE flaw in the study. people’s preferences are obviously confounded by their current habits which results from the service they use. So this isn’t really testing people’s true a priori preferences but rather, just a popularity contest.

      0
  • S.Khan

    Tuesday, September 7th, 2010 12:04

    31

    I clearly am in the minority but I hate hotmail, its user interface and most of all the spam filter (non-existent?). I actually really like Yahoo because its got a very explorer kind of feel to it which makes it very easy. But yes, I agree, Gmail is the best.

    0
  • JC Johnston

    Monday, September 6th, 2010 10:07

    21

    I have accounts with all 3 mail services. I prefer Gmail over all the services, Even my premium and work accounts. I can not stand Yahoo interface and it annoys me every time I have to log in. Hotmail is not so bad, but I rather be using Gmail and I always recommend them to my clients who need a free email for buisness.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:30

      22

      Gosh, I feel for you have to deal with all three – especially Yahoo – the whole interface just depresses me. It’s interesting to get your insight based on using all three, glad to see you came to the same conclusion too.

      Thanks for your comment.

      0
  • TingTingBen

    Monday, September 6th, 2010 09:35

    20

    Very nice !
    But aren’t all these results influenced by years of Gmail using ?
    I think all these questions should be asked to new internet users whose have never hear about Gmail, Hotmail and Yahoo UI.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:35

      23

      Yeah, it is affected by peoples learned behaviors, but those learned habits are important, and often what we need to design to. I think there is a good spread of users on this site from Yahoo, Hotmail and Gmail so hopefully we avoid most of those biases.

      0
  • Fatih

    Sunday, September 5th, 2010 02:08

    15

    I use GMail all the time. I also got a passport account earlier so I use my email in MSN Messenger too.

    – I love Gmail’s interface, Labs, Chat, Options and Efficiency!
    – I hate hotmail. I (personally) think, everything that comes from microsoft sucks! Lol. My Opinion!
    – Neutral to Yahoo! (I didn’t use it much, so I can’t rate/hate it :P)

    0
  • Theo

    Saturday, September 4th, 2010 21:58

    14

    That gmail wins was clear to me, just a bit disappointed by yahoo!, however interesting test.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:54

      24

      Thanks for the comment, glad you liked the test.

      0
  • Mariano

    Saturday, September 4th, 2010 19:55

    13

    Jacob, Hotmail is now sending you to a landing page too. Not just Yahoo. When I log into my account, it sends me to the Windows Live Home and then I have to click “Go to Inbox” or “Hotmail”.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 04:55

      25

      Thanks for letting me know. It seems the roll out lots of different ‘features’ to different segments at different times. I still don’t have a landing page – interesting to know that they’ve put one in for you. Based on these results I don’t think it will be a popular choice.

      0
  • chaitanya varma

    Saturday, September 4th, 2010 19:43

    12

    wonderful analysis!

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:08

      29

      Thanks very much! Glad you liked it.

      0
  • Terri Lockerridge

    Saturday, September 4th, 2010 07:48

    11

    1st Webdesigner asks:
    How does this compare with your actual experience of these different webmail services?
    When I started trying to learn how to access and use the internet and how to utilize email.. Hotmail was the first I tried to use. As with any Microsoft product offering.. it is the hardest and most difficult place for a newbie to try!
    They assume that the user is already knowledgeable about things and just do not try to be user friendly in a simple and straightforward way. I would try to send a simple email and I could not get through all the steps I had to work out before I was totally confused and would give up.
    Then I tried Yahoo mail and found it to be simple and so very easy to use with only a very few steps to be successful. I used their services until they did the change over to the new platform and I left them flat after they invaded my privacy in a lot of small ways, and they changed the mail site to be so complicated with so many links on it that you could not move your cursor an inch without a popup proclaiming to be my calendar or the ever present and annoyances of trying to get me to buy their premium service which I do not need. Too much garbage and you can’t find a darn thing that you use everyday easily. Too much in too small a space, invading privacy, and not listening when we give them feedback on how to improve things really turned me off of them.
    So I went to Gmail. I find it to be simple, with a chat built right in directly to it, easy to understand and access all my information, plenty of storage, GoogleTalk for free telephone services, easy to connect to and download a stored item from their website for hosting and editing documents, and they do not try to cram so many ads onto the page that I have to be really intense on where I click to avoid inadvertently clicking onto an ad.

    I have a couple of tips for any free email service out there.

    1. Keep in mind that not all people know how to speak computerlish and to keep it simple without many steps to accomplish a task.
    2. Keep your customer service well staffed and respond to any problems that are reported to them quickly… and no matter how hard it is to help someone, help them till a technical problem is resolved, stay with someone until they learn how to use the service and do not make them feel as if they are a pest, and for the love of pete.. if your going to offer the html option to create an email.. then give us some help on using it to make a great email for any reason! I have yet to find any email service who will give you any real help on this issue. I have the html capability in Gmail.. but there is no, and I mean no really simple, useful help on using it!

    Are there any other sites you’d like see compared in the future?
    I would like to see how many people are using Outlook Express and if they find it to be difficult to use as in setting up other email services as well as Hotmail into it. I do not use it because I can’t get it to accept forwarded mail or any mail other than Hotmail.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:06

      28

      Hi Terri, thanks for your comment! It’s really interesting to get input from users such as yourself, and you are right, most companies forget who their core audience is, and just design for themselves.

      That is the point of usability testing, so you can get real information from real users.

      Interesting you talk about Outlook Express. I had kind of forgotten it still existed – is it even in Windows 7? I guess I saw this from my own perspective, as I use Macs 99% of the time, and use webmail 99% of the time. I use mail.app on my iPhone, which seems to do everything fairly well, and Thunderbird occasionally when I use desktop mail.

      Not sure why anyone would even touch outlook with these options around.

      0
  • Warren

    Friday, September 3rd, 2010 18:31

    10

    Gmail is by far the best, however it does lack a beautiful interface.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:00

      27

      I think you sum up Google pretty well with that comment! Hopefully they will learn that lesson soon.

      0
  • Marcelo

    Friday, September 3rd, 2010 19:15

    8

    Amazing results, Jacob! However we must consider that the people who took the test are probably web designers, and Gmail is the preferred mail for us all, so we already know it.

    It would be interesting a similar test between all kind of people.

    Anyway, congrats!
    Marcelo.

    0
    • Jacob Creech

      Tuesday, September 7th, 2010 05:10

      30

      Usually when we publish a test we aim it at the specific market for the site. In this case I think it’s OK to test a technical audience, because we are all email users anyway. I’m not sure that the majority of readers here are Gmail users though – it seems there is a fairly even distribution of Hotmail, Gmail and Yahoo users.

      Anyway, thanks for your comment!

      0
  • Mark

    Friday, September 3rd, 2010 17:57

    7

    It’s pretty sad that Yahoo! has had YEARS to get webmail right (and a big head start on Gmail) and they STILL haven’t really gotten it figured out. What the heck are they working on over there?

    0
    • Jacob Creech

      Sunday, September 5th, 2010 02:14

      18

      Haha, I’ve been asking myself the same question for a while now!

      0
  • Mark S

    Friday, September 3rd, 2010 17:46

    6

    So lets get this completely correct, the most usable site was hotmail, the most popular was gmail. In the tests (part 2) : yahoo 1, hotmail 3, gmail 2.

    I know which I’d prefer to use :P

    0
    • Jacob Creech

      Sunday, September 5th, 2010 02:13

      17

      Well, except for the fact that replying on the same page, and banner ads vs text ads also contribute to usability, and Gmail was the only service offering those features. Peoples response to a UI is also an important factor here, and again Gmail was in the lead.

      0
      • solcroft

        Sunday, September 5th, 2010 21:58

        19

        Except that we know that people *cough*fanboys*cough* tend to offer answers skewed by their overall perception when asked a specific question about a specific area of their webmail service.

        0
        • Jacob Creech

          Tuesday, September 7th, 2010 04:59

          26

          Most of these questions it’s hard to be a fanboy, it just depends what you’re used to, what you like, and what works well.

          0
          • solcroft

            Wednesday, September 8th, 2010 15:52

            35

            Um, it’s exceedingly easy to be a fanboy for that one particular question. You could simply like *insert some other features or aspects here* and think every facet of the service is unmatched.

            When you ask a question along the lines of “which do you like best”, you’re obviously throwing objectivity out of the window.

            0
      • brad

        Wednesday, October 20th, 2010 21:50

        39

        I’m not sure I would consider having any kind of ad, whether it be banner or text , an assett. I consider them a nuisance.. I still use OE because of the simplicity. I hate to give it up. I love hotmail because it actually has spam filters that work.. Verizon certainly has none.. I do not use a phone for play time like internet or e-mails. I am a contractor and am on the phone all day ; so I only read e-mails and surf the internet on a real computer. I would like to see my 17 yr old daughter start using an e-mail, so I was reading this article with interest. She only does texting and facebook, occassionally my space. Thanks for the info.

        0
  • Jay

    Friday, September 3rd, 2010 17:15

    5

    You were testing the old interface of Yahoo Mail. Why not test the new AJAXy interface?

    0
    • Jacob Creech

      Sunday, September 5th, 2010 02:09

      16

      That was the only Yahoo interface I had access too. I checked with a number of people I know who still use Yahoo, and they said that is the screen they see too.

      In fact, I know all of these different providers are rolling out updates of one kind or another, but these are what were available, and hence what I tested.

      0
  • arun

    Friday, September 3rd, 2010 12:51

    2

    Interesting results, but we all know Gmail would win, I also took the test, and now results are similar to what i expected
    Gmail is way ahead in terms of usability that other email services
    :)

    0
    • Swamykant

      Friday, September 3rd, 2010 14:57

      3

      You are right .. Gmail is the email services – It is Simple & Smart.. However needs some more improvements :)

      0
      • Ram

        Friday, September 3rd, 2010 15:51

        4

        Software always needs improvements. Point is, it’s a lot more intuitive than the competitors. Yahoo! lost it when they gave us 10 MB mailboxes and wanted people to pay a ridiculous fees for 100MB mailboxes. Once GMail came they got blown out of the water, and deservedly so. It’s a surprise Yahoo are still around. I’d hope they went bankrupt soon.

        I don’t care for the text ads, so GMail works for me, but Yahoo were just greedy beyond anything and they deserved to no longer be the top email provider.

        0
    • Jacob Creech

      Friday, September 3rd, 2010 15:34

      9

      Thanks guys; I must say I wasn’t surprised that Gmail did well (and Hotmail did reasonably too), but I was surprised how far behind Yahoo! is, especially considering how much of the market they control still.

      It was really interesting to observe different behaviors from people as well; I’m wondering if anyone has any suggestions for what we should test next?

      0
  • Jim

    Friday, September 3rd, 2010 12:23

    1

    Nice analyzing..

    0

Comments are closed.

54.161.175.231 - unknown - unknown - US