Email UI Faceoff: Gmail vs Hotmail vs Yahoo!

Posted in Web Design • Posted on 31 Comments

I recently wrote about 9 well designed, usable websites, and what makes them great. Gmail, as one of my personal favorites, and a site many people love featured in the list, because I think it really is a truly useful, usable website. I was quite surprised to see in the comments a number of people saying that they simply couldn’t agree with Gmail as a pick.

This leads to the issue of finding a subjective method for deciding which webmail system is best. How? Read on to find out.

Making a decision: which is best?

Of course, all of our views are subjective, and it seems the fairest way to decide is to run an impartial test on between some of the most popular webmail services.

The three webmail services that are well ahead of the pack in terms of users are of course Gmail, Hotmail and Yahoo! Mail, and so today we will pit them against each other in a usability battle.

Gmail, Yahoo! and Hotmail UI

UI faceoff - who will win?

Rather than colouring the results with our own biases, this article will be broken into two parts: Round one is the testing process, and getting feedback from users, and round two will be the summary of results and what we think should or could be done differently.

So see below for the testing process, the test itself, and how it all works. Hopefully you can learn a little more about the usability testing and how it can help you too.

Part One: The Testing Process

The testing process is very straight forward. There is a brief test at video.intuitionhq.com/email-face-off which should take around 3 minutes to complete – you can read about the testing process, or just head on over to take the test now.

You will be presented with a series of tasks (17 total in this example), and all you need to do is click where ever you think is the right place. For example, you might be asked with the following image, ‘Where do you need to click to sign up’:

Click to sign up - choose the big green sign up button

I pick the big green button

This is obviously a very straight forward example, but if you imagine when you have a site or UI with important features that people can’t find, or aren’t in their expected place things might not be as usable as they should be.

Another example might be ‘Click where you’d expect to return to the home page’ with a grid to click on:

A grid for testing perceptions and reality

A grid for testing perceptions and reality

This grid can then be overlaid over a design to see if the perception and reality of the design are the same thing:

A grid overlayed over design to compare ideas with reality

The overlayed grid for comparing ideas with reality

In this example, most users would probably click in the top right hand area, either aiming at the logo (which commonly links back to the home page) or the home tab. This is a common convention these days, and it generally pays to follow it.

Not following conventions means people have to relearn. Imagine someone swapped the position of the shift and delete keys – you would be frustrated that after learned the original location (the traditional location of the enter key) that someone changed things around. Consider that when creating your sites and contemplating usability.

Of course, every great site doesn’t have to follow conventions, but there should be a good reason for ignoring them. Conventions and expected behaviour generally do help improve usability.

The results:

At the end of the test, a nice heatmap is generated (for the test creator to see) showing where people clicked, and how long it took them (not shown here), so you can judge what is or isn’t working properly, as per the example below:

An example of a heatmap with results

The final product - a heatmap with results

In this example, you can see 87% of people clicked the correct button, and a few random clicks as might be expected as this is a demo test page we like to show people, but all in all most people get the exact right place. Of course, if we’d asked people how to preview a test and they clicked publish instead, we might need to make some changes.

So, what next?

Head on over to video.intuitionhq.com/email-face-off and take the test. Help us figure out just which webmail service is the best, most usable among them. Be sure to leave your thoughts in the comments about the different services and testing process too.

We’ll be writing about the results shortly, and how we think these sites can improve, and what they’ve done right, so watch this space.

And if there are any other categories of sites you’d like us to see in a usability face off, be sure to let us know in the comments.

5 Written ArticlesWebsite

Jacob is a usability geek at IntuitionHQ.com - a new, quick, easy to use usability service from Boost New Media. When he's not indulging his passion for the internet, he is probably bloging about China and Chinese learning (having spent three years there) at Sinomatter.com. You can find him on twitter talking about usability @intuitionhq.

31 Comments Best Comments First
  • Keiron Lowe

    Thursday, August 12th, 2010 12:32

    1

    Great Idea!
    I personally use gmail and I think it is so much easier to use than Hotmail, I personally haven’t used Yahoo Mail, mainly because I kind of feel like gmail is the industry standard mail service.

    0
  • Arulrajan

    Thursday, August 12th, 2010 19:26

    10

    Did the calendar also should be part of the email client or It can be a separate web site (eg., Google Calendar)

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:34

      18

      I think in most cases they want to integrate the services together so people will use them more.

      However, the point of that task is just to see whether links to common services are laid out in a logical manner.

      Cheers.

      0
  • Joe Barstow

    Thursday, August 12th, 2010 20:00

    13

    Very insightful. Loved your approach! Refreshing to see something new, and gave me a lot to think about. Thanks!

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:35

      19

      Thanks a lot Joe, I really appreciate it. Glad you liked the article!

      0
  • Jay

    Friday, August 13th, 2010 01:01

    9

    I wish all of the web email clients would improve their search capability. One of the main reasons I love my desktop email client better is the searching.

    0
  • Matthew Corner

    Thursday, August 12th, 2010 12:09

    8

    Shame the GMail interface improved imo just a few days before this article. I personally love the new tidy changes, and am a gmail person through and through.

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:24

      15

      In fact, both gmail and hotmail are currently rolling out interface changes – although only to a limited number of accounts. Hotmail seems to have had a number of issues with their rollout though.

      I plan to cover these changes in the second part of this article though. Interesting to hear your thoughts on the new gmail design though, a number of people I’ve talked to thinks it looks a little more clunky now.

      Thanks for the feedback!

      0
  • machbio

    Thursday, August 12th, 2010 14:58

    3

    great article.. i use thunderbird

    0
    • Jacob Creech

      Thursday, August 12th, 2010 19:59

      12

      Haha, not a bad choice. Thunderbird is my preferred desktop client.

      0
  • Mark

    Thursday, August 12th, 2010 12:49

    2

    Just took the Hotmail vs Yahoo vs Gmail test… not sure what sort of result you’re aiming for but I felt the questions may be worded better…

    They generally start “Where would you click…”, though I got the impression the test is really asking “Where should you click…”
    For example… on yahoo, I was able to find and click the calander button… but It wasn’t very clear or easy to find, nor was it the first place I looked for it. Does the test know I took some time or just know that I clicked the right area?

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:22

      14

      Hi Mark,

      Thanks for your comment – the test records both the amount of time taken to click, and the location of the click.

      If someone clicked in the right place, but it took them a long time to do it, it obviously isn’t a desirable situation. By recording both metrics, it’s easier to see when a problem occurs, and to be able to solve it.

      Cheers.

      0
  • peter

    Thursday, August 12th, 2010 16:02

    4

    no bias..yer right, you show a huge screenshot of Yahoo mail and it looks big and empty (undesirable) and just a partial screenshot of gmail and hotmail

    0
    • Jacob Creech

      Thursday, August 12th, 2010 19:53

      11

      Hi Peter,

      I assume you are talking about the screenshot in the test (task 4). That screenshot is just to show the full content of each email service – it just happens that in Yahoo mail, it seems to take up the full height of your screen no matter the what resolution you are at, whereas with the others, supposing your inbox isn’t full of mail, only take up part of the screen area.

      You’ll also notice, the screenshot is one of 17 questions in the testing process, and it won’t be that one result which decides which of the email services is most usable.

      Hopefully that will help allay your fears. If you have something to say about the usability of Yahoo vs the other two, I’d love to hear that as well.

      Cheers.

      0
  • qitree

    Thursday, August 12th, 2010 16:30

    5

    i love gmail……great!!!!

    0
  • Rolf Moczarski

    Thursday, August 12th, 2010 17:09

    7

    Great stuff, i like the googlemail UI, but i also use Thunderbird and i think im just one of the rising number of Thunderbird-users.

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:32

      17

      Thunderbird has certainly been improving at a great rate. Maybe we should do some kind of test on popular desktop mail clients as well.

      Thanks for your comment.

      0
  • heathquinn

    Thursday, August 12th, 2010 16:43

    6

    Test needs fewer interface captures – if I’m familiar with an interface, I’ll respond more quickly – the interface has already trained me – the portion of the test that uses captures is a popularity contest except for the small percent of users who’ve used none of the three interfaces. Interface captures are ok if the question is about which do you like better – in other words, if a question is transparently about popularity. Maybe asking which interface user is most familiar with & discounting responses that favor that interface would right the built-in bias factor that captures bring to the test.

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:31

      16

      Thanks for your comment. You are right about people being familiar with an interface, but there is research showing that by providing a mixed array of designs, people tend to forget those interaction biases.

      Also, as the tasks measuring peoples interactions with a current design are only part of the test. The results will look at the test as a whole, and hopefully can draw some good conclusions.

      Cheers.

      0
  • Umair

    Saturday, October 22nd, 2011 03:50

    27

    love gmail when i used it :P but i am currently using hotmail & kinda used to it but i,ll surely switch to gmail soon

    0
  • Xend

    Saturday, November 20th, 2010 10:09

    26

    I strongly believe gmail is a non-starter because it seems to be the only email app written in the last 25 years that doesn’t support preview pane. Preview panes are the practical, efficient solution to reading emails. The standard gmail method is a huge waste of time. Do the math folks – minutes, hours, days, years of your life reading thousands of emails. Google’s inefficient UI is nickle and dimeing you out of a good chunk of your life. Google gmail recently introduced a “pop-up” preview. Being that it’s a pop-up means that it’s a pain but not to be confused with a pane. Not even close. I use yahoo and google email regularly (plus others) for various reasons but I only use gmail because I have to. The UI wastes my time and feels tedious. gmail users don’t know what they’re missing. Stick with yahoo if you value your time.

    0
    • Gordon

      Tuesday, March 20th, 2012 05:09

      31

      WTF are you talking about, you obviously haven’t used Gmail in the last few months, there is a preview pane – either vertical or horizontal. Know what you are talking about before spreading bs.

      0
  • Umair

    Saturday, October 22nd, 2011 03:51

    28

    love gmail :-)

    0
  • Vitaly

    Monday, November 7th, 2011 12:56

    29

    I really liked the new version…

    btw, the heigh of each row might be seen high for you, if this happens just change from the options.

    0
  • Fazil

    Sunday, November 13th, 2011 02:42

    30

    the latest version of gmail is the best and the awesomest ever and it numbs the user interface of hotmail and yahoo!!

    0
  • Thejesh15

    Saturday, November 20th, 2010 05:16

    25

    i think gmail and hot mail is the best sites for ever

    0
  • Gopinath Sundharam

    Friday, September 3rd, 2010 04:26

    24

    Gmail. The Best.
    Yahoo. The Next.
    Hotmail. The Worst.

    0
  • Terri Lockerridge

    Friday, August 13th, 2010 08:31

    20

    I prefer any other email to Hotmail. That has to be the most confusing and nonuser friendly mail that I have ever encountered on the net. So many choices and none of them lead to anywhere directly.. typical Microsoft type of thing.

    I used Yahoo mail for years until they did their platform change over and they started invading my privacy. They want you to buy a premium mail service from them and it is not worth it at all… the free mail is so hard to navigate since they put all that crap on the page .. you can’t find what your looking for any of the time.. even if it is the mail you use most of the time. They move stuff around and are always throwing ads to pressure you into the premium service.. pooey on them!

    I went to Gmail from Yahoo.. and love the chat they have .. l like the simplicity of the entire thing. Too many graphics on Yahoo, and Hotmail. The only thing I dislike about the service is you can’t get them to give you the tutorial on how to use the html option they provide to compose a custom email with images. I have not been able to find it anyway.. and when trying to use Word to compose and then merge.. it fails every time.

    Of the three .. I like Gmail the best. I am in agreement with you totally about it.

    0
    • Jacob Creech

      Tuesday, August 17th, 2010 06:38

      22

      Thanks for your comment Terri – it sounds like you’re in a good position to judge having had extensive experience of the three.

      You are right, none are perfect, but so far in my experience Gmail is well ahead of the others.

      That said, I’m going through the results of the test now, and there are a few interesting results in there. Stay tuned for that in the next week or so.

      0
  • Jarkko

    Friday, August 13th, 2010 10:27

    21

    I just took the test, and even if I’ve used Gmail for years now, I still managed to get lost in it and not find what I was looking for immediately. I think the other quite similar interfaces confused me. They all look quite dull and could probably be made more easy to navigate. Now that I’m trying to think back to the sites, the biggest differences I recall are the logos. I think Hotmail has some kind of slight gradient coloring in it too.

    I don’t know if Gmail has the clearest look, but I prefer it solely because I’m used to it.

    0
    • Jacob Creech

      Tuesday, August 17th, 2010 06:43

      23

      Thanks for your comment Jarkko. It’s interesting to see when you put the layouts side by side, and see the similarities and differences between them.

      I think part of it is that people get used to using a certain service, but another side is that some designs are just more intuitive than others.

      Look out for the results of our tests coming soon though. I think they are pretty interesting.

      0
  • Fazil

    Sunday, November 13th, 2011 02:42

    30

    the latest version of gmail is the best and the awesomest ever and it numbs the user interface of hotmail and yahoo!!

    0
  • Vitaly

    Monday, November 7th, 2011 12:56

    29

    I really liked the new version…

    btw, the heigh of each row might be seen high for you, if this happens just change from the options.

    0
  • Umair

    Saturday, October 22nd, 2011 03:51

    28

    love gmail :-)

    0
  • Umair

    Saturday, October 22nd, 2011 03:50

    27

    love gmail when i used it :P but i am currently using hotmail & kinda used to it but i,ll surely switch to gmail soon

    0
  • Xend

    Saturday, November 20th, 2010 10:09

    26

    I strongly believe gmail is a non-starter because it seems to be the only email app written in the last 25 years that doesn’t support preview pane. Preview panes are the practical, efficient solution to reading emails. The standard gmail method is a huge waste of time. Do the math folks – minutes, hours, days, years of your life reading thousands of emails. Google’s inefficient UI is nickle and dimeing you out of a good chunk of your life. Google gmail recently introduced a “pop-up” preview. Being that it’s a pop-up means that it’s a pain but not to be confused with a pane. Not even close. I use yahoo and google email regularly (plus others) for various reasons but I only use gmail because I have to. The UI wastes my time and feels tedious. gmail users don’t know what they’re missing. Stick with yahoo if you value your time.

    0
    • Gordon

      Tuesday, March 20th, 2012 05:09

      31

      WTF are you talking about, you obviously haven’t used Gmail in the last few months, there is a preview pane – either vertical or horizontal. Know what you are talking about before spreading bs.

      0
  • Thejesh15

    Saturday, November 20th, 2010 05:16

    25

    i think gmail and hot mail is the best sites for ever

    0
  • Gopinath Sundharam

    Friday, September 3rd, 2010 04:26

    24

    Gmail. The Best.
    Yahoo. The Next.
    Hotmail. The Worst.

    0
  • Jarkko

    Friday, August 13th, 2010 10:27

    21

    I just took the test, and even if I’ve used Gmail for years now, I still managed to get lost in it and not find what I was looking for immediately. I think the other quite similar interfaces confused me. They all look quite dull and could probably be made more easy to navigate. Now that I’m trying to think back to the sites, the biggest differences I recall are the logos. I think Hotmail has some kind of slight gradient coloring in it too.

    I don’t know if Gmail has the clearest look, but I prefer it solely because I’m used to it.

    0
    • Jacob Creech

      Tuesday, August 17th, 2010 06:43

      23

      Thanks for your comment Jarkko. It’s interesting to see when you put the layouts side by side, and see the similarities and differences between them.

      I think part of it is that people get used to using a certain service, but another side is that some designs are just more intuitive than others.

      Look out for the results of our tests coming soon though. I think they are pretty interesting.

      0
  • Terri Lockerridge

    Friday, August 13th, 2010 08:31

    20

    I prefer any other email to Hotmail. That has to be the most confusing and nonuser friendly mail that I have ever encountered on the net. So many choices and none of them lead to anywhere directly.. typical Microsoft type of thing.

    I used Yahoo mail for years until they did their platform change over and they started invading my privacy. They want you to buy a premium mail service from them and it is not worth it at all… the free mail is so hard to navigate since they put all that crap on the page .. you can’t find what your looking for any of the time.. even if it is the mail you use most of the time. They move stuff around and are always throwing ads to pressure you into the premium service.. pooey on them!

    I went to Gmail from Yahoo.. and love the chat they have .. l like the simplicity of the entire thing. Too many graphics on Yahoo, and Hotmail. The only thing I dislike about the service is you can’t get them to give you the tutorial on how to use the html option they provide to compose a custom email with images. I have not been able to find it anyway.. and when trying to use Word to compose and then merge.. it fails every time.

    Of the three .. I like Gmail the best. I am in agreement with you totally about it.

    0
    • Jacob Creech

      Tuesday, August 17th, 2010 06:38

      22

      Thanks for your comment Terri – it sounds like you’re in a good position to judge having had extensive experience of the three.

      You are right, none are perfect, but so far in my experience Gmail is well ahead of the others.

      That said, I’m going through the results of the test now, and there are a few interesting results in there. Stay tuned for that in the next week or so.

      0
  • Joe Barstow

    Thursday, August 12th, 2010 20:00

    13

    Very insightful. Loved your approach! Refreshing to see something new, and gave me a lot to think about. Thanks!

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:35

      19

      Thanks a lot Joe, I really appreciate it. Glad you liked the article!

      0
  • Arulrajan

    Thursday, August 12th, 2010 19:26

    10

    Did the calendar also should be part of the email client or It can be a separate web site (eg., Google Calendar)

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:34

      18

      I think in most cases they want to integrate the services together so people will use them more.

      However, the point of that task is just to see whether links to common services are laid out in a logical manner.

      Cheers.

      0
  • Jay

    Friday, August 13th, 2010 01:01

    9

    I wish all of the web email clients would improve their search capability. One of the main reasons I love my desktop email client better is the searching.

    0
  • Matthew Corner

    Thursday, August 12th, 2010 12:09

    8

    Shame the GMail interface improved imo just a few days before this article. I personally love the new tidy changes, and am a gmail person through and through.

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:24

      15

      In fact, both gmail and hotmail are currently rolling out interface changes – although only to a limited number of accounts. Hotmail seems to have had a number of issues with their rollout though.

      I plan to cover these changes in the second part of this article though. Interesting to hear your thoughts on the new gmail design though, a number of people I’ve talked to thinks it looks a little more clunky now.

      Thanks for the feedback!

      0
  • Rolf Moczarski

    Thursday, August 12th, 2010 17:09

    7

    Great stuff, i like the googlemail UI, but i also use Thunderbird and i think im just one of the rising number of Thunderbird-users.

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:32

      17

      Thunderbird has certainly been improving at a great rate. Maybe we should do some kind of test on popular desktop mail clients as well.

      Thanks for your comment.

      0
  • heathquinn

    Thursday, August 12th, 2010 16:43

    6

    Test needs fewer interface captures – if I’m familiar with an interface, I’ll respond more quickly – the interface has already trained me – the portion of the test that uses captures is a popularity contest except for the small percent of users who’ve used none of the three interfaces. Interface captures are ok if the question is about which do you like better – in other words, if a question is transparently about popularity. Maybe asking which interface user is most familiar with & discounting responses that favor that interface would right the built-in bias factor that captures bring to the test.

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:31

      16

      Thanks for your comment. You are right about people being familiar with an interface, but there is research showing that by providing a mixed array of designs, people tend to forget those interaction biases.

      Also, as the tasks measuring peoples interactions with a current design are only part of the test. The results will look at the test as a whole, and hopefully can draw some good conclusions.

      Cheers.

      0
  • qitree

    Thursday, August 12th, 2010 16:30

    5

    i love gmail……great!!!!

    0
  • peter

    Thursday, August 12th, 2010 16:02

    4

    no bias..yer right, you show a huge screenshot of Yahoo mail and it looks big and empty (undesirable) and just a partial screenshot of gmail and hotmail

    0
    • Jacob Creech

      Thursday, August 12th, 2010 19:53

      11

      Hi Peter,

      I assume you are talking about the screenshot in the test (task 4). That screenshot is just to show the full content of each email service – it just happens that in Yahoo mail, it seems to take up the full height of your screen no matter the what resolution you are at, whereas with the others, supposing your inbox isn’t full of mail, only take up part of the screen area.

      You’ll also notice, the screenshot is one of 17 questions in the testing process, and it won’t be that one result which decides which of the email services is most usable.

      Hopefully that will help allay your fears. If you have something to say about the usability of Yahoo vs the other two, I’d love to hear that as well.

      Cheers.

      0
  • machbio

    Thursday, August 12th, 2010 14:58

    3

    great article.. i use thunderbird

    0
    • Jacob Creech

      Thursday, August 12th, 2010 19:59

      12

      Haha, not a bad choice. Thunderbird is my preferred desktop client.

      0
  • Mark

    Thursday, August 12th, 2010 12:49

    2

    Just took the Hotmail vs Yahoo vs Gmail test… not sure what sort of result you’re aiming for but I felt the questions may be worded better…

    They generally start “Where would you click…”, though I got the impression the test is really asking “Where should you click…”
    For example… on yahoo, I was able to find and click the calander button… but It wasn’t very clear or easy to find, nor was it the first place I looked for it. Does the test know I took some time or just know that I clicked the right area?

    0
    • Jacob Creech

      Friday, August 13th, 2010 00:22

      14

      Hi Mark,

      Thanks for your comment – the test records both the amount of time taken to click, and the location of the click.

      If someone clicked in the right place, but it took them a long time to do it, it obviously isn’t a desirable situation. By recording both metrics, it’s easier to see when a problem occurs, and to be able to solve it.

      Cheers.

      0
  • Keiron Lowe

    Thursday, August 12th, 2010 12:32

    1

    Great Idea!
    I personally use gmail and I think it is so much easier to use than Hotmail, I personally haven’t used Yahoo Mail, mainly because I kind of feel like gmail is the industry standard mail service.

    0

Comments are closed.

54.161.133.166 - unknown - unknown - US