Email UI Faceoff: Gmail vs Hotmail vs Yahoo!

I recently wrote about 9 well designed, usable websites, and what makes them great. Gmail, as one of my personal favorites, and a site many people love featured in the list, because I think it really is a truly useful, usable website. I was quite surprised to see in the comments a number of people saying that they simply couldn’t agree with Gmail as a pick.

This leads to the issue of finding a subjective method for deciding which webmail system is best. How? Read on to find out.

Making a decision: which is best?

Of course, all of our views are subjective, and it seems the fairest way to decide is to run an impartial test on between some of the most popular webmail services.

The three webmail services that are well ahead of the pack in terms of users are of course Gmail, Hotmail and Yahoo! Mail, and so today we will pit them against each other in a usability battle.

Gmail, Yahoo! and Hotmail UI

UI faceoff - who will win?

Rather than colouring the results with our own biases, this article will be broken into two parts: Round one is the testing process, and getting feedback from users, and round two will be the summary of results and what we think should or could be done differently.

So see below for the testing process, the test itself, and how it all works. Hopefully you can learn a little more about the usability testing and how it can help you too.

Part One: The Testing Process

The testing process is very straight forward. There is a brief test at video.intuitionhq.com/email-face-off which should take around 3 minutes to complete – you can read about the testing process, or just head on over to take the test now.

You will be presented with a series of tasks (17 total in this example), and all you need to do is click where ever you think is the right place. For example, you might be asked with the following image, ‘Where do you need to click to sign up’:

Click to sign up - choose the big green sign up button

I pick the big green button

This is obviously a very straight forward example, but if you imagine when you have a site or UI with important features that people can’t find, or aren’t in their expected place things might not be as usable as they should be.

Another example might be ‘Click where you’d expect to return to the home page’ with a grid to click on:

A grid for testing perceptions and reality

A grid for testing perceptions and reality

This grid can then be overlaid over a design to see if the perception and reality of the design are the same thing:

A grid overlayed over design to compare ideas with reality

The overlayed grid for comparing ideas with reality

In this example, most users would probably click in the top right hand area, either aiming at the logo (which commonly links back to the home page) or the home tab. This is a common convention these days, and it generally pays to follow it.

Not following conventions means people have to relearn. Imagine someone swapped the position of the shift and delete keys – you would be frustrated that after learned the original location (the traditional location of the enter key) that someone changed things around. Consider that when creating your sites and contemplating usability.

Of course, every great site doesn’t have to follow conventions, but there should be a good reason for ignoring them. Conventions and expected behaviour generally do help improve usability.

The results:

At the end of the test, a nice heatmap is generated (for the test creator to see) showing where people clicked, and how long it took them (not shown here), so you can judge what is or isn’t working properly, as per the example below:

An example of a heatmap with results

The final product - a heatmap with results

In this example, you can see 87% of people clicked the correct button, and a few random clicks as might be expected as this is a demo test page we like to show people, but all in all most people get the exact right place. Of course, if we’d asked people how to preview a test and they clicked publish instead, we might need to make some changes.

So, what next?

Head on over to video.intuitionhq.com/email-face-off and take the test. Help us figure out just which webmail service is the best, most usable among them. Be sure to leave your thoughts in the comments about the different services and testing process too.

We’ll be writing about the results shortly, and how we think these sites can improve, and what they’ve done right, so watch this space.

And if there are any other categories of sites you’d like us to see in a usability face off, be sure to let us know in the comments.

Jacob Creech

Jacob is a usability geek at IntuitionHQ.com - a new, quick, easy to use usability service from Boost New Media. When he's not indulging his passion for the internet, he is probably bloging about China and Chinese learning (having spent three years there) at Sinomatter.com. You can find him on twitter talking about usability @intuitionhq.

15 Smart Tools To Help You Build Your Freelance Business

Discover the awesome tools we use in making our clients comfortable and happy in learning new things every day.

Download Now

Comments

  1. Fazil says

    the latest version of gmail is the best and the awesomest ever and it numbs the user interface of hotmail and yahoo!!

  2. Vitaly says

    I really liked the new version…

    btw, the heigh of each row might be seen high for you, if this happens just change from the options.

  3. Umair says

    love gmail when i used it :P but i am currently using hotmail & kinda used to it but i,ll surely switch to gmail soon

  4. Xend says

    I strongly believe gmail is a non-starter because it seems to be the only email app written in the last 25 years that doesn’t support preview pane. Preview panes are the practical, efficient solution to reading emails. The standard gmail method is a huge waste of time. Do the math folks – minutes, hours, days, years of your life reading thousands of emails. Google’s inefficient UI is nickle and dimeing you out of a good chunk of your life. Google gmail recently introduced a “pop-up” preview. Being that it’s a pop-up means that it’s a pain but not to be confused with a pane. Not even close. I use yahoo and google email regularly (plus others) for various reasons but I only use gmail because I have to. The UI wastes my time and feels tedious. gmail users don’t know what they’re missing. Stick with yahoo if you value your time.

    • Gordon says

      WTF are you talking about, you obviously haven’t used Gmail in the last few months, there is a preview pane – either vertical or horizontal. Know what you are talking about before spreading bs.

  5. Jarkko says

    I just took the test, and even if I’ve used Gmail for years now, I still managed to get lost in it and not find what I was looking for immediately. I think the other quite similar interfaces confused me. They all look quite dull and could probably be made more easy to navigate. Now that I’m trying to think back to the sites, the biggest differences I recall are the logos. I think Hotmail has some kind of slight gradient coloring in it too.

    I don’t know if Gmail has the clearest look, but I prefer it solely because I’m used to it.

    • Jacob Creech says

      Thanks for your comment Jarkko. It’s interesting to see when you put the layouts side by side, and see the similarities and differences between them.

      I think part of it is that people get used to using a certain service, but another side is that some designs are just more intuitive than others.

      Look out for the results of our tests coming soon though. I think they are pretty interesting.

  6. Terri Lockerridge says

    I prefer any other email to Hotmail. That has to be the most confusing and nonuser friendly mail that I have ever encountered on the net. So many choices and none of them lead to anywhere directly.. typical Microsoft type of thing.

    I used Yahoo mail for years until they did their platform change over and they started invading my privacy. They want you to buy a premium mail service from them and it is not worth it at all… the free mail is so hard to navigate since they put all that crap on the page .. you can’t find what your looking for any of the time.. even if it is the mail you use most of the time. They move stuff around and are always throwing ads to pressure you into the premium service.. pooey on them!

    I went to Gmail from Yahoo.. and love the chat they have .. l like the simplicity of the entire thing. Too many graphics on Yahoo, and Hotmail. The only thing I dislike about the service is you can’t get them to give you the tutorial on how to use the html option they provide to compose a custom email with images. I have not been able to find it anyway.. and when trying to use Word to compose and then merge.. it fails every time.

    Of the three .. I like Gmail the best. I am in agreement with you totally about it.

    • Jacob Creech says

      Thanks for your comment Terri – it sounds like you’re in a good position to judge having had extensive experience of the three.

      You are right, none are perfect, but so far in my experience Gmail is well ahead of the others.

      That said, I’m going through the results of the test now, and there are a few interesting results in there. Stay tuned for that in the next week or so.

  7. Joe Barstow says

    Very insightful. Loved your approach! Refreshing to see something new, and gave me a lot to think about. Thanks!

  8. Arulrajan says

    Did the calendar also should be part of the email client or It can be a separate web site (eg., Google Calendar)

    • Jacob Creech says

      I think in most cases they want to integrate the services together so people will use them more.

      However, the point of that task is just to see whether links to common services are laid out in a logical manner.

      Cheers.

  9. Jay says

    I wish all of the web email clients would improve their search capability. One of the main reasons I love my desktop email client better is the searching.

  10. Matthew Corner says

    Shame the GMail interface improved imo just a few days before this article. I personally love the new tidy changes, and am a gmail person through and through.

    • Jacob Creech says

      In fact, both gmail and hotmail are currently rolling out interface changes – although only to a limited number of accounts. Hotmail seems to have had a number of issues with their rollout though.

      I plan to cover these changes in the second part of this article though. Interesting to hear your thoughts on the new gmail design though, a number of people I’ve talked to thinks it looks a little more clunky now.

      Thanks for the feedback!

    • Jacob Creech says

      Thunderbird has certainly been improving at a great rate. Maybe we should do some kind of test on popular desktop mail clients as well.

      Thanks for your comment.

  11. says

    Test needs fewer interface captures – if I’m familiar with an interface, I’ll respond more quickly – the interface has already trained me – the portion of the test that uses captures is a popularity contest except for the small percent of users who’ve used none of the three interfaces. Interface captures are ok if the question is about which do you like better – in other words, if a question is transparently about popularity. Maybe asking which interface user is most familiar with & discounting responses that favor that interface would right the built-in bias factor that captures bring to the test.

    • Jacob Creech says

      Thanks for your comment. You are right about people being familiar with an interface, but there is research showing that by providing a mixed array of designs, people tend to forget those interaction biases.

      Also, as the tasks measuring peoples interactions with a current design are only part of the test. The results will look at the test as a whole, and hopefully can draw some good conclusions.

      Cheers.

  12. peter says

    no bias..yer right, you show a huge screenshot of Yahoo mail and it looks big and empty (undesirable) and just a partial screenshot of gmail and hotmail

    • Jacob Creech says

      Hi Peter,

      I assume you are talking about the screenshot in the test (task 4). That screenshot is just to show the full content of each email service – it just happens that in Yahoo mail, it seems to take up the full height of your screen no matter the what resolution you are at, whereas with the others, supposing your inbox isn’t full of mail, only take up part of the screen area.

      You’ll also notice, the screenshot is one of 17 questions in the testing process, and it won’t be that one result which decides which of the email services is most usable.

      Hopefully that will help allay your fears. If you have something to say about the usability of Yahoo vs the other two, I’d love to hear that as well.

      Cheers.

  13. Mark says

    Just took the Hotmail vs Yahoo vs Gmail test… not sure what sort of result you’re aiming for but I felt the questions may be worded better…

    They generally start “Where would you click…”, though I got the impression the test is really asking “Where should you click…”
    For example… on yahoo, I was able to find and click the calander button… but It wasn’t very clear or easy to find, nor was it the first place I looked for it. Does the test know I took some time or just know that I clicked the right area?

    • Jacob Creech says

      Hi Mark,

      Thanks for your comment – the test records both the amount of time taken to click, and the location of the click.

      If someone clicked in the right place, but it took them a long time to do it, it obviously isn’t a desirable situation. By recording both metrics, it’s easier to see when a problem occurs, and to be able to solve it.

      Cheers.

  14. says

    Great Idea!
    I personally use gmail and I think it is so much easier to use than Hotmail, I personally haven’t used Yahoo Mail, mainly because I kind of feel like gmail is the industry standard mail service.