A friendly reminder to always be wary of research findings. Always. Even perfectly reputable organizations do not always put out perfectly sound research.

The most common violation I've noticed of late is a tendency for online design commentators and bloggers to cite research results from SURL (Wichita State's Software Usability Research Laboratory) without qualification.

That is, the majority of their studies use fewer than 100 participants -- some as few as eight -- and due to the nature of their surroundings, most of those participants are college students. Additionally, their samples are self-selected (i.e. typically students who have volunteered, meaning selection is not random.)*

This means the only larger population to which their conclusions may be generalized is volunteer-minded, message-board-reading college students in Wichita, and college students are not necessarily representative of the general Internet-using population. Any findings should be considered preliminary or discussion-worthy at best.

For those who don't know and to illustrate how starkly their samples differ from a sample which could be meaningfully generalized to the population at large: You need about 1500 participants who are selected at random from that population.

* Kudos to SURL researchers for posting their methodology. This simple step puts them leaps and bounds ahead of some professional organizations.

March 13, 2003