When I was much younger, I've always thought that the best survey/rating scale would be 10 points (the engineering type who thought that being precise is best). Then I learned of the pros and cons of odd and even scales, why 7 and then 5 is better.

This is a link to an article which to me, makes a lot of sense. (http://blog.vovici.com/Blog/bid/18112/Less-Really-is-More-When-it-Comes-to-Response-Scales) Hope you can get something out of it. There's a current thread on scoring scale of internal audits. I think the same kind of logical thinking applies.

Every researcher (and every client!) should be interested in developing surveys that:

1. Look quick and easy to take
2. Generate maximal variance (i.e., good distribution among all response options)
3. Avoid "response set" (i.e., respondents mindlessly giving the same answer for every question)

Some very interesting ideas, and very innovative.

Its been a while since we "tweaked" our survey. Maybe its time.

Very interesting article, Harry. Thanks.

Though-provoking and, potentially, very useful indeed. That's a great find - thanks Harry.

Thanks for the article, Harry. :agree1:

A few thoughts...:D

The most important thing that the survey needs to be is valid. That is of the primary concern.

In making a cursory look over several articles, the general consensus suggests that 4 (forced opinion) or 5 (neutral position) is generally favored, with no more than 7 choices. Much of what to use will depend on what you are measuring and if you can obtain valid information with your scale.

http://www.du.edu/ir/links/Choosing%20an%20Odd%20or%20Even%20Scale%20Response%20for%20Survey%20Items.pdf


(http://www.socialresearchmethods.net/kb/scallik.php)

he most important thing that the survey needs to be is valid. That is of the primary concern.
Sorry to be contrary, but the most important thing to me would be getting back some useful results. If that's what you mean by 'valid' then I agree.

I loved the suggestion of 'mixing up the response wording' and thoroughly agree that same old 'poor - good - great' type of surveys are blah. I won't even fill 'em in these days.

Sorry to be contrary, but the most important thing to me would be getting back some useful results. If that's what you mean by 'valid' then I agree.

I loved the suggestion of 'mixing up the response wording' and thoroughly agree that same old 'poor - good - great' type of surveys are blah. I won't even fill 'em in these days.

Yes, we agree:D; validity is paramount. When I speak of validity, it is in terms of reliability and validity (http://www.socialresearchmethods.net/kb/relandval.php). Notice the box in the link for discriminant validity.

Surveys need to measure what they purport to measure; thus, they need to have assessed validity. And yes, there are a lot of different types of validity.

Without dissing the value of the reading, I wanted to strongly caution against thinking that in surveys to just move to 3 measures for a response scale for surveys. For the authors of the survey, it may be valid because all they do is one type of survey asking the same questions all the time. They have the luxury of measuring validity and reliability on it. Although, I don't think that is the case. This is why surveys are such a bear, and typically you will have low response. You need adequate measures of your construct; thus you may have up to 8 questions for 1 construct. The key is to only seek out 4-5 constructs of interest.

Notice they alluded to the notion that no one marks "poor" very much; so, there is no value in having 4 or 5 measures. That's not necessarily true. When rating anything, people may not want to rate something as poor (especially the social factors that go along with rating another person); but may be more inclined to mark an evaluation of "significantly needs improvement" for customer service response time. It's all in the questions, wording, construction of the scale, and their validity.

With a 3 point scale, most of the answers will probably converge to the central answer. Which may be what you want to do anyway.;) But you might have a hard time being able to detect significant differences between the values, as there is not enough discrimination.

Just saying, if you are going to have a survey, make it a good one.:agree1:

Brad, my 'if that's what you mean by valid' obviously needed the tongue in cheek icon! Yeah I understand the concept of validity and yes of course :bonk: we agree that
Surveys need to measure what they purport to measure

BUT frankly I'm willing to there's a whole lotta supposedly valid and validated and statistically fine-and-dandy surveys out there... where I'd actually query whether one can get useful results from them.

Now, I'm arguing here from an individual viewpoint and not particularly scientifically. Hell, not scientifically or mathematically sound at all!

BUT I also know that I am very, very rarely gonna give you (a surveyer) a whole lot of my time asking and answering silly bloody dreary questions, and as for 5-8 to get 1 construct? You may want, but you won't get MY feedback that way. I just refuse to take part in such things.

Nor will you get me to fill in one of them silly bloody forms, neither... But I'd be a whole lot more inclined to do it with only a few questions and with some more interesting that the standard blah responses. (Apart from anything else, it might make me consider that whoever's reading the responses has thought about the questions and is taking a more interested approach).

It's a big reason why this particular article struck a chord with me and a lot of other people I mentioned it to. And that's what I mean about getting useful results.

So if trying out this approach allowed one to pick out the odd place here and there when something wasn't good (or was outstanding), that's very useful in my book. Whether or not it's considered technically 'valid'. (But I'm also aware that a whole buncha statistically trained people are probably about to emerge from the thicket and express horror at the very idea ...

PS correct me if I'm wrong, but isn't this definition of yours a tad tautological (or at the very least circular)?
When I speak of validity, it is in terms of reliability and validity.

And as for 'discriminant validity'... yerrk. Sounds nasty to me. :lol:

And as for 'discriminant validity'... yerrk. Sounds nasty to me. :lol:
This is probably in reference to Discriminant Analysis (http://www.psychstat.missouristate.edu/multibook/mlt03.htm). It is typically used in combination with Factor and Cluster analysis in market research.



Factor analysis is used to determine what "factor" explain variation in results
Cluster analysis is used to determine if combinations of these factors form discrete clusters.
Discriminant analysis (http://www.psychstat.missouristate.edu/multibook/mlt03.htm) determines what demographic data will describe these clusters.

Brad, my 'if that's what you mean by valid' obviously needed the tongue in cheek icon! Yeah I understand the concept of validity and yes of course :bonk: we agree that


BUT frankly I'm willing to there's a whole lotta supposedly valid and validated and statistically fine-and-dandy surveys out there... where I'd actually query whether one can get useful results from them.

Now, I'm arguing here from an individual viewpoint and not particularly scientifically. Hell, not scientifically or mathematically sound at all!

BUT I also know that I am very, very rarely gonna give you (a surveyer) a whole lot of my time asking and answering silly bloody dreary questions, and as for 5-8 to get 1 construct? You may want, but you won't get MY feedback that way. I just refuse to take part in such things.

Nor will you get me to fill in one of them silly bloody forms, neither... But I'd be a whole lot more inclined to do it with only a few questions and with some more interesting that the standard blah responses. (Apart from anything else, it might make me consider that whoever's reading the responses has thought about the questions and is taking a more interested approach).

It's a big reason why this particular article struck a chord with me and a lot of other people I mentioned it to. And that's what I mean about getting useful results.

So if trying out this approach allowed one to pick out the odd place here and there when something wasn't good (or was outstanding), that's very useful in my book. Whether or not it's considered technically 'valid'. (But I'm also aware that a whole buncha statistically trained people are probably about to emerge from the thicket and express horror at the very idea ...



Jane, I don't fill out surveys either. But that is not the point to my vociferous objection with some of these notions put forth by this article.

First, you have a subset of people who flat don't fill out surveys; rule them out. Second, the people left have a subjective measurement to them as to how many responses they will be willing to answer.

My whole objection with this is the notion that 1) limiting the choices of responses makes your survey more "appealing". I submit that I can get equal # of responses to either of these questions:

Overall, the quality of the product was acceptable: Agree/ Disagree
The quality of this product given it's price is: 1 being poor; 10 being superb (on a scale of 1 to 10).Yet I can yield much more useful information (for me) with question 2.

The length of the survey is what turns people off (#of questions); not the estimation of their response (3,5,7, whatever). Also what frustrates people are poorly worded questions that they don't understand.:)

2) The next objection is the notion that it's OK to sacrifice good metrics for your survey, so maybe a few more people will fill them out. If that is the general notion, don't give a survey. :D Oh yeah, you'll get back a bunch of information. Either you have forced everyone to answer the way you want (by the construction of the questions), or they are all over the place, due to question interpretation, poor construction, etc.

3) Finally, the authors act as if they discovered the notion of switching up the questions on a survey. That has been implemented for many years in surveys, to prevent people from just going through filling in boxes. Ideally, you don't just move it to the middle (although with three responses, you don't have much choice:tg:); you reverse not only the scale (1 to 5; favorable to unfavorable; etc.) you reverse the wording of the question to get the opposite. These are ways to know, to some extent, that people are reading the questions and giving a thoughtful answer on the subject.

Jane, what I am trying to implore here is that everyone be very very cautious about cheapening surveys to make it more popular (or whatever). You'll get back results, but they may not be worth anything. The danger in that, is you won't know they are not worth anything.:D You measured something other than what you thought you were measuring. Or, by the design of the survey, you kind of used social cues to force everyone to answer a certain way anyway.:)

I could write a survey right now on, say, Apple. I can get 70% approval, or 70% disapproval, with lots of responses.;) Or, I can take my time, build a good survey, and get 8-10% meaningful response like everybody else.