Survey 101 - A Complete Guide to a Successful Survey

The Corner Store

With all the talk in the media of polls and focus groups, it's easy to think that market research is a product of the latter half of this century. But it is as old as commerce.

Think of a shop owner 150 years ago. He didn't have television dictating national tastes. There were no trade publications, no market trend reports. Yet his business has the same basic problems faced by companies today. He had to understand what his customers wanted, manage employees, and work with suppliers.

How did he accomplish all this? By asking questions. The store owner worked along side his employees. He chatted with customers during the day and personally took their requests or handled their complaints. Vendors were not anonymous but came in the person of representatives who would travel from one store to the next.

Business has changed over the years, with companies employing thousands; customers doing business by phone, fax, mail, and the Internet; and vendors from around the world competing to supply products and services. As the process of business has become so dispersed, corporate management finds it increasingly hard to keep in touch with customers, employees, and even most investors.Yet it has never been more critical to gather the types of information the store owner easily acquired. What was a natural expression of doing business at that time is really a model of what successful companies do today. Corporations commission studies of new and existing markets. Focus group leaders try to understand the issues driving customers and prospects alike. Dedicated staff handle inquiries from investors. In short, many modern businesses spend extensive time and lots of money to better understand the needs and concerns of stakeholders - the people with a vested interest in the companies.

The idea is that if management is armed with relevant information, it can come to better decisions. The question becomes how companies can keep on top of what everyone thinks and wants.

Smart Companies Use Surveys

While the personal days of the corner store are gone for most companies, it is possible to recapture the benefits of more intimate business, by using surveys.

Forget for a moment the associations with market research and professional polling. Think of a survey as a way of asking the questions you would put to people if only you had the chance to sit down and speak with them. You could see what customers liked about your products and services - and what they didn't like. Consumers could explain the types of things they wanted to buy. Employees might tell you how they really felt about new policies. Responses from investors - small as well as large - could help your company better understand them.

In other words, surveys allow companies to become "local" to their stakeholders, no matter where the stakeholders and the company are. By developing a questioning attitude, a business can spot trends as they develop, address its own weaknesses, take advantage of opportunities, and keep everyone happier and more productive. Achieving this means creating a corporate culture of constantly asking questions, learning from the answers, and adjusting strategies and tactics.

Best of all, it is possible to plan, implement, and interpret surveys without an extensive statistical or market research background. By applying basic principles, using common sense, and letting tools like SurveySolutions from Perseus Development, just about any company will find itself able to add valuable information to its decision making. And, happily, it is possible to achieve all this at a reasonable cost, in far less time than ever before.

Tip: No survey - or any other marketing research technique for that matter - is foolproof. People may not tell the whole truth and survey techniques are fallible - just look at the times when political polls don't predict the outcome of elections. It is always important to temper decisions with common sense and experience. Survey data should aid you in decision making, not make decisions for you.

Smarter Companies Use the Internet for Surveys

One problem with surveys has been the cost in administering them. Either you needed staff to telephone or meet with people to ask the questions, or you had to undertake a direct mail campaign, with the expenses of printing, mailing, paying for the return of responses, and finally entering the information into a computer for analysis.

Electronic commerce and the impact of the Internet on communications have opened new worlds for surveys. Hundreds of millions of people world-wide have access to email and the Web. Answers come back in an electronic format, so putting them into a computer is easier. A survey can reach people in a matter of seconds, rather than days or months while completing phone calls or waiting for the mail to come in.

Example:

Let's compare the costs of delivering a simple survey to 100 people -

Phone: $50 phone costs + $250 interviewers + $250 data entry = $550
Mail: $100 printing/postage + $400 open envelopes/enter data = $500
Internet: $50 create/deliver form + $5 convert data = $55

In reality, we are probably overstating the Internet costs, making it an even better alternative.

This isn't to say all company surveys can use the Internet. E-surveys presume that the people you want to interview can be reached via either email or the Web, an assumption that excludes the majority of the world's population. You might want a phone or in-person survey to allow skilled staff to probe people's answers to gain more insight. It could be important to contact subjects in a particular way; for example, a retailer might want to interview people in malls to understand their behavior while actually shopping.

If, on the other hand, you have a Web page, if your audience regularly uses email, then you may have an opportunity to perform surveys at almost no incremental cost.

Even if your common sense tells you to use more traditional methods, you will likely find SurveySolutions, as well as this tutorial, helpful. The combination should help you create better, more focused surveys that will answer your important questions in less time. And that should mean something to your bottom line.

It is fine to talk about bottom line improvements and a questioning business approach, but that alone won't help. What you need is a no-nonsense method of planning, implementing, and understanding surveys. In short, you should:

1) understand what you want to learn and write a survey that will get the information you need.

2) run the survey process so everything goes smoothly and you minimize the work necessary.

3) learn to interpret the answers you get, so you know what conclusions you can and cannot draw.

That's what the rest of this tutorial covers. It's written for the general user, so you don't need a background in statistics or market research. Just bring your business issues and questions and we will help you get the rest of the way.

Planning the Survey

In any venture, those with battle scars are bound to point out that proper preparation is a major key to success. In fact, "Well begun is well done" could become a mantra of those managing projects. Surveys are no exception to the rule. It is possible to conduct a survey without the necessary planning, but the result will be painful and unpleasant.

So we will save the aggravation and begin well. The best starting point is not scribbling down questions or wrestling with obscure mathematical formulas, but establishing your goals clearly and concisely. The whole point of asking questions of people is to get answers that will help you make important decisions, so knowing what you want to learn and why it is important. Your goals will affect the questions you ask, the people you ask them of, and the way in which you ask them. Try starting with a short explanation of the reason for the survey, similar in concept to a company's mission statement. From there you can expand into all the details that are important.

Bad survey statement: "We want to establish fiscal parameters in the customer decision making process in the plumbing and bathroom products arenas, testing price points and elasticity. After gaining this information, we will analyze its effects on marketing strategies and tactics."

Good survey statement: "We want to know how much customers are willing to pay for sinks to see if we can make more money."

The clearer you see the target, the more easily you can see if you hit it or not.

Once you have a succinct statement of what you intend to find, it's time to expand on the concept. Don't worry about structure, just start writing a list of information that could be important to your quest. The list will help you judge whether your questionnaire satisfies your goals and answers your questions.

Let's take the survey statement, "We want to know how much customers are willing to pay for sinks," and develop it. In a short time, we have additional questions or areas to cover in the survey:

Do customers consider costs of accessories and installation, or just the sink itself?

How does price sensitivity differ by consumer demographics?

Is there a difference in attitudes in different parts of the country?

What do customers think sinks cost?

When do they buy sinks? When they break? When remodeling? Do different circumstances change attitudes toward price?

This process can continue, depending on your needs.

Once you know your goals and what you want to learn, you next need to think about three things: planned use of the answers you get, deadlines, and budgets.

• How you expect to use the answers you get has as much impact on a survey as your goals. Are you ultimately writing a report rich with tables and graphs? That suggests one level of detail in the survey. Planning to show results live on a Web page as people give their answers? Then you may want to restrict yourself to simple yes/no questions.
• A project without a deadline is doomed to languish since it never comes due. Pick a final deadline that is realistic so you can get the work done but short enough so you receive results while there is still time to do something about them.
• Any project requires a budget - not just money, but also personnel time to prepare the questionnaire and analyze the results. That budget needs to be large enough to let you address all the issues in your survey that you want without disrupting your other work. If you can't afford the time to administer and analyze a long survey, scale back your plans.

The Basic Steps

After you have the foundation of your planning - budgets, deadlines, goals, and information uses - it is time to plan the survey itself. That takes five basic steps:

• choosing the right people
  
• using the right vehicle
  
• asking the right questions
  
• obtaining the right interpretations
  
• persuasively presenting results the right way

We're going to take some time on each of these topics, as they are important. But first we have to take a short statistical detour.

A Fast Note on Statistics

It's impossible to discuss surveys without getting into statistics. That means practically no one will be happy. The mathematically averse won't want to read the section, while statistical experts will undoubtedly deem the treatment superficial. But bear along as we explain our reasoning of why everyone should look through this part.

Though a branch of mathematics, the accurate use of statistics can be as much art as science. Those who don't like math must remember that surveys are based on statistics. Without the proper care, you won't know if your results are correct. Depending on your goals, you have to interview a certain number of people and realize which conclusions you can and cannot draw. Forewarned is forearmed.

When it comes to business surveys, too much statistical knowledge can be as limiting as too little. A company rarely has either the leisure or even the capacity to undertake surveys in the ways that will satisfy purists. Yet it is still possible to get reasonably accurate answers which, viewed with common sense, can aid in the decision making process.

So we will concentrate on practical statistics to help you realistically achieve believable results. Is there much more to be said about statistics? Yes, especially if you want to undertake sophisticated analyses of your results. But you can still make progress before getting a graduate degree.

Statistical Significance

Businesspeople often talk about average people, like customers, employees, or investors, but there really is no such thing. It's a pity, really, because otherwise you could speak with one person and get the information you want. Unfortunately, there is no such thing as the perfectly representative person.

That's where statistics comes in. It's the mathematical approach to finding the average customer. By talking to the right number of people in a group, called the sample, you can get a fair idea about the inclinations of the group as a whole.

Finding that right number is critical. Too many responses - also known as the sample size - and you spend more time and money administering surveys than you need to. Too few, and you can't trust the results since a single response unduly affects your totals.

In the parlance of mathematics, you need a statistically significant number. While there are formulas to determine what statistically significant is for a group of a given size, there are some good rules of thumb. Professional researchers generally agree that getting 100 responses is a good number. There are enough answers to track variations in response down to a single percent.

In another sample, we have 100 responses, each worth 1% of the total and giving us far more flexibility.

Note that we mentioned 100 responses as a good number. It is possibly, though, to get fewer responses than that. Depending on whom you ask, the minimum number of responses you need is between 30 and 50. More is better, but companies often find themselves in positions where more is also impossible.

For example, one feature of a new product design may generate disagreement about how valuable customers would find that feature. In a competitive marketplace, time is of the essence and waiting for 100 responses might cause an unacceptable delay. In such a case, you might well settle for 50 - or even 30 - since acting on information with a reasonable amount of reliability is better than waiting for greater certainty while a competitor gets to the market first.

Another situation where fewer responses can be fine is when anyone points out an embarrassing problem or tremendous opportunity. Don't focus on statistically meaningful numbers so much that you miss the gold nuggets that drop into your lap.

With the talk of small numbers, it would be unsurprising to wonder why political polls may use 1,000 or more people. That is due to subgroups.

There will be times that you want to look not only at a group as a whole, but also at different ways of analyzing the group - by demographics, perhaps, or the amount of money someone spends with you, or that you pay to your vendors. In such cases, you want to compare the results of one of these subgroups to those of another.

Now remember that we need at least 30 to 50 responses to have statistically significant results, whether we are talking about subgroups or the overall group. Here is another place where having your goals clearly in mind will affect the way you conduct the survey. Since you need a minimum number of people in each subgroup you want to analyze, the size of your overall group has to be large enough to provide them.

A public company wants to increase demand for its stock. To that end, it wants to learn how well-known its name is among potential investors. The CEO suspects that the company is recognized only on its home turf. So the survey may split the country into six regions and want a minimum of 50 interviews per region. That means there has to be at least 300 (6x50) total interviews.

Types of Samples

Whether talking about groups as a whole or subgroups, you have to recruit and select people who will provide the responses, which brings us into sampling. To avoid jargon for a second, the sample is the group you choose that is supposed to represent some category of people, like customers, investors, prospects, or what have you. Sampling is the process by which you gather responses.

Though doctoral treatises have been written about sampling, we will continue to avoid the complexities. The accuracy of a statistical approach depends on the choices of representatives for a group to be made randomly, where the choice of one person is as likely as the choice of another. Otherwise, you are stacking the deck with people who may skew your results, making any analysis faulty.

It's largely a matter of common sense, but problems can pop up in uncommonly devious ways.

Think of the call-in surveys run by newspapers and television programs that solicit audience opinions on various questions. It is wise to question whether the results are really meaningful because the sample is self selecting; in other words, subjects choose themselves. Even if the survey receives hundreds or even thousands of responses, they have not been chosen at random. It could be that people who respond to the survey tend to hold particular opinions and thoughts that would not be representative of the audience as a whole.

Is this a problem? Again we come back to looking at goals. The media outlet in question is not making decisions based on the survey's outcome, instead using it as a device to further engage the audience and increase ratings or the number of copies sold. In that case, management may be uninterested whether the responses bear any semblance of a greater reality. A business looking for marketing information would find such an approach reckless.

Remember: All surveys suffer from this to one degree or another. People refuse to answer surveys over the phone or throw away questionnaires received in the mail. If you want to make decisions based on results, make surveys as random as possible. It's better to have self-selection problems where people refuse to answer than when they choose themselves.

Other issues can sneak in when you are trying to study subgroups. You may be tempted to choose the minimum number of people needed for each subgroup. However, if you are not careful, the subgroups may appear in non-representative numbers in the overall group. Though you may be able to draw some conclusions for each division, results of the group as a whole may be flawed. To remedy this, you need to get enough responses so that the proportional make-up of the whole group is correct and all subgroups have a statistically significant number of responses.

A company wants to understand the buying habits of prospects in a given geographic area. Looking at the people who might want its products, the company finds that 20% of the prospects are men and 80% are women. A survey with 50 responses from men and 50 responses from women would have statistically meaningful results for both sexes individually, but would have responses skewed toward the male view overall. To look at both sub-groups and the group as a whole,you would need about four times the number of women as men, or 250 overall.

Buzzword

As a profession of chemistry once quipped, buzzwords are those terms which keep insiders in and outsiders out. Unfortunately, they are often necessary technical terminology with exact meanings. We will go over a number of statistical terms because you are bound to come across them while doing surveys. To understand them in depth, consider an introductory book on the subject

Mean

A mean is the statistician's average. Though its concept and use can get a bit complex, in survey work, think of it as a simple average. If you ask 100 people to rate something on a scale of 1 to 10, you would add up all the scores, then divide by 100 to get the mean. While it only makes sense with numeric responses, we will later cover how you can sometimes convert other types of answers to numbers

Median

The median is the centerpoint, ranked from low to high, of a group of numeric answers. Don't confuse it with the middle of a range you let people use. In the example of a mean, the midpoint of the scale would be 5. But the median of the 100 people could just as easily be 4 or 6 or 2. The median will depend only on the answers you receive

What They Aren't

It's easy to misuse means and medians - especially the former. Because it is mathematically the same concept as an average, many people assume that the mean is the same as an average answer. It doesn't have to be.

If you are dealing with numbers in a standard distribution, usually pictured as a bell curve, then by definition, the mean and the median are the same, and you can look at the mean as a truly average value. But when the mean and median differ, you run the risk of either alone giving an incomplete view of the information you are studying.

Let's look at the average salary in a small company. There are five workers making $10,000, $20,000, $30,000, $40,000, and $150,000 respectively. The mean of the salaries is $50,000, but four out of five employees make less than that. A median of $30,000 puts the mean into some context.

Standard Deviation

Another buzzword we will cover is standard deviation. Technically, it involves looking at the square of the difference between each number in a group and the mean. Since spreadsheets can calculate it for you, think about the standard deviation as a measure of how spread out a group of numbers is. If you have numbers that fall into a bell-type curve (and that's usually a good assumption), about 80% of the numbers will be within two standard deviations of the mean. Knowing the standard deviation further helps you understand what responses really mean.

You survey a group, asking them to rate doing business with you on a scale of 1 to 10. After putting the numbers into a spreadsheet, you find the mean is 5. You are concerned about some very low scores of 1 and 2 and are hopeful about some scores of 9 and 10 you received. You have the spreadsheet calculate the standard deviation and find it is .5. So 80% of your customers rate you between 4 and 6, meaning that even with the extremes, your customers overwhelmingly perceive your company's performance as middle of the road.

Finally, at least for abstruse terminology, is margin of error. You have probably seen this phrase used in newspaper accounts of public opinion polls. It is not a measure of how accurate the information provided by the poll is. Rather, given the same audience and questions, the margin of error indicates how often you would get the same results should you repeat the survey with a different sample.

However, this does not guarantee that your results are accurate or useful, but only how faithfully you could repeat them. The creation and arrangement of questions offers large and opportunities for unreliable results. We will cover survey questions later.

Choosing the Right People

Given everything said so far about statistics and planning a survey, it's now possible to discuss selecting the right people for a survey sample. Going over the basics, we need to choose the proper audience, which means focusing on goals again. Will it be all current customers? Particular types of customers? Prospects? Vendors? Investors? You will need to compile information on this audience that will help you identify and contact them. Since we are talking about Internet-based surveys, that means trying to obtain email addresses. Otherwise you will have to undertake a possibly expensive first step of contacting potential subjects to request email addresses. It is possible to avoid emails and to run a survey on the Web, but you may still have to contact people to invite them to take the survey. This helps control the self-selection factor. This is not always necessary, though. For example, you might want to learn something from visitors to your Web site. In that case, invitations could actually distort your results.

In any case, rest assured it is possible to run an accurate survey on the Internet, either by email or on a Web page (with or without an invitation). You do have to check the big assumption that virtually all of your intended audience uses the Internet. With expanding use of the Internet throughout the world, this has a good chance of being true. There will also be those times that you have to resort to other approaches, like telephone or mail.

For the most part, though, surveys mean actively contacting people. Some sources of information are subscription lists of magazines, trade association memberships, or lists of registered product buyers. Start the looking process weeks before you want results, as there is often turnaround time involved in getting lists

Ask what information is available about the lists you use - and even ask about what information is kept on your company's own contact lists, if you plan to use in-house names. It may be that your goals are better satisfied examining only part of your audience, and readily available information like demographics or purchase histories might help focus your efforts. You will only use a fraction of the names. A typical way of picking them is to take the total number of names and divide it by your intended sample size. Round the result to the nearest whole number N, then pick every Nth name.

Next, ask yourself if your goals are better served by examining the responses of subgroups. If so, you need to perform a stratified survey, which means picking your sample so each of the interesting divisions has enough responses. When the subgroups are the true interest, and not the audience as a whole, selecting just enough responses for each will probably reduce the amount of interviews you need to do. An unstratified survey, in which you treat the audience as a whole, is even less work - but only if you can get the answers you need.

One type of breakout of an audience that may be important is geographic. Tastes, customs, and habits often have regional variations that companies find helpful to understand. A survey that looks at geographic groups is called clustered, while one that ignores geographic distribution is unclustered.

Remember that the more factors you try to consider, the bigger an overall sample size you need. Even with the additional work and expense, it probably makes sense in a first survey to examine geographic differences, if for no other reason than to rule out their importance in the future if possible.

An earlier example looked at a company trying to understand the buying habits of prospects, 80% of whom were women and 20% men. The sample in that case was to be 250.

Now add in clustering. Assuming that the company uses five geographic regions, the company will have ten categories to study. To study both the subgroups and the sample as a whole, the company calculates that it needs 1,250 responses - and that assumes prospects are evenly distributed geographically!

In cases like this, you have to balance purity of research with budget considerations. By taking prospects as a whole, you could study geographic differences only with 250 responses.

Using the Right Vehicle

Your goals will help determine the type of survey that will work best. There are different types of information you might want to learn from people. Here are some of the most common:

• attitude - how people think and feel about something
• perception - the way people receive messages and interpret them
• needs - not only what people need, but what they believe they need; also desires
• decisions - both the choices people make, and how they make them
• behavior - how people react to situations and opportunities, and also how they think they'd react
• lifestyle - describing people in the context of how they live
• demographics - the categories someone fits into, like age, marital status, business title, industry, and so on

You may have to take different approaches to get the different types of information, more than one of which may appear on the same survey.

In a survey, we ask people directly for their age and zip code, to see if such demographics affect answers. Some areas may benefit from indirection because people often answer in ways that they think enhances their stature.

Success with a survey depends on having the right mechanisms to help people respond. That means structuring questions in such a way that the responses you receive are most useful. A basic division in how you ask questions is that between open ended and close ended. In the former, you let subjects say anything they want in answering a question. People can say exactly what is on their minds, which is an advantage. You can gain great insights because open ended questions force you to listen to the audience. A drawback that doesn't appear until you try to analyze the results is the impossibility of directly comparing one person's words to another's. You inevitably have to fit answers into categories, then perform an analysis. This results in extra work as well as potential problems from imposing structure on people after the fact.

Close ended questions are more common in surveys. By restricting people to answering yes or no, picking a value out of a scale to represent their responses, or picking from multiple choices, it becomes easier to analyze the results. There are three major structures for close ended questions:

• binary
• multiple
• choice

Notice that the differences in structure are only in how you elicit a response from someone. The binary question allows someone to choose only one of two permissible answers. There are many cases where a binary answer is applicable, like in asking people on a decision survey question whether they would or would not buy a product at a particular price.

There is a danger of misusing binary questions if the two choices do not cover the actual range of answers people might give. Subjects end up giving answers they are uncomfortable with, lowering the accuracy of the survey and possibly making the results useless.

Multiple choice questions are in reality a variation on the binary question, but with more allowable answers. Similar to the binary question, accuracy of the multiple choice depends on making the whole range of possible answers available to subjects. It is popular because of its simplicity and applicability to a host of uses.

Note that multiple choice answers work best when each choice offered is distinct from every other.

Poor use of multiple choice: How often do you purchase stock? a) seldom b) occasionally c) frequently

Good use of multiple choice: What type of broker do you prefer for stock transactions? a) online b) discount c) full-service

A scaled question is one in which someone chooses an answer from a scale or range that are all degrees of a response. For example, employees might be asked how they would rate the effectiveness of a particular manager on a scale of 1 to 5, where 1 would be poor and 5, outstanding. Consistency is vital in the presentation of scales: a number representing the low end of the scale in one question should not become the high end in another question in the same survey. Otherwise you run the risk of confusing people, who might continue using the references from a previous question. Note that it doesn't matter whether the scale runs low to high or high to low, like 1 being the top or the bottom, so long as you are consistent.

Answers to scaled questions are often phrased as numeric scales, like asking employees how many hours a day they spend in meetings or asking consumers how many years of formal education they have. Such questions are good because their meaning is unambiguous and interpreted the same way by different people. These scales are called qualitative.

Sometimes a numeric scale is not enough. If you ask people to rank on a scale of one to five how good a cracker tastes with no other explanation, subjects are likely to interpret the numbers differently. That is why many surveys use quantitative scales, in which levels are defined. An example is asking whether someone strongly disagrees, disagrees, is neutral, agrees, or strongly agrees with a given statement. These two approaches are also combined, like when someone ranks their satisfaction with a service and the high and low points in the scale have verbal definitions.

With scaled questions, there are times to use broad scales and other times to use narrow, like 1 to 10 versus 1 to 3. When should you use which? It depends on your goals and how detailed you need to be. You might be satisfied with knowing that consumers planned to buy your product, would consider a purchase, or would not use it. On the other hand, understanding how strongly someone holds an opinion needs more subtle degrees of interpretation.

The more detailed the list of responses, the more time it takes to take the survey, which can discourage people from doing so. A greater number of responses also complicates analysis. A rule of thumb is that 10 different responses on the scale is about the practical maximum. If in doubt, it usually makes sense to err on the side of more choices, rather than less. You can always combine answers during analysis to create fewer categories, but there is no way to disentangle answers forced into the same category at the outset.

Poor use of scale: How would you rate the quality of financial information available on our Web site? Outstanding, Excellent, Good, Fair.

In this example of something a publicly held company might ask investors, the scale is incomplete as it misses a choice of Poor.

Keeping the customer with you

No matter what the format of a survey, the quality or quantity of questions, or the accuracy of sampling, a survey will provide nothing if people will not fill it out. The most important part of a survey mechanism is a plan for keeping the subject with you, giving their answers. To achieve this, follow two rules:

1. Respect subject intelligence. There are few things more off-putting than listening to someone who treats you as a child or fool. Be condescending and people will toss your survey into a waste can, electronic or actual. Use jargon an audience has no reason to know, and they will consider you a snob, with the same results.

2. Respect subject time. Everyone is busy, or at least they think they are. Even 82-year-old grandmothers retired and at home will say how they don't have enough time in the day to get things done. People are doing you a favor to answer your questions, so don't abuse the help. As your goals must be concise, so should your questions. Though it is possible for a survey to last half an hour or more, aim for surveys to take only 10 to 15 minutes, shorter if possible.

It is also important to decide up front whether the user will see the results of the survey and, if so, in what form. Providing the results of surveys can at times be an inducement for participation, such as a salary survey for a recruiting firm. Most people will want the results for their own career planning and negotiations and may find having them reason enough to provide the information the recruiting firm wants. The curiosity of people answering a public opinion poll online might also be enough to have them take part, but the types of questions, degree of depth, and even format of answers would have to be different, such as percentage divisions between the two choices of a binary question.

Asking the Right Questions

You have the right audience and the right vehicle. Now you need the right questions. For any question, you must be sure you are clear in what you ask and how you expect someone to answer. Anyone in the sample must understand the question and possible answers in the same way.

Any question must also use the right language. Technical terminology might be necessary if your sample comprises engineers but will certainly scare off the average consumer. To write questions for a particular audience, get into their shoes to know how they think and talk. It is a variation of writing for your audience.

Not every question will apply to each member of the sample. For instance, a question of interest only if the respondent is married has no relevance to single people. Asking tropical zone residents about woolen winter coats is an issue of theory only. Let people opt-out of a question when it doesn't apply.

Good survey questions share a number of characteristics:

• they are pointed
• as short as practically possible
• single-minded
• well-ordered

A question becomes pointed by getting to the point. Don't beat around the bush. If you want to know what consumers think about your prices, come out and directly ask their opinion of your pricing.

Blunted Question: What has your experience been with our auto repair service?

Pointed Question: Did our auto repair service properly diagnose your car's problem in your last visit?

Questions should be short to help maintain the attention of people being asked and increase the chance of having the survey completed. The wording of one should be just long enough to correctly ask the question. Strive to make a Maine farmer look wordy.

Too long: Out of your refrigerator, your stove, or your dishwasher, which did you buy last?

Better: Which major kitchen appliance did you last purchase?

Not only do we save six words, but we remain open for other responses, like a garbage compactor or second oven.

To avoid confusion, ask only one thing in any given question. Though you may think you are saving time, by covering more than one topic at the same time you run the risk of muddying the waters and getting inaccurate responses. If you have additional questions on a topic, ask them separately. Beware of unintentionally introducing additional topics through unspoken assumptions.

Too Much: When did you buy your last new car and how satisfied were you with it?

More Measured: Have you ever owned a new car?

If so, when did you buy your last new car?

How satisfied were you with it?

Good Ordering

Questions do not exist by themselves in a survey. Think of a survey as an outline version of a conversation. You supply the questions and others offer the answers. As with any conversation, there has to be a natural flow, with transitions between one thought and the next. The answer someone gives to a given question may be influenced by previous questions and answers. It may be easier for someone to take a survey if certain types of questions or topics are grouped together.

While it is important to write good questions, what a good survey needs is an organized body of questions. After developing your questions, try grouping them in different ways. Find an approach that makes it easy to go through the survey.

The purpose of survey questions is to determine as honestly as possible the opinions, thoughts, inclinations, and interests of a given audience. Unfortunately, the biggest potential for error is not in how you choose a sample or even in your goals. The single largest source for inaccurate results are poor questions. It is all too easy to stack the deck, so a survey will almost inevitably show a particular set of responses. This is sometimes done intentionally by people with an axe to grind, but even more often by accident. We will go over some of the more common sources of bias in surveys.

As we mentioned in multiple choice questions, answers for any question should anticipate the whole gamut of responses someone in the audience might give. Be sure that you haven't left anyone out ahead of time.

Questions with an axe to grind not only affect the response, but may also repel some of the chosen sample from answering the survey. In a leading question, the phrasing used directs most people to a particular answer. A loaded question contains in implicit assumption or argument that influences choices. It is important to phrase all questions in as neutral a way as possible.

Leading Question: Don't you believe dedicated employees are willing to work overtime?

Better Version: Should employees be willing to work overtime?

Loaded Question: Are more expensive products worthwhile because of their higher quality?

Better Version: Do more expensive products have higher quality than less expensive products?

People are strange creatures. We will unconsciously act to increase our status, bolster our self-image, make ourselves look good in the eyes of others, and will say things that people like to hear. We are threatened by some subjects which we want to avoid and are ready to be contentious about others. Though most people would disagree with the generalizations, they are actually a source of considerable bias in survey results. There are steps you can take to limit this bias. Try to structure questions so no answer is the obvious "positive" choice; strive for the same neutrality that you would bring to the question itself. Try abstracting issues, like talking about imaginary people instead of addressing the subjects themselves, if people might find their truthful answers embarrassing.

Another source of bias is the ordering of questions. One question can set someone's frame of mind in such a way to affect the answer to a following question. Examine all the trains of thought in the "outline" presented by your survey. Be a ruthless editor, eliminating all generators of bias under your control.

Leading Question: Don't you think volunteering to work overtime will help our company stay competitive?

Better: Will regular overtime improve the company's competitiveness?

Biased Pairing: Did you know our company invests in your community?

Will you use our company the next time you need this type of service?

Creating the Questionnaire

The questionnaire is both the vehicle that carries your carefully crafted questions and more. It must embody a structure that is easy for a subject to navigate, control the flow of the survey, and encourage people to provide all the responses you want. To create an effective questionnaire, first organize your questions. You may find it easiest to group them by topic. Sometimes, the form of questions may suggest a different structure, where all the questions using a particular scale for response appear together to make the process easier for the subjects.

Be sure to examine your questions for any that depend on answers of others. It makes little sense to ask people to rank their favorite candy bars if you haven't determined that they regularly eat the sweets.

Keep in mind some of the practicalities. A questionnaire administered by someone via phone or in person can rely on proper training. A self-administered survey, such as those deployed on the Web, must be crystal clear in its instructions. Even its physical design should be pleasing and inviting to participants.

The success of your questionnaire will be shown not in its use, but in initial testing. You need to try all phases, including recruiting people to take it. For an online survey, consider administering test cases in person, to see how people manage with it.

Implementing the Survey

You've identified the issues, selected the potential participants, crafted the questions, then created and tested the questionnaire. Now it's time to actually administer the survey. The traditional methods of doing so are through the mail, by telephone, or in person. Each has its strengths and weaknesses. Mail is probably the most economical of the three, but it is harder to get a good amount of participation, with high percentages of people opting out of the entire process. It is also a relatively slow process. Telephone is faster, but more expensive, both in phone charges and in salaries, since someone has to talk to the participants. If you want to fulfill a large number of surveys in a short time, you have to dedicate multiple people, office space, and telephones. In-person surveys give the takers the opportunity to observe non-verbal reactions in addition to what someone says. It is also the best form for probing in open-ended questioning - and the most expensive, including travel expenses.

That is why we will focus on email and Web-based surveys. These combine a number of desirable characteristics. They are much less expensive than even surface mail surveys, since so many companies already have email and Web resources. Since the Internet provides almost instant communications, they are also much faster than any traditional method, with immediate delivery to any number of people simultaneously. The responses come back in an electronic format, so you also save a step or two in processing them.

In addition, SurveySolutions from Perseus can help you in most aspects of implementing surveys, from generating the forms, even in ASCII formats for email, to managing the distribution and even collecting and analyzing the results.

Though SurveySolutions handles the mechanics of the survey itself, there are considerations and planning you need to make.

Email Surveys

Email is a natural format for surveys. You send the survey to someone, receive the answers, and process them. Remember Internet etiquette, though. Many people dislike receiving long, uninvited messages, so send a query to a potential subject first. Tell the person about the survey and ask for their cooperation. (We will cover convincing people to participate a little later.) When someone agrees, then send the survey.

Make life easy on the members of your sample. Let them reply to the email and indicate their responses in that message.

An email survey will likely be a type of bulk mail with similar messages going to multiple people. Consider talking to your ISP in advance to understand any requirements or restrictions it might place on bulk email. Know how to send the emails so someone does not receive a survey with a header including the address of dozens of recipients. Bulk mailing software might come in handy. Also try to send each with an individual identification number in the subject line, so you can trace who sent which response. That way, if you have additional demographic information from a third party, you can use it.

Web Surveys

Web surveys have some advantages over the email variety. You have more control over the physical appearance of the survey and can actually create attractive and inviting forms with the use of either HTML editors or SurveySolutions, which will automate the process.

But Web pages are typically more open environments than email, which means that unless you want anyone coming by your site taking the survey, you need to plan so the people who should take the survey do and that others don't. You might handle this by having the page on a hidden page without links from your Web site, or even creating a separate site. In any case in which you want controlled access, be sure to set the meta tags for the page so it is not picked up by search engines.

Simultaneously, you need to promote the survey. You can email potential recipients, including a link to the Web page. This combines the directness of email with a short message and the ease of using a Web format.

Surveys are useless without responses, so consider what you can do to convince your invitees to take the survey. You can offer someone valuable information - the salary survey offered by an employment agency as previously mentioned is a fine example. A small thank-you gift can be appropriate, as might a discount for a third party product that might interest your audience. One approach often used is to run a drawing or sweepstakes of some form, but there are strict laws governing these, so research the issue before running one.

Also include a cut-off date for responses. You want to analyze the results in a timely manner while they still might apply to your business goals.

Now that you have the completed questionnaire, test it. The questions may seem reasonable, the flow understandable, but you are too close to it. You need other people to actually take the questionnaire because it is only in action that flaws will come through. After people in your company try it and you make any necessary changes, plan on running a pre-survey. Use some of your audience list to take the survey, as though it was the final version. Examine the responses. Are there sections that have been left blank? Any unusually high or low ratings of items that seem suspicious and smacking of questions skewing answers? No news may be good news, but overly good news should be troubling.

As the responses come back, you will need to check them in a number of ways. One is for completeness. Some people will partly fill out surveys and return them. You must examine each survey to see its degree of completeness. Completed ones go into one group as do those that are obviously so incomplete as to be unusable. For the remaining, go through each question to see if there are salvageable results. It might be that you can use those answers in at least part of your analysis.

As you check for completeness, examine the answers to ensure that the responses are valid choices. SurveySolutions from Perseus can do this error checking for you before the survey is submitted. If there is no way of knowing what the intended answer was, throw out that result.

Analyzing Results

Surveys are meaningless unless you can extract their meaning through analysis. The fundamental type of analysis you will need is numeric - examining means, medians, and standard distribution to examine how people in your audience respond to the questions. It is also vital to look at the distribution of answers, because it can alert you to unusual circumstances. Perhaps the mean and median seem reasonable in a question about satisfaction with your customer service, but the scale of 1 through five shows large numbers of responses at 2 and 4. The stark contrast might show an inherent problem in the customer service function that might be due to an identifiable factor, like location or time of service. This is where common sense comes in, because the understanding of the numbers must go hand in hand with the understanding of the business.

There may also be times when you want very fine analysis of data, such as correlating answers with demographics or trying to build a predictive model for audience behavior. Such are the realm of advanced statistical techniques far beyond what we can cover here. It's best to arm yourself with some reading from a text on statistics and software to actually perform the calculations. SurveySolutions provides a wide range of summary statistics and frequency analysis. Spreadsheets like Excel or Lotus 1-2-3 will also have many statistical functions built-in. There are also standalone statistics packages that can offer more exotic choices that may provide the insight you need.

Presenting the Analysis

Ultimately you are not the only person who has to know the survey results. Other people in your company will want to see them and understand their implications. Plan on both a written report and graphics. Having the combination satisfies those who prefer verbal results and others who like a picture of the data. Each format will add insight to the results. Graphs show numbers in a purer form; writing helps explain and interpret the numbers.

Keep the writing short and sweet. Succinctly report the results and draw any conclusions that are supported by the data. You may find it useful to periodically add in snippets of open-ended answers, chosen to give a representative flavor.

Graphics will most likely be split between charts and cross-tabs. The charts may be pie graphs, bar graphs, line graphs, area charts, or any other form of representation that will aid people in understanding the data. Some forms lend themselves better to one type of data than another. For example, you would use a pie chart to show percentage breakouts, but bar graphs better compare raw numbers to each other.

Cross-tabs are a table, and so show how values in rows compare to values in columns. Unlike most tables, though, they add a third factor to the cells, letting you show another degree of relationship. An example might have store locations plotted across the top, sales by category labeling the rows, and a further breakout by age of customer in the cells.

SurveySolutions can help you create reports and presentations of the results, saving extensive preparation time.

Numbers themselves can mislead. A perfectly acceptable sounding analysis might fall apart when you realize that you had some outlying data - statistical aberrations that skew your results. Remember that numeric analysis involves a reduction of information to a comprehensible form. Anytime you pour a gallon container into a pint bottle, you lose liquid. Don't assume that the liquid is still there.

Graphs add entirely new worlds of misconception. Each time people look at a figure, they make assumptions. Lines that go up are good, lines that go down are bad, and both bring assumptions of time. Pie charts are interesting to look at, even if there is no information indicating what the pie as a whole represents. Chart formats can help you make a point or work against you. They can lull many into dullness and hide critical lacks in information. Be exceptionally careful when you use charts, that each describes what you intend and doesn't imply something else.

Survey Mistakes

The last stand for error in surveys is during analysis. These will lead to bad decisions and potentially make you look foolish at the same time, so search for them even harder than you would track grammatical errors and misspellings in an important memo.

One pernicious mistake is assuming that association is the same as cause. It may be true that older consumers tended to spend more with you, but that is far from saying that older consumers are necessarily the best for you to approach. Were there other factors at work? How profitable were the customers? Did service issues eat up the extra income? Surveys can add insight, but as we have said before, they are no substitute for smart decision making.

Another problem is analyzing returns that are too small. Any time you look at a subset of your sample, you need at least 30, and preferably more, responses to make any reasonable statements.

Take care. Many supposedly experienced market researchers are over-eager to analyze and make assumptions, running off at the mouth when their survey results should have them lying down until the fit passes.

Presenting Persuasive Results

The entire reason for surveys is to gather information for better decision making. Decisions however are useless if action does not follow. Action requires persuasion, and getting people to understand the need for your recommendations and decisions. Treat the presentation of the results and your conclusions as you would any important communication.

Here are some steps to successful and persuasive presentations:

• Gauge your audience and tailor the presentation to meet their knowledge and concerns. It if doesn't interest them, nothing will come of it.

• Present information and conclusions, not wishful thinking. Have the information to back up your conclusions and suggestions. If you lose others on any one point, they will distrust the entire survey.

People more easily agree to what they want to hear, but covering up unpleasant news will become disaster. Better to air the dirty laundry before washing day.

As part of your report, plan on the necessary further work. Develop an actionable strategy to meet the objectives you started with. If no action will result from the survey, then you have wasted your time.

Note that further work may mean further study. The survey process can be iterative. What you learn in one study can better inform the next. After all, for the questioning company, there is no final answer, just smarter questions.