|
 |
|
| A
news reporter states the president has 65%
support among voters. In fine print the report also shows the sampling
error is +/- 1%. The +/-
1% sampling error, or margin of error, implies (assuming the standard
95% confidence level) that a sample of n=8750 voters was used to form
the 65% estimate. By anyone's standards, +/- 1% represents a very
low degree of sampling error and thus a very high degree of statistical
reliability.
Changing channels, you hear from another reporter that the president has 35% support among voters. This second report states a sampling error of +/- 8%, which arises from a sample size of n=135. This second estimate of 35% support is decidedly less reliable, statistically speaking, than the former estimate of 65% support. Does this mean you should rely on the 65% number as the better of the two estimates and ignore the second estimate of 35%? Yes, all else being equal. But all else usually is not equal and herein lies the problem. What if you were told that the first sample of 8750 voters had been taken among only poorer, older voters from the president's own home state? …and that the second sample, albeit smaller in size with only n=135, was precisely proportionate to the United States as a whole with regard to gender, age, income, race, and state of residence? Clearly the second estimate of 35%, yielded from the more balanced sample design, is the more valid result, the more accurate answer. You should rely on 35% +/- 8%, and not 65% +/- 1%, as being the better estimate of the true level of support for the president, regardless of the lower level of statistical reliability associated with the 35% estimate. |
The above is merely an example. Think about your own research. When you ask yourself or your research consultant if the results will be reliable (in the everyday lay sense meaning both statistically reliable and valid) is the response merely in terms of statistical reliability or does the response draw a distinction between reliability and validity? Be wary of the typical consultant response "Yes, the results will be very reliable." That consultant may simply be reporting that the sampling error will be low. This miscommunication trap is all too prevalent in the research consulting industry.
Low sampling error really has nothing to do with validity. In fact, if an answer is not foremost accurate (valid) then high statistical reliability (low sampling error) merely gives a false sense of security. You will be inclined to feel more confident about an inaccurate, wrong answer!
Accuracy (validity), unlike statistical reliability, is not a function of sample size. Instead, accuracy must be fostered into each research project. This fostering boils down to avoiding bias in its many forms, through careful attention to things such as:
As we seek truth, it is indeed accuracy (validity) that is of primary importance. While low sampling error (high statistical reliability) is certainly a priority, it should only be secondary to accuracy. In all we do, Accurate Analytics strives to assure first and foremost that your research results are accurate. The strength of our conviction in this philosophy is not only witnessed in our company name and Website domain name but also in our company slogan:
