In a world deluged with data and stats, a few takeaways from this petite little book that is an absolute delight to read…
Like the “little dash of powder, little pot of paint,” statistics are making many an important fact “look like what she ain’t.” A well-wrapped statistic is better than Hitler’s “big lie”; it misleads, yet it cannot be pinned on you.
That average American…
Similarly, the next time you learn from your reading that the average American (you hear a good deal about him these days, most of it faintly improbable) brushes his teeth 1.02 times a day – a figure I have just made up, but it may be as good as anyone else’s – ask yourself a question. How can anyone have found out such a thing? Is a woman who has read in countless advertisements that non-brushers are social offenders going to confess to a stranger that she does not brush her teeth regularly? The statistic may have meaning to one who wants to know only what people say about tooth-brushing but it does not tell a great deal about the frequency with which bristle is applied to incisor.
A biased sample? No way…
A psychiatrist reported once that practically everybody is neurotic. Aside from the fact that such use destroys any meaning of the word “neurotic,” take a look at the man’s sample. That is, whom has the psychiatrist been observing? It turns out that he has reached this edifying conclusion from studying his patients, who are a long, long way from being a sample of the population. If a man were normal, our psychiatrist would never meet him.
More bias and now with stratified random sampling…
The purely random sample is the only kind that can be examined with entire confidence by means of statistical theory, but there is one thing wrong with it. It is so difficult and expensive to obtain for many uses that sheer cost eliminates it. A more economical substitute, which is almost universally used in such fields as opinion polling and market research, is called stratified random sampling.
To get this stratified sample you divide your universe into several groups in proportion to their known prevalence. And right there your trouble can begin: Your information about their proportion may not be correct. You instruct your interviewers to see to it that they talk to so many blacks and such-and-such a percentage of people in each of several income brackets, to a specified number of farmers, and so on. All the while the group must be divided equally between persons over forty and under forty years of age.
That sounds fine but what happens? On the question of black or white, the interviewer will judge correctly most of the time. On income he will make more mistakes. As to farmers – how do you classify a man who farms part time and works in the city too? Even the question of age can pose some problems which are not easily settled by choosing only respondents who obviously are well under or well over forty. In that case the sample will be biased by the virtual absence of the late-thirties and early-forties age groups. You can’t win.
On top of all this, how do you get a random sample within the stratification? The obvious thing is to start with a list of everybody and go after names chosen from it at random: but that is too expensive. So you go into the streets – and bias your sample against stay-at-homes. You go from door to door by day – and miss most of the employed people. You switch to evening interviews – and neglect the movie-goers and night-clubbers.
So getting a truly stratified random sample will cost you an arm and a leg. But not for the Facebooks and the Googles and the Snapchats of the world and hence their valuation story. Because as they say, data is the new oil.
That Literary Digest error…
You have pretty fair evidence to go on if you suspect that polls in general are biased in one specific direction, the direction of the Literary Digest error. This bias is towards the person with more money, more education, more information and alertness, better appearance, more conventional behavior, and more settled habits than the average of the population he is chosen to represent.
You can easily see what produces this. Let us say that you are an interviewer assigned to a street corner, with one interview to get. You spot two men who seem to fit the category you must complete; over forty, and urban. One is in clean clothes, neat. The other is dirty and he looks surly. With a job to get done, you approach the more likely-looking fellow, and your colleagues all over the country are making similar decisions.
The p-value implication…
How can you avoid being fooled by unconclusive results? Must every man be his own statistician and study the raw data for himself? It is not that bad; there is a test of significance that is easy to understand. It is simply a way of reporting how likely it is that a test figure represents a real result rather than something produced by chance. This is the little figure that is not there – on the assumption that you, the lay reader, wouldn’t understand it. Or that, where there’s an axe to grind, you would.
The collision of statistics with the human mind…
If you can’t prove what you want to prove, demonstrate something else and pretend that they are the same thing. In the daze that follows the collision of statistics with the human mind, hardly anybody will notice the difference. The semiattached figure is a device guaranteed to stand you in good stead. It always has.
You can’t prove that your nostrum cures colds, but you can publish (in large type) a sworn laboratory report that half an ounce of the stuff killed 31,108 germs in a test tube in eleven seconds. While you are about it, make sure that the laboratory is reputable or has an impressive name. Reproduce the report in full. Photograph a doctor-type model in white clothes and put his picture alongside.
But don’t mention the several gimmicks in your story. It is not up to you-is it? – to point out that an antiseptic that works well in a test tube may not perform in the human throat, especially after it has been diluted according to instructions to keep it from burning tissue. Don’t confuse the issue by telling what kind of germ you killed. Who knows what germ causes colds, particularly since it probably isn’t a germ at all?
On comparing apples to oranges…
The death rate in the Navy during the Spanish-American War was nine per thousand. For civilians in New York City during the same period it was sixteen per thousand. Navy recruiters later used these figures to show that it was safer to be in the Navy than out of it. Assume these figures to be accurate, as they probably are. Stop for a moment and see if you can spot what makes them, or at least the conclusion the recruiting people drew from them, virtually meaningless.
The groups are not comparable. The Navy is made up mainly of young men in known good health. A civilian population includes infants, the old, and the ill, all of whom have a higher death rate wherever they are. These figures do not at all prove that men meeting Navy standards will live longer in the Navy than out. They do not prove the contrary either.
A causes B or B causes A? Or C causes both A and B?
Someone once went to a good deal of trouble to find out if cigarette smokers make lower college grades than non-smokers. It turned out that they did. This pleased a good many people and they have been making much of it ever since. The road to good grades, it would appear, lies in giving up smoking; and, to carry the conclusion one reasonable step further, smoking makes dull minds.
This particular study was, I believe, properly done: sample big enough and honestly and carefully chosen, correlation having a high significance, and so on.
The fallacy is an ancient one which, however, has a powerful tendency to crop up in statistical material, where it is disguised by a welter of impressive figures. It is the one that says that if B follows A, then A has caused B. An unwarranted assumption is being made that since smoking and low grades go together, smoking causes low grades. Couldn’t it just as well be the other way around? Perhaps low marks causes students not to drink but to tobacco. When it comes right down to it, this conclusion is about as likely as the other and just as well supported by the evidence. But it is not nearly so satisfactory to propagandists.
It seems a good deal more probably, however, that neither of these things has produced the other, but both are a product of some third factor. Can it be that the sociable sort of fellow who takes his books less than seriously is also likely to smoke more? Or is there a clue in the fact that somebody once established a correlation between extroversion and low grades – a closer relationship apparently than the one between grades and intelligence? Maybe extroverts smoke more than introverts. The point is that when there are many reasonable explanations, you are hardly entitled to pick one that suits your taste and insist on it. But many people do.
So you are an online e-tailer and you think that the more information you provide to a customer about a given product he or she is contemplating purchasing, the less are the chances of that product being returned. So you slowly start ramping up the information content you provide to customers about the products you have on your platform and sure as hell, the customer return rate starts to go down, down, down until…until you build in so much expectations about the products that once the customers receive them, they feel like meh and then they start to return them. So the return rate declines initially with more information but at some point, it reverts and starts to go up. This is not a theory. This is a proven fact.
Another thing to watch out for is a conclusion in which a correlation has been inferred to continue beyond the data with which it has been demonstrated. It is easy to show that the more it rains in an area, the taller the corn grows or even the greater the crop. Rain, it seems, is a blessing. But a season of very heavy rainfall may damage or even ruin the crop. The positive correlation holds up to a point and then quickly becomes a negative one. Above so-many inches, the more it rains the less corn you get.
I hate statisticulation…
Statisticulation, short for Statistical Manipulation – misinforming people by the use of statistical material.
An art or a science…
The fact is that, despite its mathematical base, statistics is as much an art as it is a science. A great many manipulations and even distortions are possible within the bounds of propriety. Often the statistician must choose among methods, a subjective process, and find the one that he will use to represent the facts. In commercial practice he is about as unlikely to select an unfavorable method as a copywriter is to call his sponsor’s product flimsy and cheap when he might as well say light and economical.
Sharpening your statistical eye…
This suggests giving statistical material, the facts and figures in newspapers and books, magazines and advertising, a very sharp second look before accepting any of them. Sometimes a careful squint will sharpen the focus. But arbitrarily ejecting statistical methods makes no sense either. This is like refusing to read because writers sometimes use words to hide facts and relationships rather than to reveal them.
How To Look A Phony Statistic in the eye and face it down?
Is the reported statistic inherently biased based on who is reporting it? Whose interest does it serve? Look for conscious bias. Reporting of favorable data and suppression of unfavorable. Shifts in units of measurements. The average they report…is that a mean, median or mode. Based on what you want to prove, this can be used interchangeably.
Look for evidence of biased samples – one that has been selected improperly or has selected itself. Look sharply for unconscious bias – the more dangerous kind.
Causes and effects. Correlation vs causation.