"There are three kinds of lies: lies, damned lies, and statistics." — Benjamin Disraeli, attr. by Mark Twain
"If you torture the data long enough, it will confess." — Ronald Coase

For some time now, contributors to EL&U have offered NGrams in support of their arguments. Now, there is nothing wrong with this practice per se: I have done so myself, and have seen others do it in a way that acknowledges the margin for error inherent in a flawed system. When done well it is done in a spirit of inquiry, citing the NGram as possible evidence; when it is done poorly, it is trumpeted as absolute proof of someone's contention.

How flawed is the system? According to @Kosmonaut (replying to a different meta question),

[T]here is a ton of metadata error in Google Books — enough to possibly be concerned about a lot of the conclusions one might draw from it. As it happens, I am working on a project that was originally going to use some Google Books data, but in-depth analysis seems to indicate that dates are way off (as in, 25% of the pre-1800 tokens I have looked at so far seem to be off on their publication dates by an average of ~100 years).

There is also the matter of comparison terms. Sometimes one can find comparisons that will work, but very often the terms can't be compared exactly. For example, in this answer one of our high-rep users attempts to adduce an NGram to answer a question about nuances of meaning:

enter image description here

One wonders how relative frequency is germane to any discussion of the meanings of those words.

Another flaw involves book results being used to draw inferences about spoken language as well, or at least to conflate the two. In this answer [comment chain since then has been deleted for unclear reasons], another contributor uses NGrams to show that one usage is vastly more common than another, which obviously feels counter-intuitive to him because he admits in the comment that "FWIW, it surprised me too. I would probably say 'go for a swim' myself." The conclusion I draw is that Google NGrams are a hammer looking for a nail.

Here is another case where someone draws a faulty inference based on an NGram search:

enter image description here

I leave it to the eloquent @MrHen to debunk the chart:

How does that NGram support any particular usage? Isn't it just tracking uses of "a week hence"? How would it know if that hence was for the future or not?

Even when they can define their terms, users of NGrams succumb to the simpleton awe inspired by the sight of a dramatic projection of minuscule data points to show great trends and differences. They ignore the scale on the Y-axis, which reports the differences in a range that may be only a few cases in tens of millions.

And very often NGram answers garner upvotes because people don't think about the data behind the charts and simply look at the pretty line graphs and upvote the answer.

I'm not against Google NGrams. Used intelligently, they can offer some valuable information. But I believe them not to be the unimpeachable source people pretend they are on EL&U. I'm asking for some kind of standards. I don't know what kind, but I hope you all have some suggestions.

ADDENDUM

As if to prove my point, we have another NGram travesty "proving" a point with faulty data. Because of a bug (or quirk, if you will) in the Google NGram Viewer, hyphenated words invariably flatline. A hyphenated word needs to be separated into a trigram by adding spaces around the hyphen. Notice the difference between these two charts:

enter image description here

enter image description here

Sorry I switched the colors. But at least I resurrected "upper-case" from oblivion. But there is another flaw in this chart — can you spot it? How about cases where "upper case" is not used as an adjective describing a letter? Or how about cases where, even though the subject is typography, one is trying to use case as a noun: "He used the upper case quite a bit in his emails." Of course, the author's point about usage is probably in the main supportable, even without reference to these charts. But given that these charts were introduced as evidence, the answer is not really accurate, and accuracy is something we should at least strive for on this site. We aren't always 100% accurate — nobody's perfect — but if we passively give the nod to glaring errors, relying instead on the base feeling that whatever garners the most votes must be true, we become little better than the Urban Dictionary.

MORE

Finally, for your amusement and to show the relative scale of usage we're talking about, consider the following series of charts. Note especially how one single high-frequency word causes all the rest to flatline. And none of the other words may be considered obscure in the least! (Also, it is obvious from an NGram search that apples are better than oranges. Or that people like them more. Or that ... well, draw your own conclusions. The data are there for your interpretation.)

enter image description here

enter image description here

enter image description here

enter image description here

Use of "is" in the corpus peaks at 1% — one out of a hundred words. Use of those other common words is negligible by comparison. And use of obscure word combinations may not even be worth talking about from a genuine statistical point of view. If there is a statistician among you who can define the terms for such a discussion, I am open to being educated. Until then, I will look at answers involving NGrams with a jaundiced eye.

Update

See this question and its ham-fisted NGrams, some of which have already been deleted, for more evidence of NGrams Gone Wild. At one point, two answers used the same query term and arrived at opposite conclusions. You can't make this stuff up, folks.

FULL DISCLOSURE

Even I have not been immune to the lure of Google NGrams in the past. @Mechanicalsnail pointed out to me an instance of my own seduction, which was written before I had fully grokked how NGrams should and should not be used.

share
    
@Robusto I just noticed this question on Meta English SE: meta.english.stackexchange.com/questions/2325/… Content is specifically about Google Ngram viewer usage, and the single (up voted) answer states that providing a link to the ngram itself is unnecessary (I think the exact word was "irrelevant"). –  Feral Oink May 26 '12 at 13:09
1  
I'll admit I didn't read the whole post, but my major concern is that via some hive groupthink, something that is already popular will be chosen simply out of its popularity and no other merits. –  shinyspoongod Jun 20 '12 at 0:21
    
You said it all in this statement: There is nothing wrong with this practice per se. End of story. Ngrams are what they are. This meta question can serve as a useful caveat about their use, so it is worthwhile. Otherwise, let it go - no need to police the use of ngrams or provide guidelines/caveats everywhere. We should not define a set of standards for their use. Users can call out specific ngrams that are misleading or point out specifically what a given ngram does and does not indicate/support. –  Drew Sep 17 at 17:59
    
@Drew: I think anyone who presents them as statistical evidence should be required, as I suggest in the title of my post, to qualify their use. That to me says it all. –  Robusto Sep 17 at 18:33
    
@Robusto: Just what do you mean by "presents them as statistical evidence"? Any posting of an ngram might be construed by some as presenting statistical evidence, even without any explicit argument claiming that. Better to let it be and leave it up to S.E. users to point out when something is misleading in a particular context. That, plus providing this question as general background. –  Drew Sep 17 at 20:29

5 Answers 5

up vote 14 down vote accepted

I love the Google Ngram viewer. And I am a statistician, by education and profession. I also agree with Robusto, regarding the tendency for misuse of Ngrams.

Something to keep in mind: This is English Language and Usage SE. We are not linguists. Well, a few of us are. Yet the only one, that I know of off-hand, doesn't often use Ngrams to support his answers. The Google Ngram viewer makes it very easy to bludgeon other respondents' answers into wrongness (I'm sorry, that is a terribly phrased sentence). I specifically refer to this thread.

I was thinking about the hazards of subjectivity, and moral relativism recently. When considering grammar and standard English usage, we accept variation from an absolute "right" or "wrong". But we do this logically, with rules, such as use of tags e.g. "British English" or "American English" or "slang". This accommodates the work in progress that is language.

There are grammatically and idiomatically correct versus incorrect answers to questions on EL&U SE. But unstructured text analysis and other quantitative methods have so much of an aura of authority in an age of reason that they tend to overwhelm everything else. Barry often cites ODE. That is a very good reference source. But it isn't as flashy or colorful as a festive multi-colored line graph. I don't like to see incorrect answers receive more votes, or be designated as the accepted answer, when there are concise, correct answers submitted too.

Okay, exposition over. Here's an answer to the question.

IF (an Ngram is used to answer a question on this site) 
    THEN ( [the Ngram must be accompanied by a paragraph of prose explanation] 
           AND [the Ngram must comply with validity criteria] )

Validity criteria should include, at a minimum: Only data between the years 1800 and 2000 allowed, per the Google ngram website warning. (I can get the URL for that if needed).

How do we enforce this? On a case-by-base basis, I guess.

Final thought

It would be helpful if we could systematically limit Ngram usage on a per user basis. How much is too much? I don't know. IF it could be done, THEN we could address the matter of frequency, both value and periodicity. (See, I'm already quantifying this to excess...)

share
    
Excellent way to say it, which would apply to anything; using any tool should come with an explanation. Your 'final thought' however is a bit too authoritarian (and I would suppose difficult to implement). –  Mitch Mar 4 '12 at 18:33
    
Thank you @Mitch! That did sound bossy, didn't it? I wouldn't want to ask the crowd over on Meta StackOverflow if that could be implemented. They are very tolerant of fine tuning and "features" but an Ngram usage cap is a little much ;o) –  Feral Oink Mar 4 '12 at 18:44
    
Post-final thought: I think that should be OED for Oxford English Dictionary instead of ODE... Sorry about that! –  Feral Oink Mar 4 '12 at 18:45
2  
This is exactly the sort of clear analysis I was looking for. Thank you. Can you elaborate a bit on what criteria we might use to assess validity? That's what we really need. In other words, going beyond a minimum? –  Robusto Mar 4 '12 at 18:51
    
@Robusto You are welcome! Regarding additional criteria, errr, yes. The first that came to mind was medial s, or rather, avoiding situations where it could cause invalid results. That is too technical, to say the least. I wrote (casually!) about ngrams on a different site. I'll try to find that, and will post here if I find anything useful. –  Feral Oink Mar 9 '12 at 22:27
1  
I would like to know what you are referring to as the "Google ngram website warning." –  Callithumpian May 27 '12 at 15:18
    
@Callithumpian Yes, certainly. I will get the link and section for that and post here. I keep meaning to respond to your comment, sorry for delaying this long! –  Feral Oink Jun 17 '12 at 18:46
    
I hate statistics with a passion, mostly because after careful calculation they're almost always misinterpreted so horribly, so I'm glad we have people like you who can suffer through the worst of at least one part of the process for us. –  shinyspoongod Jun 20 '12 at 0:23
    
@shinyspoongod You are very sweet! As you said, it is the misinterpretation that is the problem. –  Feral Oink Jul 16 '12 at 3:58
    
@Callithumpian Here you go "Watch out for the time period your are looking into: the best data is the data for English between 1800 and 2000. Before 1800, there aren’t enough books to reliably quantify many of the queries that first come to mind; after 2000, the corpus composition undergoes subtle changes around the time of the inception of the Google Books project." Source: Developers of Google N-Gram Viewer, URL is culturomics.org/Resources/A-users-guide-to-culturomics –  Feral Oink Jul 16 '12 at 4:01
    
@Callithumpian The Culturomics URL is provided on this "official" Google Ngram info page books.google.com/ngrams/info Look about half way down, after the charts. It is the suggested resource for further info. –  Feral Oink Jul 16 '12 at 4:10

I think you made the point when you wrote that they can be "presented as statistical evidence without qualification". Google NGrams can provide important information, but they lack authority.

Why? They are not 100% reliable and if we add this to the fact that many people present it as something "absolute", then it could lead to misinformation, at best.

My suggestion is to accept them but with reservation. They can be used, as they still can provide some usefulness, but preferably along with something else more reliable to back up the information.

I think I addressed everything, but if I forgot something, let me know.

Edit: As suggested from the comments, here are the NGrams.

The first one shows that my conclusion was not that convincing:

enter image description here

While any counter argument, except for a moment, were far away from mine:

enter image description here

share
4  
You forgot the NGram supporting your conclusion. –  KitFox Feb 27 '12 at 17:38
2  
@KitFox Done! :D –  Alenanno Feb 27 '12 at 17:49
    
Small point: I did not say "Google NGrams are useful, but up to a certain point" in my question title or anywhere else. –  Robusto Feb 27 '12 at 19:06
    
@Robusto It wasn't meant to be a verbatim quote, but rather a re-wording of what I think you meant. You meant that, no? –  Alenanno Feb 27 '12 at 19:08
2  
@Alenanno: Well, words either mean something or they don't. I'm not channeling any negative energy in your general direction, but I thought I did say what I meant in the title. It would be more accurate to characterize the general sense of my question as that, but not to attribute those exact words to me. –  Robusto Feb 27 '12 at 20:11
    
@Robusto I can re-word it if you want, but I have no idea how... –  Alenanno Feb 27 '12 at 20:12
    
@Alenanno: I edited your first graf to be what I would call an accurate representation of something you could say. If this works for you, good. If not, you are of course free to say whatever you like. –  Robusto Feb 27 '12 at 20:19
    
@Robusto I re-re-word it. :) Now I really quoted your words. :P –  Alenanno Feb 27 '12 at 20:35
1  
OK, now that that's over with, let me address your point. How do we "accept them, but with reserve"? (And did you mean "with reservation"?) Because unless there is some warning given about the inherent flaws, the average visitor will not know any of that and will fall under the spell of shoddy statistical mumbo-jumbo just the same. –  Robusto Feb 27 '12 at 21:02
    
Ah yes, I meant that. I didn't think about that, as you didn't ask to propose something (no?), but since the major problem is that the words are often searched like that without even considering the context where they were used, I think we could address this in the FAQ or in a FAQ-Meta-Question. The usage of NGrams here are basically "bad corpora searches"; "bad" because in the actual corpora search, you see the sentence where a certain word/expression appears. –  Alenanno Feb 27 '12 at 21:35
    
If Robusto is right, then counter-argument in your lower NGram needs to be entered as counter - argument (i.e. with a space surrounding the hyphen on each side) for the NGram algorithm to work correctly. –  Erik Kowal Jul 31 at 8:56

I agree that Ngrams can be used inappropriately. The resources that people use to support their answers all have to varying degrees their difficulties. But I think the existing methods of justifying them will work:

  • if there is a problem with the word choice, the results, whatever, the answerer should either point them out if they know
  • someone can comment on the failures of the usage
  • someone can edit the answer graph/link

But that's in general for any data (from OED, from quotes, whatever). Specifically for Ngrams, the problems that should be watched out for are:

  • dealing with punctuation and caps (as you pointed out in your example with the hyphen)
  • selecting the right pairs to compare (make sure they are comparable)
  • making sure the context is right, both semantically and in surrounding strings

Looking at the links to sources in the linked Ngram is really the only way to judge.

share
2  
All sounds good to me. NGrams can be misleading, but they're not that complicated - if someone posts up a grossly misleading graph, someone else will probably be falling over themselves to point out the flaws (Robusto himself, for example! :) –  FumbleFingers Feb 28 '12 at 2:32
7  
@FumbleFingers: I think Robusto's point is that it's too easy (and has been done too often) to thoughtlessly pop over to NGram, create a graph and insert it in an answer without thinking of all the issues. The author of the answer should go to the trouble first rather than rely on others to fix things up. –  Mitch Feb 28 '12 at 3:52
1  
Particularly over recent months, a huge percentage of questions asked here are really basic - oftentimes the questioner neither seeks, nor would understand, a full discussion of why some particular usage is to be preferred over another. And in any case, the reason is often complex, disputed, or simply nets down to "It's idiomatic". It's too easy (and has been done far more often) for people who aren't even native speakers to invoke "inappropriate logic" and post grossly misleading answers as to what constitutes "standard English as she is spoke". –  FumbleFingers Feb 29 '12 at 17:10

Should we allow Google NGrams to be presented as statistical evidence without qualification?

Should we define a set of standards for their usage?

We could define a set of standards, but what would those standards be? Your question contained a whole bunch of instances where the data could be mistaken or mis-construed, and I agree with them. But how do you translate those problems into a set of standard for their usage?

Disallowing them seems a bit extreme. As with any answer, if you feel that it's citing dubious sources or providing incorrect information, comment and down-vote. It seems better to combat the problem through the voting system and education rather than regulation.

share

Selection bias and Semantics

Statistics over published matter, whether books, newspapers, internet should always be taken with an iceberg of salt.

Selection bias: have a go at "bull" vs "cow" and "cherry" vs "plum" and try to explain the crazy oscillations before year 1800. What you'll see is an example of either insufficiency of printed material or periods when the earlier in each pair bore sexual connotation.

Semantics: consider a query "speakeasy" vs "bar" for a drinking establishment. The earlier became commonplace word since prohibition, that is 1920's, yet curiously same word was used in 1880's. The latter can be used in dozens of senses, from law bar to crowbar, assigning statistical prevalence of "bar" over "speakeasy" to "bar" in a sense of a drinking establishment is completely wrong. Same goes for any query where one option is shorter than other or was ever used to mean anything else.

share

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .