Google Ngrams is a great resource for provenance tracking of words and phrases in published English books. It is essentially a histogram computer for frequency of word use over the date which allows comparison among a number of words/phrases.

That is purely a search on books in Google's scanned catalog. Is there any similar app to do the comparison on words that are on web pages?

I noticed the lack of such a tool when I went to compare two words in order to answer a question. Google ngrams gave zero hits, which surprised me because others were referring to a number of google hits. Then I realize they were using the regular search for that. But there was no way to do a comparison.

I understand that there's a difficulty with -dating- an occurrence of a word, but one could at least return just the number of hits -now- for a set of words/phrases (instead of doing each search separately and recording the number of hits for each one).

So is there a tool like Google Ngrams for the words on web pages?

Google NGrams should be used with caution, because it's based on, essentially, OCR data. –  Marthaª May 24 '11 at 3:28
@Martha: Yes, the trick is to check as many of the references as they give. –  Mitch May 24 '11 at 3:32
Also, the metadata for a lot of the Google Books stuff is way off. So for dates and language/dialect of origin, you have to be wary also. –  Kosmonaut May 24 '11 at 3:56

1 Answer 1

up vote 7 down vote accepted

Yes, Google has the raw data for this as well: the Web 1T 5-gram, but it costs money.

You must either pay for access to the Linguistics Data Consortium where you can get this data, or else pay $150 to get it outright (note: actually, for this particular data, it looks like you don't get an LDC discount, so no reason to join).

This is synchronic data, and not diachronic as with the Google Books ngrams.

But like I said in the comments, there is a ton of metadata error in Google Books — enough to possibly be concerned about a lot of the conclusions one might draw from it. As it happens, I am working on a project that was originally going to use some Google Books data, but in-depth analysis seems to indicate that dates are way off (as in, 25% of the pre-1800 tokens I have looked at so far seem to be off on their publication dates by an average of ~100 years). Still some work to do to make sure this is really the case before publishing it though.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .