16

Google Ngrams is a great resource for provenance tracking of words and phrases in published English books. It is essentially a histogram computer for frequency of word use over the date which allows comparison among a number of words/phrases.

That is purely a search on books in Google's scanned catalog. Is there any similar app to do the comparison on words that are on web pages?

I noticed the lack of such a tool when I went to compare two words in order to answer a question. Google ngrams gave zero hits, which surprised me because others were referring to a number of google hits. Then I realize they were using the regular search for that. But there was no way to do a comparison.

I understand that there's a difficulty with -dating- an occurrence of a word, but one could at least return just the number of hits -now- for a set of words/phrases (instead of doing each search separately and recording the number of hits for each one).

So is there a tool like Google Ngrams for the words on web pages?

3
  • 3
    Google NGrams should be used with caution, because it's based on, essentially, OCR data.
    – Marthaª
    May 24, 2011 at 3:28
  • @Martha: Yes, the trick is to check as many of the references as they give.
    – Mitch
    May 24, 2011 at 3:32
  • Seems like Microsoft had something similar thing too web.archive.org/web/20100924093601/http://… .
    – Pacerier
    Oct 4, 2015 at 20:16

2 Answers 2

12

Yes, Google has the raw data for this as well: the Web 1T 5-gram, but it costs money.

You must either pay for access to the Linguistics Data Consortium where you can get this data, or else pay $150 to get it outright (note: actually, for this particular data, it looks like you don't get an LDC discount, so no reason to join).

This is synchronic data, and not diachronic as with the Google Books ngrams.

1
  • 1
    Without Google however, is there an alternative? If Google ngram is inaccurate, how would we overcome that limitation? How do you "touch up" that data?
    – Pacerier
    Oct 4, 2015 at 20:14
4

Some of the BYU corpora are web corpora. And the best part is that it's freee!

  • iWeb: The Intelligent Web-based Corpus "14 billion words from the Web"
  • News on the Web (NOW) "8.7 billion, Web news, 2010-last month"
  • Global Web-Based English (GloWbE) "1.9 billion, Web, 20 countries"
  • Wikipedia Corpus "1.9 billion, Wikipedia"
  • Corpus of Online Registers of English (CORE) "53 million words, Web genres"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .