Wow @ Google
Frabjous day!
This isn’t directly related to Blogamundo, but, uh, it’s just too awesome not to post about:
Official Google Research Blog: All Our N-gram are Belong to You
We believe that the entire research community can benefit from access to such massive amounts of data. It will advance the state of the art, it will focus research in the promising direction of large-scale, data-driven approaches, and it will allow all research groups, no matter how large or small their computing resources, to play together. That’s why we decided to share this enormous dataset with everyone. We processed 1,011,582,453,213 words of running text and are publishing the counts for all 1,146,580,664 five-word sequences that appear at least 40 times. There are 13,653,070 unique words, after discarding words that appear less than 200 times.
Things I’m wondering right off the top of my head:
- Is it gonna cost… like… a lot?
- Please, FSM, let it not be a trillion woirds of English.
- Many cheers for the Linguistic Data Consortium. (Perhaps someone at Google took Mark Liberman’s open letter to Microsoft to heart?)
The first project that comes to my mind would be to try to sample writing systems—what the Unicode folks call scripts. It’s a given that most text on the web is in the Latin script or its derivatives, and second place could very well be Chinese (but which encoding?)… but then things get interesting.
Just how much Ethiopic is out there? Georgian? Persian? Tifinagh?
A trillion words.
A TRILLION!
That’s even comparable to the US debt!
2 comments.
Technorati tags: No Tags
Mark Leiberman could put his money where his mouth is and make the UPenn resources a lot more accessible.
I think that’s a pretty cheap shot, anonymous. It’s not like hosting is free; the LDC has to keep its head above water somehow. Yes, it would be great if huge corpora were free on the web.
Why not criticize Google for not opening up their database to all comers?