The fact that your Google search results are based upon a very tiny coverage of the web is startling to many. I have heard that the Dark Web, this is the dark, unsearched, un-catalogued parts of the web which may be…well over 80% of what is theoretically accessible. I have heard that is may be closer to 99%…
In the last several years Google has been spending a lot of money trying to encapsulate as much of this ‘dark matter’ on the internet. They have some quite obvious reasons for this (they generate a lot of money from search results, and since they have (at best) 25 billion sites catalogued to some degree, the remaining75 -100 billion other sites are a huge potential for them…to lose search hegemony.
Much of this underworld of the web is filled with Spam, WAREZ sites, and even organized crime and terrorism communications avenues…
This is an interesting enough perspective (that we have a huge ‘unconscious’ part of our mutual online world), but it is also a great topic to justify taking some small pains to maintain your own security here on the internet…
The Unlit Social Graph
http://thenextweb.com/location/2011/04/06/the-unlit-social-graph
Flash forward to 2011. We are now witnessing a similar dynamic, but the battlefield has shifted from search to social. Just as Google had early dominance in lighting up a portion of the web, Facebook has early dominance in lighting up a portion of the world’s social graph. But much like the Dark Web, there exists network upon network not yet graphed by Facebook, waiting to be mapped, organized, and optimized for communication.
This is the unlit social graph, and this is where Facebook is vulnerable.
Let’s talk examples.
For years I have been looking for a solution to the pick-up basketball problem. I have a large-ish network of people that I play hoops with in San Francisco. This network has not yet been lit up by any online service. While most of these people are on Facebook, they are hard to organize as I don’t know many of their last names. And furthermore, even if I did know their last names, I would feel awkward friending them on Facebook, as they’re not really my friends.
The Dark Web Explained
http://witnessthis.wordpress.com/2009/12/14/the-dark-web-explained-2
How is the Deep Web different from the Surface Web?
To put it very simply, the web is defined as a collection of hyperlinks that are indexed by search engines. In other words, the pages/content that appears when we do a Google search is the Internet as we know it – called the surface web.
The Dark Web, also known as the deep web, invisible web, and dark net, consists of web pages and data that are beyond the reach of search engines. Some of what makes up the Deep Web consists of abandoned, inactive web pages, but the majority of data that lies within have been crafted to deliberately avoid detection in order to remain anonymous.
The dark side of the internet
http://www.guardian.co.uk/technology/2009/nov/26/dark-side-internet-freenet
Spidering the "Dark Web"
http://www.readwriteweb.com/archives/spidering_the_dark_web.php
The Dark Web Project
Federally funded through the National Science Foundation, the Dark Web’s spiders have been crawling through the web for the past five years. As of 2007, they estimated there were about 50,000 sites of extremist/terrorist content when they looked beyond just traditional web pages. This number was a great increase from Dr. Gabriel Weimann of the University of Haifa’s estimate that there were only 5000 terrorist web sites in 2006. From 2006-2007, the lab found the greatest increase in terrorist activities was on various new "web 2.0" sites, (a term they use to describe any new-generation web site including video sites, blogs, virtual worlds, etc.)
Currently, the Dark Web collection consists of the complete contents of only 1000 web sites in Arabic, Spanish, and English and the partial contents of 10,000 other sites. This collection is 2 TBs in size making it the largest open-source extremist/terrorist collection in the academic world. Researchers who would like to use this data in their own studies can contact the research center for access.
The Dark Web: 500x Larger than the World Wide Web
http://www.neatorama.com/2009/11/28/the-dark-web-500x-larger-than-the-world-wide-web
The World Wide Web is big, really big. As of July of 2008, Google found 1 trillion (that’s 1,000,000,000,000) unique URLs on the web at once. The search engine has only indexed a fraction of those web pages (the last count I found was 25 billion in 2006).
But that’s nothing compared to the "Deep Web”, a part of the Internet that is not easily accessible by search engines (for example, dynamically generated content that exists only momentarily). People have estimated that the Deep Web is several orders of magnitude larger than the "surface Web". There is, however, another part of the Deep Web that is more sinister: the dark side of the Internet used by criminals.
