OSTIblog Articles in the crawling Topic

The Science Knowledge Imperative: Making non-Googleable Science Findable

Just as science progresses only if knowledge is shared, accelerating the sharing of knowledge accelerates science. All of us engaged in disseminating science knowledge have the opportunity and obligation to do our jobs better, for to do so accelerates science itself. 

To this end, I propose a grand challenge--to make more science available to, and searchable by, more people than ever before. A momentous milestone will be achieved once we enable everyone with web access the ability to search with unparalleled precision a billion pages of authoritative science. Already, considerable progress has been made.  

My organization, the U.S. Department of Energy (DOE) Office of Scientific and Technical Information (OSTI) is responsible for the scientific and technical information operations of the Department. Over the last 11 years, OSTI has become entirely web-based. Of course, we are just one among many entities who connect people to knowledge using the web. Most notably, Google, Yahoo!, and other conventional search engine providers do this, too.

Google and other conventional search engines do for the web what publishers have long done for books--they create an index so that customers can quickly find information. Web users value this service so highly that search companies have become phenomenally successful enterprises. 

However, an important misunderstanding has sprung up about Google and the others. That is, the false presumption, especially among young people, that most useful information is available via conventional search engines such as Google and Yahoo!

In fact, much of the information on the web is inherently unavailable to Google and Yahoo! This key limitation would come as a surprise to...

Related Topics: crawling, E-Print Network (EPN), federated search, google, science, Science.gov, WorldWideScience.org (WWS)