Search engines can't keep up

 作者:鄢摺     |      日期:2019-03-08 02:13:00
By Kurt Kleiner in Washington DC The amount of information on the World Wide Web is outstripping the ability of search engines to index it, according to a new study. There are about 800 million publicly available pages on the Web, but even the best search engines index only about 16 per cent of them. Two years ago, the best engines indexed a third of the 320 million pages that were on the Web (This Week, 28 June 1997, p 12). Steve Lawrence and C. Lee Giles of the NEC Research Institute in Princeton, New Jersey, arrived at their estimate of the size of the Web by sampling all possible Internet addresses to see which ones actually represented servers, then sampling servers to see how much information was available. They found 2.8 million servers with an average of 289 publicly available pages each. But when they submitted test queries to the 11 most popular search engines, they found that the vast majority of pages out there simply didn’t show up. The news is a little better if you conduct a metasearch, submitting the same query to as many different search engines as possible. Together, they cover about 43 per cent of the Web. There’s more bad news—even when the search engines do turn up new pages, it can take them months to do it. The average time between a new page going up and the time it shows up on a search engine is six months, the pair report (Nature, vol 400, p 105). Some hope that a new Web language using metatags, which show what kind of information each page contains, will make indexing easier. However, the researchers found that only 34 per cent of all pages use the simple metatags that are already available, making it unlikely that many will trouble to use the more complex language. The good news for scientists is that only about 6 per cent of the material on the Web is scientific. Since that’s only about 4.8 million pages, it’s conceivable that a specialised search engine could maintain a complete index of all the science information out there,