The Invisible Web

For many of us, we peruse and search the internet using only our favourite search engine.

But did you know that by doing this, you are only covering a small portion of the web?

So, even though there may be thousands of websites popping up from your search, there could be even more vital websites you’re missing out on.

Basically, the internet can be classified into two categories – the surface web and the invisible web.

The surface web is what search engines index and dig up for us to see and what is easily found.

But the invisible web is an exception. Also known as the deep web, these websites are made to be much harder to find. They won’t show up in a typical search engine search, and if they do, they are hard to access.

These websites might be deliberately excluded by the owners using code, or the websites might be invisible because they don’t hold much significance.

It is expected that invisible web is several times bigger than the surface web, although it is hard to measure that isn’t clearly visible.

Amanda Spink, who is the professor of information technology at the Queensland University of Technology will be joining us on the program today to delve into the details of the invisible web and discover what kind of sites there are.

She says there are a number of websites you can’t access for many reasons. Organisations on the web might host their valuable company information online for easy accessibility for employees, but it usually restricted access with log-in protection.

Cyber criminals also convene online but these sites won’t be found easily.

These websites can be kept hidden embedding codes into the HTML, such as Meta Tags, which stop web crawlers from finding them.

Although there are some pages on the web that might not technically be made invisible; but become invisible as a result of narrow search techniques.

Amanda Spink says search engines don’t index every website and only cover a portion of the web. So one search engine might cover a portion of the web that another search engine might not.

There are ways to combat this and expand your searches, by using a unique and versatile type of search engine called metasearch engines such as Dogpile. These search engines are a quick and efficient way to do searches because they utilise multiple search engines.

Amanda Spink says the information on the invisible is in no way substandard to the information on the surface web. In fact the information is actually a lot more valuable.

So next time you do a search, keep in mind that there is more than meets the eye. What might be available on one search engine may not be available on another and the web stretches much further than what we normally see.

There are many other ways to do searches and get the most of what you’re looking for.

All you need is to find the right search tools, which may include federated search engines such as databases; or by using human crawlers instead of algorithmic crawlers such as StumbleUpon.