Brisbane Search Engine Optimization » SEO « Training - Advice - Tools - Resources » robot

Google Indexing Robots.txt Files? - Advanced Forum Thread

admin — Mon, 10 Nov 2008 11:05:10 +0000

Google Indexing Robots.txt Files? - Advanced Forum Thread
In today’s forum thread of the week, it was actually ME who had a question! I noticed that our robots.txt file had been indexed by Google and was curious if others had ever had this happen.

Blocking Complicated URLs with Robots.txt

admin — Thu, 06 Nov 2008 15:05:05 +0000

Blocking Complicated URLs with Robots.txt
If you have a large web site you might have some content that you do not want the search engines to index — perhaps for duplicate content reasons or you simply don t want someone casually stumbling across it. You know you can use robots.txt but what if you need to block thousands of pages or block only certain files within a folder This article will explain some of the more advanced uses of robots.txt. You will even learn how to block dynamic pages …

Using Robots.txt to Control Search Engines

admin — Sun, 19 Oct 2008 00:05:07 +0000

Using Robots.txt to Control Search Engines
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt implements the Robots Exclusion Protocol, which allows you as a web manage…

What On Earth Is A Search Engine Algorithm?

admin — Wed, 18 Jun 2008 06:34:00 +0000

Each search engine has something called an algorithm which is the formula that each search engine uses to evaluate web pages and determine their relevance and value when crawling them for possible inclusion in their search engine. A crawler is the robot that browses all of these pages for the search engine.

The GOOGLE Algorithm Is Key

Google has a comprehensive and highly developed technology, a straightforward interface and a wide-ranging array of search tools which enable the users to easily access a variety of information online.

Google users can browse the web and find information in various languages, retrieve maps, stock quotes and read news, search for a long lost friend using the phonebook listings available on Google for all of US cities and basically surf the 3 billion odd web pages on the internet!

Google boasts of having world’s largest archive of Usenet messages, dating all the way back to 1981. Google’s technology can be accessed from any conventional desktop PC as well as from various wireless platforms such as WAP and i-mode phones, handheld devices and other such Internet equipped gadgets.