It tells a search engine that the current page can be indexed by the spider
and added to the search engine and to also follow any/all links on the page
for additional scanning.
You would want to put this on any page that you want to appear in the search
engine and that will also have links to pages that you want in the search
engine. Put this all over the site except on pages that could contain
information that you would want your users to see, but perhaps not want a
search engine to find.
