How to make dynamic aspx pages indexable by spiders

  • Thread starter Thread starter moondaddy
  • Start date Start date
M

moondaddy

I'm writing an ecommerce app in asp.net/vb.net and need to make the pages
searchable and crawlable by spiders, particularly Google's. As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this. asp.net offers lots of
great technology, but if the success of the site depends on people's ability
to find it, then all of this work is to no avail. I'm surprised to find
very little (none actually) threads or documentation about this. the only
information I can find on optimizing a site for search engine ratings are by
non-ms portals and they all talk about static html pages. its like they
never heard of asp.net.

Can anyone help on this under talked about topic?

historically I've built business applications and never had to worry about
his. I spent tons of time building and optimizing the site for usability
and performance, and now that I'm trying to get it indexed by the search
engine, I'm finding that my high performance database driven site wont be
seen by the spiders, or I'm just not finding the right documentation.
 
Fully dynamic? It is not going to happen as the spider does not know if it
can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes
up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was
very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

**********************************************************************
Think Outside the Box!
**********************************************************************
 
did you mean this one?
http://www.ftponline.com/vsm/2002_02/online/online_print/jgoodyear/default.aspx
If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.

--
(e-mail address removed)
Cowboy (Gregory A. Beamer) said:
Fully dynamic? It is not going to happen as the spider does not know if it
can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes
up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was
very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

**********************************************************************
Think Outside the Box!
**********************************************************************
moondaddy said:
I'm writing an ecommerce app in asp.net/vb.net and need to make the pages
searchable and crawlable by spiders, particularly Google's. As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this. asp.net offers lots of
great technology, but if the success of the site depends on people's ability
to find it, then all of this work is to no avail. I'm surprised to find
very little (none actually) threads or documentation about this. the only
information I can find on optimizing a site for search engine ratings
Click to expand...
are
Click to expand...
 
Ok maybe I missed the point if this was in fact the article you were talking
about. The light when on when I read another article about this problem:
http://www.seochat.com/c/a/Search-Engine-Optimization/Dynamic-URLs-In-The-Eyes-Of-A-Search-Engine/
where they were talking about php pages, but the same concept (although it
didn't get into the mechanics, just an overview).

so in short is this how it would work: Rather than passing in a parameter,
I'd pass in a url like http://www..mysite/procucts/category/animals and on
the backend I'll use the http handler to intersept this, parse it, and run
select case on the last word for example (in this case animals), then call
the real page using a redirect passing in the real parameter like
products.aspx?89 (where 89 could be the category id animals). Is this the
basic idea?

--
(e-mail address removed)
moondaddy said:
did you mean this one?
http://www.ftponline.com/vsm/2002_02/online/online_print/jgoodyear/default.aspx
If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.
Click to expand...
 
"Rewrite URLs on the Fly" by Jeff Prosise (ASP.NET C#)

The article will show up here [1] but I do not know when as it was
just published by asp.netPRO, March 2004 and can only be accessed
via [2] by magazine subscribers.

[1] http://www.wintellect.com/resources/articles/articles.aspx?authorID=1
[2] www.aspnetPRO.com/download

--
<%= Clinton Gallagher
A/E/C Consulting, Web Design, e-Commerce Software Development
Wauwatosa, Milwaukee County, Wisconsin USA
NET (e-mail address removed)
URL http://www.metromilwaukee.com/clintongallagher/
 
As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this.
Click to expand...

This is incorrect, at least in terms of Google. Google can handle query
strings just fine, with a few caveats:

1) It will index your site slower than if it does not see querystrings (this
is actually because it is polite...if it sees a query string, it assumes a
database, and doesn't want to overload it with queries).

2) It seems to not like more than 3 values in the query string, with two
being the most common limit.

The other issue with query strings, of course, is that they are not very
human readable. If this is a big issue for you, look into a URL rewriting
tool like ISAPI rewrite.

-Darrel
 
Back
Top