OT: Search Engine Indexing of Dynamic Sites

[Jon Spivey]

Hi,

I know this is OT but thought it was interesting. As most of us know when
search engines see a dynamic page eg showproduct.aspx?productid=123 they
either won't follow the link atall or will only follow a few.

Yesterday I found a neat way to use asp.net to rewrite dynamic links so that
showproduct.aspx?productid=123 can be written as 123.aspx. So far so good -
the search engine doesnt see this as dynamic and will happily follow the
link. Tested it out on one of my sites and seemed to work nicely. Then a few
minutes ago google hit the site followed all the links and opened 101
database connections in about a minute - at which point the site fell over.
This is against a sql server 2000 db - obviously access would bottle out way
before.

Has anyone ever tried getting search engines to follow dynamic links without
that result?

 

[chris leeds]

I haven't. I'm surprised that sql server would crash like that. I though
it was indestructible.
the closest thing I've seen to what you're describing was a new feature in
apache....something like apache mod.rewrite.

do you think your technique will work on "regular" .asp?

 

[Jon Spivey]

I'm assuming my host limits open connections, only moved this site to them
this week - the fact that it fell over at 101 makes me think 100 is the
limit. I've got an email into them but havent heard back yet.

The feature isn't in asp - it's asp.net only. Basically the site sells
books - details of each book are displayed on showbook.aspx which accepts
the isbn of the book in the query string, so a link might look like
showbook.aspx?isbn=0340828994. Asp.net allows rewriting of links so I could
write this link as 0340828994.aspx and the page displayed would actually be
showbook.aspx?isbn=0340828994 although 0340828994.aspx would still show in
the address bar. You could probably achieve something similar with asp by
using a custom 404 page but it would be messy.

Very cool feature of asp.net - or so I thought

 

[chris leeds]

I'm sure it's an "artificial" limitation imposed by your host and I hope it
works out for you because it's too good of a trick not to be able to use it.
good luck!