Web Spiders and bots vs ASP dynamic

**invitro** · Nov 4th, 2004, 08:54 PM

Ok, I heard a lot of spiders have a problem with dynamic content on a website. Meaning, if I had meta tags that generated from a dynamic function on every page, they would not find them. Is this true?

I have default.asp that looks like this in basic code:

Code:

Top(); //Contains <HTML META> tags

Content is inserted here;

Bottom() //containts all the </HTML> Statements

If a spider hits a page, it SHOULD recieve the full content of the page since the server generates the URL on the spot... right?....

**mendhak** · Nov 4th, 2004, 11:50 PM

Right.

And there shouldn't be a problem.

**invitro** · Nov 7th, 2004, 05:04 PM

allright so i guess my second question is how do you get a site 'out there'. I searched on google but it only gives me links to companies that will do it for a sum of money. Now, if I dont have an answer I guess i will have to go to one of them but id rather come up with something of my own.

**Danial** · Nov 7th, 2004, 08:37 PM

Originally posted by invitro
allright so i guess my second question is how do you get a site 'out there'. I searched on google but it only gives me links to companies that will do it for a sum of money. Now, if I dont have an answer I guess i will have to go to one of them but id rather come up with something of my own.

It really depends on which search engine you are talking about. The algorithm they use are different. Google how ever is a difficult one because the way it work. The way it ranks is different then others, one of their main method is to check how many sites links to your site and depending on how popular those site that linked you they give you score. There are other elements in which they decided but they do not share all their techniques to thwart the attempt of people manupulating their search ranking. So if you want to have top ranking in google make sure you get other site/people to link to your site. The more the links the higher the ranking goes up. Ofcourse you will have to ovbious things such as meta tags and keywords to attract the Bots/crawler in the first place. Some site submits your site to various search engine which ensure that your site gets crawled.

Hope this helps.

BTW : Like your new sig

**mendhak** · Nov 8th, 2004, 12:28 AM

http://www.google.com/search?hl=en&l...ps&btnG=Search

Also, it takes quite a while for google to index your pages. Mendhak.com took four months.

**invitro** · Nov 8th, 2004, 07:32 PM

Sites That Use Dynamic URLs
Most search engines cannot or will not list any dynamic URLs. Dynamic URLs are most commonly used on database driven sites, or sites that are running scripts. They commonly look something like this one from MSN Search:

http://search.msn.com/results.asp?q=dynamic+url

If your URL contains any of the following elements it is considered a Dynamic URL as far as the search engines are concerned:

?, &, %, +, =, $, cgi-bin, .cgi
(commas above are just used as a separator)

If your URL contains any of the above elements it is very unlikely that it will get listed at any of the major search engines. We recommend that you submitting a different URL that is not dynamic if possible. If all of your URLs are dynamic you should probably consider making static pages (i.e. not dynamic) with URLs that do not contain any of the elements noted above so that you can get listed in the search engines.

What if the URL is something like
www.whatever.com/default.asp

its an ASP extension, does that mean its considered dynamic or is it only dymic when you start generating querys like ?ID= etc

**Danial** · Nov 8th, 2004, 07:51 PM

Originally posted by invitro
What if the URL is something like
www.whatever.com/default.asp

or is it only dymic when you start generating querys like ?ID= etc

Yep thats what they are referring to.

The article mentions that dynamic url does not get crawled by majro search engine, but if you notice google result you will see they do.

Also in my site(fully dynamic), i have access log and i have seen crawler going through evrey possible links without any problem.

I suspect they would start indexing with the main site e.g www.whatever.com instead of www.whatever.com/default.asp?myId=10

**invitro** · Nov 8th, 2004, 08:07 PM

ahh excellent, thanks a lot for all the help. I guess I have to wait now.

**mendhak** · Nov 8th, 2004, 11:22 PM

Meanwhile, you can watch your plants grow. That's what I did.

Thread: Web Spiders and bots vs ASP dynamic

Thread Tools

Display

Web Spiders and bots vs ASP dynamic

Posting Permissions