gonzo
08-31-2006, 10:40 AM
Avoid dynamic URLs that contain ?, &, $, =, +, and % characters, cgi-bin, session IDs, or cookies.
These URLs are usually the result of query strings on dynamic pages. Some search engine crawlers are wary of such URLs because they could potentially be infinite links that would overload the crawler. As a result, they may not index pages that use these URLs, especially when three or more of these characters are included in the address. As Eytan Seidman, MSN Search Program Manager, states, “The algorithm starts to wonder whether it is going to get stuck in a loop endlessly crawling every single permutation of the query parameters. Thus, URLs with many (definitely more than 5) query parameters have a very low chance of ever being crawled.” (Eytan Seidman, Crawling the Internet…, msnsearch’s WebLog (November 18, 2004).)
These URLs are usually the result of query strings on dynamic pages. Some search engine crawlers are wary of such URLs because they could potentially be infinite links that would overload the crawler. As a result, they may not index pages that use these URLs, especially when three or more of these characters are included in the address. As Eytan Seidman, MSN Search Program Manager, states, “The algorithm starts to wonder whether it is going to get stuck in a loop endlessly crawling every single permutation of the query parameters. Thus, URLs with many (definitely more than 5) query parameters have a very low chance of ever being crawled.” (Eytan Seidman, Crawling the Internet…, msnsearch’s WebLog (November 18, 2004).)