I was wondering about a couple meta tags...
name="revisit-after" content="7days"
name="robots" content="index, follow"
Do these tags tell bots and spiders to follow the links on the page they are crawling and to come back again in 7 days?
I was also wondering if a spider or bot followed the links, does it include links outside of the site it is crawling... if so how do you control it?
|