Ok in the last example, that would be two commands, both included in the same robots.txt file? Like you're saying first and foremost that google has access to everything, and secondly, any other agent cannot access the specified files?
Thanks. Always wondered exactly what robots.txt was for. Any other good uses?
|