# This allows the bot to spider all pages User-agent: * Disallow: