# allow csjgoogle to crawl website. User-agent: gsa-crawler Disallow: # don't allow any web crawlers to crawl website except our google mini search. User-agent: * Disallow: /