Cognition’s Crawling Policy
How we crawl and how site owners can opt out
Cognition AI’s Web Crawls
Like many AI companies, Cognition uses bots to gather data from the public web for model development, to perform web searches, and for other purposes. This includes the European Union, United States, and other jurisdictions.
These are the primary bots we use:
| The Bot | What it’s For | Effect of Opt-Out |
|---|---|---|
| CogBot | We use this bot to gather data from the public internet. | Blocking this bot will exclude your data from collection for our model training. |
| Devin | When a user asks Devin a question, this bot gathers information to respond. | Blocking this bot will prevent your site from being returned in a user query. |
For transparency, Cognition discloses the identity of the bots it uses when collecting data or browsing. When crawling, we identify our bots. We respect robots.txt opt outs. We also promote transparency by identifying our bot. Finally, we honor anti-circumvention technologies like CAPTCHAs, and we do not bypass paywalls when presented to our bots. These principles are important to us, and we strive to honor them.
To opt-out from data crawling, simply list our bot’s name in the robots.txt in your top-level directory. Please list our bot for the subdomains you wish to opt-out as well. Alternately, you may block all crawlers, and we will honor that too. Blocking is as simple as entering the following into your robots.txt:
User-agent: CogBot
Disallow: /
You can also block our IP address, but this method is not reliable, as we treat your robots.txt as the ultimate source of truth. This is a list of the current IP addresses we use for crawling.
Please note that we may update this page and our practices from time to time, including the name of our crawlers and our IP address. If you have questions or concerns, you can contact us at cogbot@cognition.ai.