PRODINT
Sign in Scan my site

For site owners

The Prodint crawler

If you've seen prodint in your logs, this page explains what it is, what it reads, and how to limit or block it.

Last updated 25 September 2026

What it is

Prodint is a product research tool. Its users open a “dossier” on a product, often a competitor's, and give it starting points such as a marketing site or documentation. The crawler reads those pages in a real browser to build a map of the product's features. It's a research reader, not a search engine and not a scraper for resale.

How to recognise it

Every request carries a browser user agent that ends in prodint/0.1:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 prodint/0.1

What it will and won't do

  • It follows your robots.txt (details below).
  • It reads pages one at a time per crawl, and reads only a sample of pages that share a shape, such as /customers/:slug, rather than every one.
  • It never submits forms. It skips logout, sign-up, delete and checkout-style paths, and aborts any request other than a GET that a page interaction would trigger.
  • It doesn't try to get past logins or captchas on its own. When it meets one, it stops and hands over to the person running the crawl, who can sign in with an account they are entitled to use, or skip that source.
  • It refuses to fetch private network addresses.

Limiting or blocking it with robots.txt

Prodint honours Disallow and Allow rules; the longest matching rule wins. If your robots.txt has a User-agent: prodint group, Prodint uses that group instead of the User-agent: * rules. Otherwise it follows the * rules.

To block Prodint from your whole site:

User-agent: prodint
Disallow: /

To keep it out of part of your site only:

User-agent: prodint
Disallow: /account/
Disallow: /internal/

Prodint reads your robots.txt at least every time a crawl starts, so a change applies from the next crawl, if not sooner.

Reviews and traffic

Separately from crawling, Prodint reads public reviews of a product from Trustpilot and the App Store, and looks up public traffic rankings (Tranco, and Similarweb when configured). Those requests go to the review and ranking services, not to your site.

Contact

Questions or a problem with the crawler? Email [email protected] with your domain and, if you can, the time and path of the requests. We'll look into it.