PaperSuitorBot

The crawler of PaperSuitor

What it does

PaperSuitorBot keeps PaperSuitor up to date. PaperSuitor is a free guide to journals in law and the other humanities and social sciences that helps authors decide where to submit. For each journal it shows the length limits, the review process and other conditions for authors, and links to the page where the journal states them.

What it reads

It does not log in, fill in forms, or follow links to accounts, shopping carts or social media.

How often, and how fast

It runs every day. From October 2026 it also visits the journals of 23 more fields, about 25,700 of them, some 2,000 a day over about two weeks; after that it runs in small batches again. A journal's pages for authors are read again after 60 days, and its article pages and feed after 28 days. A journal whose guidelines could not be found is tried again after 3, 7 and 14 days, then once a month, and at once when a new address for it turns up.

It sends one request at a time to each organisation's servers, at least one second apart. If your robots.txt sets a Crawl-delay, it waits the full delay between requests. After a 429 (Too Many Requests) answer, it waits at least 10 seconds between requests to that organisation for the rest of the run, or the Retry-After time if that is longer. If Retry-After asks for more than a minute, after a 429 or a 503 (Service Unavailable) answer, it sends nothing more to that organisation in that run, and nothing in a later run before the time it gave has passed. A redirect to another organisation's site is a request to that organisation, under the same rules.

Where it has kept a copy of a page, it asks whether the page has changed since (If-None-Match and If-Modified-Since), so an unchanged page is not sent again.

When a site says no

It follows robots.txt. Rules written for PaperSuitorBot apply to it; if there are none, it follows the rules for all robots (*). It reads robots.txt afresh at every run. Before following a redirect to another site, it checks that site's robots.txt too. If your robots.txt cannot be reached (a server error or a time-out), it leaves your site alone until the next run. To keep it off your site:

User-agent: PaperSuitorBot
Disallow: /

It never tries to get around bot protection and never solves CAPTCHAs. Its browser stops at a bot check in the same way. Some pages refuse it with 401, 403 or 503, or show a bot check. It leaves such a page alone until its next scheduled visit, 60 days later.

Checking that a request is ours

Its user-agent names PaperSuitorBot and gives the address of this page:

Mozilla/5.0 (compatible; PaperSuitorBot/1.0; +https://(this website)/crawler.html)

Anyone can copy a user-agent. The crawler runs from a single server, whose address we give to site operators on request, and it has a signing key: the public half is published at https://(this website)/.well-known/http-message-signatures-directory. Since 23 October 2026, 14:36 UTC, it has signed every request with that key, using HTTP Message Signatures as the Web Bot Auth protocol describes; Cloudflare has accepted it into its verified bots programme. A signed request also carries the header Signature-Agent: "https://(this website)". A request whose signature verifies against that key comes from this crawler.

What it keeps

PaperSuitor shows only facts, the journal's own sentence and a link to its page. The crawler keeps no copy from a site whose robots.txt sets use=immediate in its Content-Signal, that reserves text and data mining with the TDM Reservation Protocol (a tdm-reservation header or meta tag, or /.well-known/tdmrep.json), or whose terms we know to reserve it. From such a site it keeps only the facts, and it deletes any copy kept before; the copy is gone from the backups within a month. If your terms reserve text and data mining and your site still has copies kept, write to the address below.

Contact

Questions, problems, or a request that it stop visiting your site: [email protected]

Back to PaperSuitor