PaperSuitorBot
The crawler of PaperSuitor
What it does
PaperSuitorBot keeps PaperSuitor up to date. PaperSuitor is a free guide to journals in law and the other humanities and social sciences that helps authors decide where to submit. For each journal it shows the length limits, the review process and other conditions for authors, and links to the page where the journal states them.
What it reads
- robots.txt, before anything else on a site. Before keeping a copy of a page, also the site's
/.well-known/tdmrep.json. - Journals' home pages, their pages for authors, such as author guidelines and submission instructions, and their aims and scope pages, as web pages or PDFs. It finds them from the home page and the site's menus, at most three clicks and 25 pages deep, from the site's sitemap, from a DOI, or from the Internet Archive's list of the site's pages. When a journal's address no longer works, it looks for a link to the journal on the school's or publisher's site. A PDF made of scanned pages is read with text recognition on our own server.
- Pages for authors that are drawn by script and arrive almost empty. For these it runs the page in a headless browser, one page at a time and at most 100 a day. The browser fetches only the page's scripts and the data they ask for, never images, style sheets, fonts or media. Every request it makes goes out under the same name and the same rules as all the others. It does not click, scroll, answer cookie banners or fill in anything.
- Up to 12 recent article pages of a journal, for the submission and publication dates they show. It reads these only when the publisher does not deposit those dates with Crossref.
- The OAI-PMH feed of journals on Open Journal Systems or Digital Commons, for the titles and abstracts of recent articles. It reads a feed only when other sources hold too few of the journal's articles.
- The page of a journal's current issue, for the date of its latest issue and the language of its articles.
It does not log in, fill in forms, or follow links to accounts, shopping carts or social media.
How often, and how fast
It runs every day. From October 2026 it also visits the journals of 23 more fields, about 25,700 of them, some 2,000 a day over about two weeks; after that it runs in small batches again. A journal's pages for authors are read again after 60 days, and its article pages and feed after 28 days. A journal whose guidelines could not be found is tried again after 3, 7 and 14 days, then once a month, and at once when a new address for it turns up.
It sends one request at a time to each organisation's servers, at least one second apart. If your robots.txt sets a Crawl-delay, it waits the full delay between requests. After a 429 (Too Many Requests) answer, it waits at least 10 seconds between requests to that organisation for the rest of the run, or the Retry-After time if that is longer. If Retry-After asks for more than a minute, after a 429 or a 503 (Service Unavailable) answer, it sends nothing more to that organisation in that run, and nothing in a later run before the time it gave has passed. A redirect to another organisation's site is a request to that organisation, under the same rules.
Where it has kept a copy of a page, it asks whether the page has changed since (If-None-Match and If-Modified-Since), so an unchanged page is not sent again.
When a site says no
It follows robots.txt. Rules written for PaperSuitorBot apply to it; if there are none, it follows the rules for all robots (*). It reads robots.txt afresh at every run. Before following a redirect to another site, it checks that site's robots.txt too. If your robots.txt cannot be reached (a server error or a time-out), it leaves your site alone until the next run. To keep it off your site:
User-agent: PaperSuitorBot Disallow: /
It never tries to get around bot protection and never solves CAPTCHAs. Its browser stops at a bot check in the same way. Some pages refuse it with 401, 403 or 503, or show a bot check. It leaves such a page alone until its next scheduled visit, 60 days later.
Checking that a request is ours
Its user-agent names PaperSuitorBot and gives the address of this page:
Mozilla/5.0 (compatible; PaperSuitorBot/1.0; +https://(this website)/crawler.html)
Anyone can copy a user-agent. The crawler runs from a single server, whose address we give to site operators on request, and it has a signing key: the public half is published at https://(this website)/.well-known/http-message-signatures-directory. Since 23 October 2026, 14:36 UTC, it has signed every request with that key, using HTTP Message Signatures as the Web Bot Auth protocol describes; Cloudflare has accepted it into its verified bots programme. A signed request also carries the header Signature-Agent: "https://(this website)". A request whose signature verifies against that key comes from this crawler.
What it keeps
- A private copy of each journal's pages for authors and of its aims and scope page, as web pages or PDFs (a page drawn by script, as the browser drew it), so that they can be read again without asking your site. A new copy is kept only when a page has changed. The copies are never published or shared. A backup of them is kept on storage of our own for at most a month.
- A journal's stated limits, with the sentence that states them, the page's address and the date it was read.
- How long the journal takes from submission to publication, as a median over recent articles.
- The titles and abstracts of recent articles, used to match manuscripts to journals.
- The date of a journal's latest issue and the language of its articles.
PaperSuitor shows only facts, the journal's own sentence and a link to its page. The crawler keeps no copy from a site whose robots.txt sets use=immediate in its Content-Signal, that reserves text and data mining with the TDM Reservation Protocol (a tdm-reservation header or meta tag, or /.well-known/tdmrep.json), or whose terms we know to reserve it. From such a site it keeps only the facts, and it deletes any copy kept before; the copy is gone from the backups within a month. If your terms reserve text and data mining and your site still has copies kept, write to the address below.
Contact
Questions, problems, or a request that it stop visiting your site: [email protected]