Tendubot

A small web crawler that reads public college pages about dance programs.

What it does

Tendubot reads public pages on college and university websites about dance degree programs. It looks for admissions facts that families ask about: application and audition dates, what an applicant has to submit, and similar requirements. It records the dates and requirements as the school states them, with a link back to the page.

It may also note program details the school publishes, such as the faculty a program lists. It does not log in, fill in forms, or read anything behind a sign-in.

How it behaves

• It obeys robots.txt (RFC 9309) and re-reads it at least once a day.
• It is rate-limited: it fetches one page at a time per site, with pauses between requests, and re-checks a page it already cites about once a week to see whether the school has changed it.
• It stops at bot checks and CAPTCHAs. It does not try to get around them.

It identifies itself with this user agent:

Mozilla/5.0 (compatible; Tendubot/1.0; +<this page>)

How to block or limit it

To keep Tendubot off your whole site, add this to your robots.txt:

User-agent: Tendubot
Disallow: /

To keep it out of part of the site only:

User-agent: Tendubot
Disallow: /private-section/

Changes take effect within a day.

Contact

To ask it to slow down, report a problem, or ask a question, write to crawler@tendubot.fyi.