Distill-Bot

Distill-Bot fetches public files that publishers are required by law to post, or have licensed for reuse, such as hospital price-transparency files and state sales-tax rate files. It identifies itself as Distill-Bot/<version> (+https://data.spencermcw.com/bot; contact: <operator contact>), and never as a browser.

How it fetches

• It honors robots.txt. To stop it, disallow Distill-Bot.
• At most one request at a time per host, and at most one request a second.
• Conditional requests (ETag, Last-Modified), so unchanged files are not downloaded again.
• It backs off on 429 and 503, and resumes large downloads instead of restarting them.

Contact

Operator contact: bot@spencermcw.com