Fleebs-Logo
Details werden geladen...

readystack-robots-txt-ai-crawler-audit · PyPI

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI-search citation, or user fetch - and which lines silently do nothing. (bundled Node.js runtime)