</>Cyber-Spaghetti.
[SYS.CRAWL // ROBOTS + SITEMAP]

TELL CRAWLERS
EXACTLY WHAT TO DO.

Both crawl files, generated and downloadable in one pass.

Set your domain, blocked paths and page list. We emit a valid robots.txt — with explicit allow rules for AI answer engines — and a matching sitemap.xml you can drop into your site root.

[ CONFIG ]
[ ROBOTS.TXT ]
# robots.txt — generated with cyberspaghetti.io/robots-sitemap

User-agent: *
Allow: /
Disallow: /admin
Disallow: /cart
Disallow: /thank-you

# AI search engines and answer engines

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bytespider
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://example.com/sitemap.xml
[ SITEMAP.XML ]
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-08-18</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-08-18</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://example.com/services</loc>
    <lastmod>2026-08-18</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-08-18</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://example.com/contact</loc>
    <lastmod>2026-08-18</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>