# Crawler policy for calebsargeant.com (the www host 301s here). # # This is a CV site. Being read, indexed and quoted (by search engines and by # AI assistants alike) is the entire point of it, so the default is Allow and # the AI stance is permissive. # # The site is published in full in every language it supports: English at the # root, Dutch under /nl/. They are translations of one another, not competing # pages, and each one says so with hreflang in its and in the sitemap. # Nothing here needs to prefer one over the other: blocking a translation is how # you lose the language a reader actually searched in. # # Content-Signal (contentsignals.org) states how automated systems may use this # content: # search=yes let search and AI search index and link to these pages. # ai-input=yes let assistants read the pages to answer questions about Caleb. # ai-train=yes training on a public CV is fine. It is the one line here worth # a second thought, but a CV that a model already knows about is # a CV that gets recommended. # # For assistants there are plainer ways in than the HTML, all generated from # the same data as the pages by scripts/build.py: # /llms.txt a markdown map of the site and the PDFs (llmstxt.org) # /llms-full.txt the whole CV in one markdown file # /index.md every page as clean markdown, e.g. /experience/index.md # and /nl/experience/index.md. Each is served with a # Link rel="canonical" back to its page, so search engines # index the page, not the copy. # /.well-known/agent-skills/index.json # Agent Skills for answering questions about Caleb and # for connecting to the MCP server # and an MCP server, https://mcp.calebsargeant.com/ (public, read-only, no # authentication), that searches this site and the docs. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # /print/ is the same content as /cv/ and /experience/ with the design stripped # out: it exists only as the source the PDF renderer prints from. Indexing it # would put duplicate, uglier copies of every page in the results. # # Disallow is a prefix match, so /print/ does not cover the translations: every # locale's print sheets need their own line, and a locale added to LOCALES in # scripts/build.py needs one adding here. Disallow: /print/ Disallow: /nl/print/ # The AI Catalog, for agent registries that read robots.txt for it (the # Agentmap line from the Agentic Resource Discovery draft). It lists the MCP # server; crawlers that do not know the directive ignore it. Agentmap: https://calebsargeant.com/.well-known/ai-catalog.json # One sitemap for the whole site. It lists every page in every locale, each # entry naming the others as hreflang alternates, so there is nothing per # language to point at from here. Sitemap: https://calebsargeant.com/sitemap.xml