# Crawler policy for calebsargeant.com (the www host 301s here).
#
# This is a CV site. Being read, indexed and quoted (by search engines and by
# AI assistants alike) is the entire point of it, so the default is Allow and
# the AI stance is permissive.
#
# The site is published in full in every language it supports: English at the
# root, Dutch under /nl/. They are translations of one another, not competing
# pages, and each one says so with hreflang in its
and in the sitemap.
# Nothing here needs to prefer one over the other: blocking a translation is how
# you lose the language a reader actually searched in.
#
# Content-Signal (contentsignals.org) states how automated systems may use this
# content:
# search=yes let search and AI search index and link to these pages.
# ai-input=yes let assistants read the pages to answer questions about Caleb.
# ai-train=yes training on a public CV is fine. It is the one line here worth
# a second thought, but a CV that a model already knows about is
# a CV that gets recommended.
#
# For assistants there are plainer ways in than the HTML, all generated from
# the same data as the pages by scripts/build.py:
# /llms.txt a markdown map of the site and the PDFs (llmstxt.org)
# /llms-full.txt the whole CV in one markdown file
# /index.md every page as clean markdown, e.g. /experience/index.md
# and /nl/experience/index.md. Each is served with a
# Link rel="canonical" back to its page, so search engines
# index the page, not the copy.
# /.well-known/agent-skills/index.json
# Agent Skills for answering questions about Caleb and
# for connecting to the MCP server
# and an MCP server, https://mcp.calebsargeant.com/ (public, read-only, no
# authentication), that searches this site and the docs.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
# /print/ is the same content as /cv/ and /experience/ with the design stripped
# out: it exists only as the source the PDF renderer prints from. Indexing it
# would put duplicate, uglier copies of every page in the results.
#
# Disallow is a prefix match, so /print/ does not cover the translations: every
# locale's print sheets need their own line, and a locale added to LOCALES in
# scripts/build.py needs one adding here.
Disallow: /print/
Disallow: /nl/print/
# The AI Catalog, for agent registries that read robots.txt for it (the
# Agentmap line from the Agentic Resource Discovery draft). It lists the MCP
# server; crawlers that do not know the directive ignore it.
Agentmap: https://calebsargeant.com/.well-known/ai-catalog.json
# One sitemap for the whole site. It lists every page in every locale, each
# entry naming the others as hreflang alternates, so there is nothing per
# language to point at from here.
Sitemap: https://calebsargeant.com/sitemap.xml