# The live robots.txt, serving since the launch on 2026-08-26.
#
# It replaced a pre-launch file that disallowed everything. That file is gone
# rather than kept beside this one: two candidates in one directory is how the
# wrong one gets restored. To close the site again, `Disallow: /` here is one
# line, and it is the smaller half of the job — see src/lib/preview.ts for the
# gate that actually hides the pages.
#
# Two jobs, and only two.
#
# The first is naming the sitemap. It is the one directive here that does
# something a page cannot: `Sitemap:` is how a crawler that has never seen a
# link to this site finds every product page in one request, and there is
# nowhere else to put it. The URL is served by the `api` function through a
# hosting rewrite, so it is generated from the catalogue rather than written
# by hand — see functions/lib/sitemap.js.
#
# The second WAS keeping crawlers out of `/api/`, and it is gone, because it
# had become a contradiction. `/llms.txt` tells an agent to call
# `/api/products` and `/api/mcp`; this file told it not to. Agent fetchers
# read robots.txt before a direct fetch — ChatGPT's and Claude's both do — so
# the one restriction here was aimed squarely at the readers the API exists
# for, and at the MCP server a builder connects their own agent to.
#
# The crawl-budget worry it replaced was real and small: the JSON endpoints
# are CDN-cached and none of them can be indexed, so a search engine that
# wanders in costs a cached response and gets nothing it wants. Being callable
# is worth more than that.
#
# WHAT IS DELIBERATELY NOT HERE: a `Disallow` for `/mine`, `/admin` or the
# edit pages. Disallowing a URL stops it being CRAWLED, not indexed — a
# disallowed page that somebody links to can still appear as a bare blue link
# with no description, because the crawler was forbidden from reading the very
# tag that says to leave it alone. Those pages carry `noindex` instead (see
# src/lib/meta.ts), which requires being crawled to work.
#
# THE THIRD JOB, added 2026-09-03: saying what may be DONE with what is read.
#
# `Content-Signal` is a preference expressed to whoever is already allowed to
# fetch the page. It is not access control — nothing enforces it, and the
# `Allow` below is still what decides who gets in. It is a stated position, and
# the position follows from what this directory promises a builder:
#
#   search=yes     Their listing exists to be findable. This is the whole offer.
#   ai-input=yes   Being quoted by an assistant answering "what tools are there
#                  for X" is the same offer, through a newer door. Refusing it
#                  would contradict every line on /for-builders.
#   ai-train=no    The one that gives a builder nothing back. Training on this
#                  catalogue does not make their product findable; it copies
#                  their words into a model that will not cite them. The text
#                  is theirs, not ours, which is why this is the one we decline
#                  on their behalf.
#
# Flip `ai-train` if that reasoning ever stops holding. It is a policy line, not
# a technical one, and it is a one-word edit.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://whatareyoubuilding.ai/sitemap.xml