Framer sitemap.xml and robots.txt after converting to code

When you convert a Framer site with Deframer, its robots.txt and sitemap.xml are copied into the project's public/ folder as the site published them, with the published domain inside. They work as static files on any host. Before launch, replace the domain if it is changing; after launch, either keep editing the static files when pages change or replace them with a sitemap generated from your routes and CMS records, so new pages appear without anyone remembering to add them.

This post explains exactly what you get, how to edit it, and how to switch to generated files in Next.js and the other frameworks. Deframer is an independent tool and is not affiliated with, endorsed or sponsored by Framer.

What does Framer publish?

A published Framer site serves two crawler files at its root:

• /robots.txt, which tells crawlers what they may fetch and where the sitemap is.
• /sitemap.xml, which lists the site's public URLs, CMS detail pages included. A multi-locale site publishes a sitemap index pointing at one sitemap per locale.

Framer writes both for you and updates them on every publish. That convenience is what you give up when you leave: in code, nothing updates them unless you make it.

What the converted project contains

Two details are worth knowing:

1. The domain inside is the published one. If the site lived at yoursite.framer.website or www.example.com, that's what the files say.
2. Only URLs the project serves are listed. A domain can serve more than the Framer project (another app on some paths, for instance). The converter keeps the sitemap entries that match the project's own routes, so the sitemap doesn't advertise pages the project can't answer.

The same files come with every framework Deframer writes: Next.js, TanStack Start, Astro, React Router and Vite. Each serves public/ at the site root.

How do you update them for a new domain?

If your domain stays the same (you move www.example.com from Framer to a new host), the files are already correct. If it changes, for example from a framer.website address to your own domain, update them before launch.

1. Open public/robots.txt and change the Sitemap: line:

   User-agent: *
   Allow: /

   Sitemap: https://www.example.com/sitemap.xml

2. Replace the domain in every <loc> of public/sitemap.xml. From the project folder:

   grep -rl "yoursite.framer.website" public/*.xml public/robots.txt \
     | xargs sed -i '' 's#https://yoursite.framer.website#https://www.example.com#g'

On Linux, drop the '' after -i.

3. On a multi-locale site, repeat for each sitemap_<locale>.xml and the index.
4. Check the root layout's metadataBase too: it sets canonical URLs and must match the domain in the sitemap. The SEO migration guide covers it.

Common robots.txt edits

Framer's default robots file allows everything. A few changes people make once the site is code:

• Block a staging host. Don't do it in robots.txt on the production build. Most hosts add X-Robots-Tag: noindex to preview deployments; check yours, and keep production's file permissive.
• Keep form endpoints out. API routes such as /api/forms/... only accept POST requests, so crawlers have nothing to index there; a Disallow: /api/ line is harmless if you want it explicit.
• AI crawlers. You can allow or disallow specific user agents (for example GPTBot or Google-Extended). Decide per crawler; the AI search guide discusses the trade-off.

Remember that robots.txt controls crawling, not indexing. A page blocked in robots.txt can still be indexed from links; use a noindex robots meta tag (in the route's metadata) to keep a page out of results, and don't block it in robots.txt, or crawlers never see the tag.

When do the static files go stale?

A static sitemap is a snapshot. It goes out of date when you:

• add a CMS record to src/cms/<collection>.json, which gets a page at the next build but no sitemap entry;
• add, rename or delete a page;
• add a locale.

For a small site that rarely changes, editing public/sitemap.xml by hand is fine. For a blog or a growing CMS, generate it.

How to generate the sitemap in Next.js

Next.js supports a sitemap.ts file in the app folder that returns the entries and serves them at /sitemap.xml. Delete public/sitemap.xml first, since a static file and a generated route at the same path conflict.

// src/app/sitemap.ts
import type { MetadataRoute } from "next";
import posts from "@/cms/blog.json";

const site = "https://www.example.com";
const pages = ["/", "/about", "/pricing", "/blog"];

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    ...pages.map((path) => ({ url: `${site}${path}` })),
    ...posts.map((post) => ({ url: `${site}/blog/${post.slug}`, lastModified: post.publishedDate })),
  ];
}

Adjust the collection name and fields to yours. The README lists every route (URL to route file), which is a good source for the pages list; src/site/ also holds the site's routes table if you'd rather read it programmatically. Next.js has a matching robots.ts convention if you want to generate robots.txt too, for example to point at a different sitemap per environment. Check the Next.js documentation shipped in node_modules/next/dist/docs/ for the current API.

Multi-locale sites can either generate one sitemap per locale or include alternates with language versions in each entry. Keep whatever structure the published site had unless you have a reason to change it.

The other frameworks

• Vite. The site is dist/, built by npm run build. Generate the sitemap with a small Node script that reads src/cms/*.json and the routes, writes public/sitemap.xml, and runs before the build ("prebuild": "node scripts/sitemap.ts").
• Astro. The same script approach works, or Astro's sitemap integration if you prefer one. If you add the integration, remove the copied file so you don't publish two.
• TanStack Start and React Router. Keep the static file and regenerate it with a script before each build, or add a server route that returns the XML. A prebuild script is the simpler of the two.

A minimal script:

// scripts/sitemap.ts
import { writeFileSync, readFileSync } from "node:fs";

const site = "https://www.example.com";
const posts = JSON.parse(readFileSync("src/cms/blog.json", "utf8")) as { slug: string }[];
const urls = ["/", "/about", "/blog", ...posts.map((p) => `/blog/${p.slug}`)];
const xml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${urls.map((u) => `  <url><loc>${site}${u}</loc></url>`).join("\n")}
</urlset>
`;
writeFileSync("public/sitemap.xml", xml);

Node.js 22.18 and later run TypeScript files like this directly.

How to check the result

1. Build and start the site, then fetch both files:

   curl -s http://localhost:3000/robots.txt
   curl -s http://localhost:3000/sitemap.xml | head -20

2. Confirm every <loc> uses the production domain and returns 200, not a redirect:

   curl -s http://localhost:3000/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' \
     | sed 's#https://www.example.com#http://localhost:3000#' \
     | xargs -n1 curl -s -o /dev/null -w '%{http_code} %{url_effective}\n' | grep -v '^200'

No output means every URL answered 200.

3. After launch, submit the sitemap in Google Search Console and Bing Webmaster Tools, and watch the indexing reports for errors.