How we test a Framer to Next.js converter: our methodology

To test a Framer to Next.js converter fairly, pick sites by a rule instead of by hand, convert each from its public URL, and check five things: every published URL has a route, next build and tsc --noEmit pass, full-page screenshots match the live site at several widths, the server answers URLs, forms and 404s correctly, and the code is something a developer can maintain. It describes the Next.js export; Deframer also exports the same conversion to TanStack Start, Astro, React Router and Vite.

This post describes the method we used for our real conversion tests on five websites and the aggregate benchmark. It is written so you can apply it to any converter, including ours.

What does a fair test of a converter look like?

A fair test measures the conversion, not the tester's choices. That means a site list drawn by a rule, the same commands for every site, raw numbers kept next to the claims, and failures reported with the successes.

How do you pick test sites without cherry-picking?

Decide the rule first, write it down, then apply it. We used two lists:

1. Framer Marketplace template previews. A list of 150 template demo sites spread evenly over the marketplace's alphabetical catalog. Templates are public, built by many different designers, and cover most site types.
2. Sites on custom domains from a larger list of public Framer sites, sampled at a fixed interval by name.

For the five-site test we needed five types (a one-page landing page, a multi-page agency site, an animated portfolio, a CMS-driven blog and a multilingual site). For each type we took the first site in alphabetical order that met a structural rule we could check from the published site, for example "two or more locales" or "the most appear-animation data among portfolio templates". The animated-portfolio rule deliberately picked the heaviest animation load, not the lightest.

Things that make a sample easier than reality, so avoid them or say so:

• Re-running until a site passes.
• Dropping sites that fail to convert or build (count them as failures).
• Testing only the home page.
• Using templates only. Real client sites carry more custom code, embeds and third-party scripts. We mixed in custom-domain sites for that reason.

We anonymize third-party sites in our posts (by category, never by name or URL) and don't show screenshots of their designs. The list is kept with the raw results and is available on request.

How do you check route coverage?

Compare what the live site publishes with what the converted project serves. The published side is the site's sitemap.xml (Framer writes one for every published site, including CMS item pages). The converted side is the list of prerendered paths after next build, which Next.js writes to .next/prerender-manifest.json.

For each site, record:

• URLs in the sitemap with no route in the project (a real gap).
• Routes in the project that are not in the sitemap. These are usually pages the designer excluded from search, such as legal pages, or the 404 page, and are not errors, but look at each one.
• CMS collections and item counts, compared with the number of item URLs in the sitemap.
• Locales, and whether each locale's URLs exist.

Then request every sitemap URL from the running build (next start) and record the HTTP status. A route file that exists but answers 500 is not coverage.

How is visual difference measured?

We use full-page screenshots and a pixel comparison. Our script (bin/verify.ts in the converter's repository) does this for each route and width:

1. Load the live page twice and the local build once in headless Chrome, at a viewport of the given width and 900 px high, device scale factor 1.
2. Before each capture, scroll through the page so lazy images and in-view effects load, wait for fonts and images to decode, rewind every video to its first frame, and wait until the page is at least 6 seconds old so intro animations have finished. Framer's "Made in Framer" badge is hidden on the live page.
3. Compare pixel by pixel. A pixel differs when the summed difference of its red, green and blue values is more than 48 (out of 765). A local pixel counts as a mismatch only if it differs from both live captures, so a slider that moved between the two live captures does not count against the conversion.
4. Report the mismatch ratio (mismatched pixels divided by the larger page area). Areas one capture has and another lacks (a page that is taller or shorter) count as mismatched, so a layout shift that changes page height shows up.
5. Also report the share of pixels that differ between the two live captures ("unstable"), the page heights, and console errors that the local page raises but the live page doesn't.

Hygiene rules that keep the number honest:

• A local capture above 1% mismatch is retaken once and the better capture counts (Chrome sometimes captures a tall page before it has painted).
• Captures whose own assets failed to load are retried and flagged unreliable if they still fail.
• Routes the live host doesn't serve from that Framer project (404, redirected elsewhere) are skipped and listed, not counted as passes.
• Console errors the live site raises too (a third-party embed returning 404) are counted separately as "faithful" errors.

A pixel ratio is a blunt number. One wrong font can move every line of text on a page and produce a large mismatch even when the page "looks right" to a person; a missing button is a few hundred pixels and barely moves it. Always open the diff images for anything above about 1% before drawing conclusions.

Which widths should you test?

Test one width inside each of Framer's default breakpoints. Most Framer sites use desktop, tablet and phone breakpoints, commonly 1200 px and up, 810 to 1199 px, and below 810 px (check each site's own breakpoints), so we used 1440, 810 and 390 px for the five-site test and 1440 and 390 px for the larger benchmark. See Framer breakpoints and responsive layouts in Next.js for how breakpoints map to CSS.

Widths right at a breakpoint edge (809 px vs 810 px) catch rounding bugs. They are worth a spot check but make the benchmark noisier.

What functional tests matter?

Screenshots can't tell you whether the server behaves like the original. We added HTTP checks against next start:

• [ ] Every sitemap URL answers 200 (or the redirect the live site uses).
• [ ] An unknown URL answers 404, as on the live site, not a soft 200.
• [ ] robots.txt and sitemap.xml are served.
• [ ] Each page's title, description, Open Graph image and robots tag match the live page.
• [ ] Each native form's endpoint refuses an empty or malformed submission and accepts a honeypot-filled one quietly (no real submissions are sent).
• [ ] Published redirects answer with the same status and target.

We also ran our interaction checker (hover, clicks, menus and scroll, compared with the live page). On this run it could not locate elements on the converted pages, because it matches elements by Framer's data-framer-name attribute and the converter renames that attribute to data-name in its output. We report that as a gap in our tooling rather than a pass or a fail. Until it is fixed, check menus, accordions and sliders by hand at phone width.

What should you look for in the code?

Read the project the way the developer who maintains it will:

• Language and files: are all source files TypeScript, and are there leftover .js bundles?
• Dependencies in package.json: exact versions, nothing unexpected, no dependency on the original platform's runtime.
• Whether next.config.ts disables type checking (typescript.ignoreBuildErrors) and why.
• What stays remote: assets on a CDN, fonts, embeds.
• How CMS content, forms and redirects are represented, and whether you can edit them without the original tool.

What you get in a converted Next.js project shows what we expect to find in a Deframer project.

How do you verify the build?

Run the commands a developer would run, on a clean checkout:

npm install
npx tsc --noEmit
npm run build
npm run start

Record build success, build time, the number of routes Next.js reports, and the tsc error count with the first few errors. A project can build while tsc reports errors if type checking is turned off for code the designer wrote in Framer's code editor, which Framer never type-checks; count those errors separately from a failed build.

For bulk runs we build against one shared node_modules install pinned to each project's exact versions, which is faster but not identical to a fresh npm install. Spot-check a few projects with a fresh install.

What are the limits of this method?

• Live sites change. A designer can republish between capture and comparison. We convert and capture within the same hour.
• Noise. Autoplaying media, random content and network timing move pixels. The two-capture rule removes most of it, not all.
• Pixel ratios hide structure. Accessibility, semantics and interaction states aren't measured by screenshots.
• Templates are cleaner than client sites. Our samples are mostly marketplace templates.
• One machine, one network. Wall times depend on both. We report the machine and treat times as indicative.
• Sample size. Five detailed sites and a 12-site benchmark show patterns, not guarantees.

How do we handle our conflict of interest?

We make Deframer, and our own tools wrote most of these measurements. To limit the bias:

• The site rule was written before converting, and no site was replaced after it failed.
• Failures and gaps are published with the successes, including bugs in our own converter that we found during the run.
• Measurements and judgments are separated in the posts: numbers come from the tools, and anything we call "good" or "minor" is labeled as our judgment.
• The method is tool-agnostic. Everything above works with any converter's output, or with a hand-built rebuild.

We did not test competing converters in this round, so these posts make no comparative claims. For the landscape, see the best Framer to Next.js converters.

How can you reproduce this with any tool?

1. Write your site rule and list before converting anything.
2. Convert each site with the tool you're testing and note the wall time.
3. Run npm install, npx tsc --noEmit and npm run build; save the output.
4. Fetch the live sitemap.xml and compare it with the built routes.
5. Capture full-page screenshots of the live page twice and the local build once at 1440, 810 and 390 px, with the same waiting rules, and count pixels that differ from both live captures. Playwright plus any image library is enough.
6. Run the HTTP checks against npm run start.
7. Open the code and answer the inspection questions.
8. Publish the raw numbers with the list of sites, or keep them available on request.