Skip to content
TestingExperience

AI Tool Tests

Independent tests of AI tools

27 guides on testing detectors, watermarks, provenance pipelines and disclosure labels.

Open section

Start here, in order

Methods and protocols

How we test 2.0%CI 1.3 to 3.1 False positive rate in the worked benchmark: 20 of 1,000 human-written documents wrongly flagged. Read the protocol
Editorial methodGlossaryReport an errorNo tool maker pays for a test

Software testing

Reference guides for testers

26 guides on planning, automating and designing tests, written to outlive any tool.

Testing Techniques

Sections

Recently updated

Test ManagementTest AutomationAgile TestingTesting TechniquesGuides name no products

Magazine

The magazine, 2008 to 2014

A free quarterly for professional software testers, published in Berlin across 28 issues.

The archive

Archive

Issue pages

28 issues Each issue page states the date and the theme where it is known, and nothing beyond it. The original issues are not distributed.
The archiveAll issuesEditorial methodRecords, not reproductions

Independent | No sponsorship | Method shown with every result

Independent tests of AI tools, with the method shown

Testing Experience tests the AI tools people are asked to trust: text detectors, AI writing tools and provenance checks. Every test uses a labelled sample set built before the run, a threshold fixed in advance, results with 95 percent intervals, and the date and version tested. The method is published next to the result, so you can rerun it on your own documents. Underneath sit 26 reference guides for professional software testers and an archive of the magazine published under this name from 2008 to 2014.

See how we test AI tools
A test bench for text: printed documents boxed and stacked, a ruled score sheet with a blue pencil beside them, and a stamped test report.

How we test an AI tool

Eight steps, identical for every tool. Everything that could bias the result is fixed before the first score is seen.

  1. Step 01

    Define the population and the decision

    Who wrote the documents and what happens to a flagged one.

    Step 02

    Build the human-written set

    At least 300 documents per slice, known to be written by people.

  2. Step 03

    Build the machine-written set

    Same prompts and topics, plus edited, paraphrased and mixed versions.

    Step 04

    Quarantine the sets

    No document is ever shown to the tool's maker or used for tuning.

  3. Step 05

    Fix the threshold in advance

    The score that counts as "flagged" is chosen before the run.

    Step 06

    Run and record every score

    Raw score and verdict, with tool version, settings and date.

  4. Step 07

    Compute the matrix and the intervals

    False positive and detection rates with Wilson 95 percent intervals.

    Step 08

    Report against the real base rate

    What a flag means at the share of machine text you actually expect.

Figure 1 | Worked benchmark n = 2,000 | threshold fixed before run | one square = one document

1,000 human-written documents

expected: none flagged
False positive rate2.0%95% CI 1.3 to 3.1
20 wrongly flagged980 cleared

1,000 machine-written documents

expected: all flagged
Detection rate94.0%95% CI 92.4 to 95.2
940 flagged60 missed
A worked benchmark, not a measured product: 20 of 1,000 human-written documents wrongly flagged is a 2.0 percent false positive rate.Intervals: Wilson score, 95%

The full protocol, with worked numbers, is in How to benchmark an AI text detector.

If you have an AI detector result in front of you

Four guides, read in order. Together they explain what the number can and cannot tell you.

  1. A detector score sheet with a gauge beside a printed essay and a pencil.
    1 of 4

    What an AI detector score means for a writer, teacher or editor

    What a percentage does and does not tell you about who wrote the text.

  2. One printed page with three score slips beside it, each showing a different result for the same text.
    2 of 4

    Why AI detectors disagree about the same text

    Different training data, thresholds and score scales on the same document.

  1. A hand-drawn two by two confusion matrix on squared paper, with the false positive cell circled.
    3 of 4

    False positives, base rates and the confusion matrix

    Why a 2 percent false positive rate can still mean many wrong accusations.

  2. Stacks of printed essays separated by tabbed dividers labelled by writer group, ready to be tested slice by slice.
    4 of 4

    Fairness testing for AI detectors: non-native English writers

    The slice where detectors fail most often, and how to test it.

Latest pages

Newest first | test reports will appear here as they publish

Software testing guides

AI Tool Tests: 27 guides on testing detectors, watermarks, provenance pipelines and disclosure labels.

The magazine, 2008 to 2014

Testing Experience was a free quarterly magazine for professional software testers, published in Berlin across 28 issues between March 2008 and December 2014. The archive gives each issue's date and theme and points to the guides that continue its subjects. The original issues are not distributed here.

All 28 issuesThe archive

Four issues from the archive | blue bars on the timeline
Issue Date Quarter Theme
Issue 1March 2008Q1 2008Launch issue: test management
Issue 4December 2008Q4 2008
Issue 12December 2010Q4 2010Open source testing tools
Issue 6June 2009Q2 2009Security testing

Who runs the tests

Daniel WeberEditor | signs every report

Tester, automation engineer and test manager before editing this site. He designs each test, signs each report and publishes the method with the result. No tool maker pays for a test, sees a result before publication, or can buy a placement.

Common questions

Who runs these tests?

Daniel Weber, the site's editor, designs and runs each test and signs the report. The site is independently owned and is not connected to any AI tool maker.

Do AI tool makers pay for tests or results?

No. No tool maker pays for a test, a placement or a result, and none sees a result before it is published.

How do you test an AI detector?

With a labelled sample set built before the run, a threshold fixed in advance, every score recorded with the tool version and date, and rates reported with 95 percent intervals. The full protocol is in the benchmarking guide.

Can I rerun a test myself?

Yes. Each report describes its sample set, its procedure and the settings used, so you can repeat it on your own documents and compare.

Is this the Testing Experience magazine?

Same name, relaunched in 2026 under independent ownership. The magazine stopped in 2014; this site keeps a page for each well-known issue and does not distribute the original issues.

Can I reuse the figures and tables?

Yes, for internal documents, training and presentations, with a credit to testingexperience.com. See the terms page.

Why do test reports name tools when the guides do not?

A guide teaches a method that should outlive any product, so it names none. A test report measures one tool at one version on one date, so it names the tool, the version and the date. Both use the same protocol.