Skip to content
TestingExperience

AI Tool Tests

Independent tests of AI tools

27 guides on testing detectors, watermarks, provenance pipelines and disclosure labels.

Open section

Start here, in order

Methods and protocols

How we test 2.0%CI 1.3 to 3.1 False positive rate in the worked benchmark: 20 of 1,000 human-written documents wrongly flagged. Read the protocol
Editorial methodGlossaryReport an errorNo tool maker pays for a test

Software testing

Reference guides for testers

26 guides on planning, automating and designing tests, written to outlive any tool.

Testing Techniques

Sections

Recently updated

Test ManagementTest AutomationAgile TestingTesting TechniquesGuides name no products

Magazine

The magazine, 2008 to 2014

A free quarterly for professional software testers, published in Berlin across 28 issues.

The archive

Archive

Issue pages

28 issues Each issue page states the date and the theme where it is known, and nothing beyond it. The original issues are not distributed.
The archiveAll issuesEditorial methodRecords, not reproductions

Recommended Reading for Software Testers

The original site carried a page of reading recommendations. This is its replacement: the books, standards and communities that are still worth a tester's time, grouped by what you need them for.

Reading listUpdated

A reading list is only useful if it says why. Each entry below earns its place for one reason a working tester would recognise, and the list is deliberately short. It is organized by need rather than by type, because nobody wants 'a standard'; they want to know what to cite when a customer asks for a test process, or what to read before their first exploratory session. No vendor documentation appears here, and no entry is an affiliate arrangement; titles and authors are given so that you can find them wherever you buy or borrow books.

For the vocabulary and the exam

  1. ISTQB Foundation Level syllabus and glossary

    Syllabus

    ISTQB

    Free, current and the shared language of the profession; read them for definitions, not for thinking. Serves: For the vocabulary and the exam

  2. ISO/IEC/IEEE 29119

    Standard

    Standards body

    The international test process and documentation standard; part 3 is what regulated customers ask about. Serves: For the vocabulary and the exam

For test management

  1. Managing the Testing Process

    Book

    Rex Black

    The most complete treatment of running a test effort end to end. Serves: For test management

  2. Lessons Learned in Software Testing

    Book

    Cem Kaner, James Bach, Bret Pettichord

    293 numbered lessons and the best book to hand a new test lead. Serves: For test management

  3. TMMi model document

    Book

    TMMi Foundation

    Freely available and, with Erik van Veenendaal's work on the model, the reference for test process improvement. Serves: For test management

For thinking about testing

  1. Writing on rapid software testing

    Book

    Michael Bolton, with James Bach

    Bolton's blog and their joint writing are the sharpest current arguments about what testing is for. Serves: For thinking about testing

  2. Perfect Software and Other Illusions About Testing

    Book

    Jerry Weinberg

    Short and permanently relevant on what management misunderstands about testing. Serves: For thinking about testing

  3. Explore It!

    Book

    Elisabeth Hendrickson

    The practical manual for exploratory testing, and the second book for anyone who must test well by Friday. Serves: For thinking about testing

For agile teams

  1. Agile Testing and More Agile Testing

    Book

    Lisa Crispin, Janet Gregory

    Define the tester's role on cross-functional teams, and help when testers sit idle early in each sprint. Serves: For agile teams

  2. Specification by Example

    Book

    Gojko Adzic

    The reference for acceptance test driven development and living documentation. Serves: For agile teams

  3. Writing on program management

    Book

    Johanna Rothman

    Covers the multi-team ground of the original Knowledge Transfer tutorial on agile program management. Serves: For agile teams

For techniques

  1. A Practitioner's Guide to Software Test Design

    Book

    Lee Copeland

    The clearest book on the classical techniques, with one chapter per technique. Serves: For techniques

  2. The Art of Software Testing

    Book

    Glenford Myers

    The 1979 original that still frames black-box and white-box thinking. Serves: For techniques

  3. xUnit Test Patterns

    Book

    Gerard Meszaros

    The reference on why automated tests rot and how to structure them so they do not. Serves: For techniques

  4. OWASP Testing Guide

    Book

    OWASP

    A free security testing methodology, and the one to use for web applications when an assessor is coming. Serves: For techniques

  5. OSSTMM

    Book

    ISECOM

    A free security testing methodology, and the one to use for the operational scope of an assessment. Serves: For techniques

Communities

  1. Ministry of Testing

    Community

    Community

    Runs the largest practitioner community and a long-running conference series. Serves: Communities

  2. Association for Software Testing

    Community

    Community

    The non-profit behind the context-driven school of software testing. Serves: Communities

  3. ISTQB member boards

    Community

    Community

    Run the national certification bodies and, in several countries, free evening meetups. Serves: Communities

Standards worth reading, and which part

  1. ISO/IEC/IEEE 29119-1

    Standard

    Standards body

    Concepts and vocabulary, published in 2022; read it once. Serves: Standards worth reading, and which part

  2. ISO/IEC/IEEE 29119-2 (2021)

    Standard

    Standards body

    The test processes in three layers, and the part to cite when asked what a test process is. Serves: Standards worth reading, and which part

  3. ISO/IEC/IEEE 29119-3 (2021)

    Standard

    Standards body

    Test documentation and its templates, and the part contracts name; it superseded IEEE 829. Serves: Standards worth reading, and which part

  4. ISO/IEC/IEEE 29119-4 (2021)

    Standard

    Standards body

    Test design techniques with coverage measures, the reference behind this site's technique guides. Serves: Standards worth reading, and which part

  5. ISO/IEC/IEEE 29119-5 (2016)

    Standard

    Standards body

    Keyword-driven testing, of interest only if you build that kind of framework. Serves: Standards worth reading, and which part

  6. Technical report 29119-11 (2020)

    Standard

    Standards body

    Guidelines on testing AI-based systems, the closest thing to a standard for testing AI systems. Serves: Standards worth reading, and which part

  7. ISO/IEC 25010

    Standard

    Standards body

    The quality characteristics that performance and usability testing are measured against. Serves: Standards worth reading, and which part

  8. IEEE 1028

    Standard

    Standards body

    The standard to read for reviews and inspections, outside the 29119 family. Serves: Standards worth reading, and which part

Choosing between them by need. If you must write a test plan a customer will audit next month: ISO/IEC/IEEE 29119-3 for the contents, Black for the management around it. If you are a developer or new tester who has to test well by Friday: Kaner, Bach and Pettichord first, Hendrickson second. If your team has just gone cross-functional and the testers are idle for the first week of each sprint: Crispin and Gregory. If your acceptance tests are long, brittle and owned by nobody: Adzic on specifying with examples, then Meszaros on why automated tests rot. If you need to design tests for a rule with many inputs: Copeland, one chapter per technique. If you are asked to prove your test process is mature: the TMMi model document, free, then van Veenendaal. If a security assessor is coming: the OWASP Testing Guide for web applications and the OSSTMM for the operational scope. If your manager thinks testing can be finished: Weinberg, and give them the copy.

Standards worth reading, and which part. ISO/IEC/IEEE 29119 is five parts and a technical report, and nobody needs all of them. Part 1 (2022) is concepts and vocabulary; read it once. Part 2 (2021) is the test processes in three layers (organisational, management, dynamic) and is the part to cite when asked what a test process is. Part 3 (2021) is test documentation, the templates, and the part contracts name. Part 4 (2021) is test design techniques with coverage measures, the reference behind this site's technique guides. Part 5 (2016) is keyword-driven testing, of interest only if you build that kind of framework. Technical report 29119-11 (2020) is guidelines on testing AI-based systems, the closest thing to a standard for the Testing AI Systems hub. Outside 29119: ISO/IEC 25010 for the quality characteristics that performance and usability testing are measured against, and IEEE 1028 for reviews and inspections. All of these are paid documents; the ISTQB syllabi, the TMMi model, the OWASP guides and the OSSTMM are free.

Sources

  1. The original recommendations page, Internet Archive capture of February 2010
  2. Lisa Crispin and Janet Gregory, Agile Testing and More Agile Testing
  3. Elisabeth Hendrickson, Explore It! (The Pragmatic Bookshelf)
  4. ISTQB Certified Tester Foundation Level (CTFL) v4.0 syllabus and overview