AI Visibility Study: Methods, Data, and Corrections

This page holds everything behind the post I built a tool that scores AI visibility. Then I tested whether the score predicts anything. Twice. The protocols were written before the data existed. The amendments are dated. The results files include the corrections, with the wrong numbers shown beside the right ones. If you want to check the work, start with the freeze dates, then the protocols, then the mention data.

When was each part fixed?

Study 1's amendments were written after its capture and, in two cases, after a first result was seen. Both cases are documented in the amendments themselves, and both corrections moved the result toward the hypothesis, not away from it.

Study 1: enterprise WordPress agencies

Population: the 64 agencies in WordPress VIP's service partner directory as of August 26, 2026. Instrument: our structural audit, run from the outside on every site. Capture: ten buyer questions, ChatGPT and Perplexity, fresh sessions, 22 runs, 148 mentions. Result: named firms averaged 73.4, never-named 70.5, a gap of 2.9 points, p = 0.13, minimum detectable effect about 6 points at 80% power. The size covariate was not collected; that gap is stated in the results and was the reason study 2 was designed the way it was.

Files, in reading order:

1. Query protocol with amendments A1 to A6 (query-protocol.md)
2. Results (RESULTS.md)
3. Mention data, one row per firm named per run (mentions.csv)
4. Domain verification log for firms named from outside the directory (t22-domain-verification.md)
5. Response transcripts and coding notes, 22 runs (responses folder)

study-1-materials.zip

Study 2: IRS-certified PEOs

Population: the 60 brands consolidated from the 122 legal entities on the IRS active certified PEO list as of August 27, 2026; 45 domains audited and scored, one edge-blocked and excluded, six with no discoverable site. Capture: ten buyer questions including a positive control, ChatGPT and Perplexity, fresh sessions, 26 runs, 137 mentions. Result: named firms averaged 73.6, never-named 72.3, a gap of 1.3 points, p = 0.56, minimum detectable effect about 8 points. Company size, collected before capture, separated the groups at p = 0.0004.

Files, in reading order:

1. Feasibility gates, frozen before the corpus audit, with verdicts (gates.md)
2. Query protocol 2, frozen before capture (query-protocol-2.md)
3. Results, including sensitivity cuts and the one departure from the frozen protocol (RESULTS-2.md)
4. Mention data, 26 runs (mentions.csv)
5. Company-size ranks, one per scored domain (sizes-rank.csv)
6. Scored corpus domains (corpus-domains.csv)
7. Pilot mentions, gate evidence only, never merged into the analysis (pilot-mentions.csv)
8. Response transcripts and coding notes, 26 runs (responses folder)

study-2-materials.zip

Analysis code

The permutation test, the power simulation, and the cross-study comparison are included so the numbers can be reproduced from the mention data and the score lists. The audit scorer itself is not included; it is the product, and its limits are described in both results files.

1. analyze_mentions.py, the primary test and confound check
2. power_check.py, the minimum detectable effect simulation
3. compare_studies.py, bootstrap intervals for both studies side by side

analysis-code.zip

What is not here

The audit JSON for each site, because it contains the scorer's internals. The roster construction notes for study 2, because they include entity-resolution evidence about individual companies. Screenshots of each session, which exist and can be provided on request for any specific run.

Corrections

Study 1's analysis was corrected twice after a first result was written: a non-reproducible permutation pool, and a benchmark site inside the comparison group. Both are amendment A5. Study 1's null was rescoped as a bounded null with a published minimum detectable effect in amendment A6. Study 2's analysis ran at the wrong random seed; the specified seed gives the same verdict, and both values are in RESULTS-2. Seven wrong-entity domains were caught across the two studies during verification and are logged in the files above.

If you find something else, email [email protected]. Corrections go on this page with a date.