Case 1 6 September 2026
The CDA-AMC Sponsor Summary of Evidence, filled from the STEP 1 CSR
The case behind Can’t ChatGPT just do that?. The same two files, the agency’s template as issued and the pivotal trial’s clinical study report, given to ChatGPT in the browser and to Palmera. Every field in every returned file was compared with the blank template, every figure was looked for in the CSR, and every quoted passage was looked for word for word. The files and the checks are below, so anyone can repeat the count.
What was run
| Form | CDA-AMC Sponsor Summary of Evidence template, as issued: 802 fields, 32 tables, 50 pages |
| Source | STEP 1 clinical study report, NN9536-4373, semaglutide 2.4 mg versus placebo: 4,121 pages. One source on purpose, so every answer could be checked. Not redistributed here. |
| ChatGPT | chatgpt.com, Pro account, GPT-6 Astra (Medium) in Work mode, both files attached. Run 1 sent 19:25 UTC; a follow-up in the same chat at 20:04 naming the missed fields; run 2 in a fresh chat at 20:08 with the same request. |
| Palmera | Beta build of 6 September. A new project set up through the normal browser screens, click by click, nothing scripted: PICO typed in, CSR uploaded and prepared, deliverable created from the template, one click to fill, four suggestions accepted, two rewritten into prose, one tidied by hand in Word (the CSR’s reference numbers removed), a reviewer’s comment left on one field, the file exported. Fill started 22:47 UTC; the last two accepts and everything after them were done the next morning. |
Results
| ChatGPT, run 1 | ChatGPT, run 2 | Palmera | |
|---|---|---|---|
| Time to the first pass | 23 min 21 s | 24 min 19 s | 10 min: 4 preparing the CSR, 6 min 24 s filling 791 fields |
| Fields answered | 235 of 802 | 260 of 802 | 4, each accepted by a person · 435 suggestions offered |
| Fields written over with a “not reported” note | 567 | 542 | 0 · 356 marked “no supported answer”, guidance left as issued |
| Fields left exactly as issued | 0 | 0 | 798 of 802 in the exported file |
| Figures in the answers found in the CSR | 431 of 431 | 885 of 885 | 3,781 of 3,781 |
| Figures on the page the answer cites | 367 of 431 | 665 of 885 | 749 of 752 quoted passages are on the page cited |
| Quoted passages found word for word | none quoted | none quoted | 752 of 763 |
| The template afterwards | table captions turned into headings · 50 pages became 71 · the follow-up added 162 fields and changed 43 | 50 pages became 61 | the agency’s, 4 fields changed |
| Run again | agrees with run 2 on 229 answers, differs on 37 | agrees with run 1 on 229 answers, differs on 37 | two runs the same day: 404 fields answered in both, 71 in only one |
“Answered” means field text that is neither the agency’s guidance nor a note saying the CSR has nothing (“Not reported in the supplied CSR”, “No second study in CSR” and the like). Figures are matched with thousands separators removed and against every page an answer cites, including ranges. No figure in any ChatGPT answer was absent from the CSR. The ChatGPT counts in the table are from the classification and figures files, not from the first-pass field-by-field files, which are kept for the field texts. Palmera’s fill covered 791 of the 801 fields on its template; the ten it did not take on are counted neither as answered nor as “no supported answer”.
What went wrong on each side
- ChatGPT changed the agency’s file. Table captions turned into headings, the form grew from 50 pages to 71, and the follow-up prompt added 162 fields of its own. 64 of run 1’s 431 figures and 220 of run 2’s 885 are not on the page the answer cites. No passage is quoted, so nothing can be checked word for word.
- Palmera offered suggestions in fields the CSR cannot answer. 161 suggestions in second-study and extension columns drew on STEP 1 data; 84 more sat in indirect-comparison, systematic-review and economic fields. Each shows its passage and section beside it and is left out in a click, but the software should have withheld them. A scope rule is being added.
- Some Palmera suggestions quote the right CSR, wrong table. Of 28 suggestions about this trial read by hand, about a third answered the field; more than half quoted a real passage from another table, most often one SF-36 sensitivity table offered for body-weight, disposition and P-value fields. Table-to-field matching is being tightened.
- 87 fields about this trial came back “no supported answer” though the CSR probably answers them: baseline means, screening-failure reasons, per-protocol counts.
- 11 of 763 Palmera passages are imperfect: one not found word for word (a patient listing on page 688), five close but not exact, five too short to check; and 3 of the 752 found are cited to the wrong page.
- Palmera’s template carries 801 of the agency’s 802 fields. The missing one is an empty heading line the agency’s file marks as a field.
- ChatGPT’s counts depend on the counting rule above. An earlier count treated “No second study in CSR” as an answer and put run 2 at 507; the corrected figure is 260.
Files
Everything needed to repeat the count. The ChatGPT files are exactly as downloaded from the chat. The Palmera export is exactly as downloaded from the project. The checks are the scripts’ output; the scripts are included.
| File | Size | What it is |
|---|---|---|
| cda-amc-sponsor-summary-of-evidence-template.docx | 906.5 KB | The agency’s template as issued, the input on both sides |
| chatgpt-run1-completed.docx | 899.7 KB | ChatGPT run 1, as returned |
| chatgpt-run1-after-follow-up.docx | 903.2 KB | ChatGPT run 1 after the follow-up prompt, as returned |
| chatgpt-run2-fresh-chat.docx | 893.4 KB | ChatGPT run 2, fresh chat, as returned |
| chatgpt-run1-printed.pdf | 406.6 KB | ChatGPT run 1, printed to PDF |
| chatgpt-run2-printed.pdf | 384.1 KB | ChatGPT run 2, printed to PDF |
| chatgpt-screenshots.zip | 395.1 KB | Screenshots of the ChatGPT chats: the prompt, the wait, the download |
| chatgpt-page-renders.zip | 700.7 KB | The Table 1 page of the blank and of each ChatGPT file, as images |
| palmera-export.docx | 913.4 KB | Palmera’s export after four accepts, two rewrites, one hand edit and one comment (9b), as downloaded |
| palmera-export-printed.pdf | 366.4 KB | The same export, printed to PDF |
| palmera-suggestions-with-evidence.json | 1.4 MB | Every Palmera suggestion with its passage, page and section, as the product records it |
| palmera-screenshots.zip | 1.9 MB | Screenshots of the Palmera run, in order |
| palmera-run-log.txt | 6.9 KB | Timestamps of the Palmera run |
| chatgpt-run1-field-by-field.json | 287.9 KB | ChatGPT run 1, every field: text, cited pages, figures. First pass: its state and figure totals are superseded by the two classification files below |
| chatgpt-run1-after-follow-up-field-by-field.json | 298.9 KB | ChatGPT run 1 after the follow-up, every field. First pass, as above |
| chatgpt-run2-field-by-field.json | 330.6 KB | ChatGPT run 2, every field. First pass, as above |
| chatgpt-answer-vs-note-classification.json | 61.6 KB | Which ChatGPT fields count as answers and which as notes, per run: the “fields answered” and “written over” counts in the table come from this file |
| chatgpt-figures-on-cited-page.json | 327.8 KB | Every figure in every ChatGPT answer, the pages it cites, and where the figure was actually found: the figure counts in the table come from this file |
| palmera-passages-and-figures.json | 382.8 KB | Palmera, every field: passages found, page checks, figures |
| palmera-passages-and-figures.csv | 92.7 KB | The same, one row per field |
| check-scripts.zip | 12.8 KB | The scripts that produced the checks |