Week 8: Concept representation: validity audit and review
Whose cases do your errors fall on?
Design stage: Concept representation. Turn a concept into measures, compare the instruments that produce them, and audit what the measures can be trusted to show.
Prepare for class
Read: Revisit Chapters 5–6; the validity and reliability audit. The chapters are in Canvas, and the companion page has the materials that go with them.
Reading focus: Reread Chapter 5 on validity, reliability, and the marginal-cases problem, then Chapter 6’s audit section. The audit asks for evidence at the level of each class, not one overall score.
Bring both label sets from the stress test, and a random sample of documents you have coded by hand without looking at the model’s labels.
Weekly time budget
Outside class: 360 minutes (6 clock hours) maximum planned work per student. These allocations include all required preparation and submissions; they are planning estimates, not time logs for grading.
| Work | Planned minutes |
|---|---|
| Reading and discussion preparation | 60 |
| Exercises and project work, including troubleshooting | 240 |
| Writing, documentation, and reflection | 60 |
| Total | 360 |
Run the stress test, check agreement against a sample you coded by hand first, and carry out the validity and reliability audit on your strongest measure.
How the weekly cap works. If required work will exceed the budget, save your attempt and contact the instructor to adjust scope or arrange support.
From the book
Exercise 6.6 is the audit that closes the chapter’s exercises: a criteria check, at least two validation strategies, a marginal-cases audit with class-level metrics, and an honest assessment of what remains uncertain.
- Exercise 6.1.3: Prompt-based coding with LLMs. Required.
- Exercise 6.6: Validity and reliability audit. Required.
- Exercise 6.3: Triangulating deductive and inductive coding. Optional.
- Exercise 6.4: Diagnosing representation effects. Optional.
- Exercise 6.5: From empirical to theoretical. Optional.
The book gives each exercise’s steps. What to submit, and when.
On the companion website: Chapter 6 materials · Prompting and its failure modes · Supervised learning, gently, with an audit checklist
Monday, March 1: Stress test and validity audit
12:45–2:00 PM · PV A203
Finish Exercise 6.1.3: run the careful and the flawed prompt on the same few hundred documents, compare the label distributions, and read ten documents that flipped. Then do Exercise 6.6 on your strongest measure: check the construct against at least three of the conceptual-goodness criteria, apply two validation strategies, and audit the marginal cases.
Check the reasoning: Report precision and recall for each class against a majority-class baseline, with the counts behind them. Say whether the errors fall on the cases your question cares most about.
Wednesday, March 3: Exercise review 2: your measures
12:45–2:00 PM · PV A203
Demonstrate one disagreement between your instruments and what the source text shows. A classmate codes ten of your documents from your construct definition alone, and you read the disagreements together. Close with the instructor’s worked audit.
Check the reasoning: A disagreement has three possible causes: the model was wrong, you were wrong, or the case is ambiguous. Sort your reviewer’s disagreements into those three, and revise the prompt, your reading, or the definition to match.
Work to carry forward
Revise after the review and assemble Milestone 3. Draft Reflection 2 from the divergence between your instruments and the marginal-cases audit.
Coming up: Milestone 3: Concept representation (20 points) and Reflection 2: measurement (5 points), due Wednesday, March 10.
Exercise review 2, ungraded. Bring: Your coding corpus, both dictionary versions, the promptbook, both label sets from the stress test, and the audit as far as you have taken it. Your reviewer codes ten of your documents from your construct definition alone, compares the result with your labels, and reads the disagreements with you. This work goes into Milestone 3. How the reviews work.
Assignments and deliverables · Your project · The book and companion website