Week 7: Concept representation: automated coding
How do different coding instruments change the same evidence?
Design stage: Concept representation. Turn a concept into measures, compare the instruments that produce them, and audit what the measures can be trusted to show.
Prepare for class
Read: Chapter 6; automated coding. The chapters are in Canvas, and the companion page has the materials that go with them.
Reading focus: Read the sections on dictionary coding and large language models for this week’s two exercises. Skim supervised classification and topic analysis so you know what the optional exercises offer.
Bring your frozen coding corpus and construct definition. If you have never written a coding prompt, read the companion’s prompting primer first.
Weekly time budget
Outside class: 360 minutes (6 clock hours) maximum planned work per student. These allocations include all required preparation and submissions; they are planning estimates, not time logs for grading.
| Work | Planned minutes |
|---|---|
| Reading and discussion preparation | 90 |
| Exercises and project work, including troubleshooting | 210 |
| Writing, documentation, and reflection | 60 |
| Total | 360 |
Build the dictionary and the first promptbook on a small subsample. Develop on about 20 documents before running anything larger. A classifier and topic models are optional.
How the weekly cap works. If required work will exceed the budget, save your attempt and contact the instructor to adjust scope or arrange support.
From the book
Exercises 6.1.2 and 6.1.3 build two deductive instruments for the same concept. The chapter treats a prompt as a codebook written for a machine coder. The promptbook holds the instrument in force, and the promptlog holds its history.
- Exercise 6.1.2: Dictionary-based coding. Required.
- Exercise 6.1.3: Prompt-based coding with LLMs. Required.
- Exercise 6.1.4: Supervised coding with labeled data. Optional.
- Exercise 6.2.1: Analyzing topic modeling tools. Optional.
- Exercise 6.2.2: Topic modeling analysis. Optional.
The book gives each exercise’s steps. What to submit, and when.
On the companion website: Chapter 6 materials · Prompting and its failure modes · Automated coding tools · Worked notebooks
Monday, February 22: A dictionary as an instrument
12:45–2:00 PM · PV A203
Do Exercise 6.1.2. Draft include and exclude terms for each category and apply them to your corpus. Read the highest- and lowest-scoring documents for each category, then revise the dictionary once and record what changed. Keep both the raw count and the rate per thousand words.
Check the reasoning: Find a term that is counted in the wrong sense, and a relevant document your terms missed. Each is a reason for a revision you can document.
Wednesday, February 24: A prompt as an instrument
12:45–2:00 PM · PV A203
Begin Exercise 6.1.3. Write the promptbook entry: construct and coding rules, unit and input format, output schema, worked examples, guardrails, and the model version and settings. Run it on about 20 documents and parse the output. Then write a deliberately flawed variant for the stress test.
Check the reasoning: Your prompt should be able to answer “insufficient evidence.” Find a label the model could only have reached from background knowledge and not from the text, and add an instruction that requires quoted evidence.
Work to carry forward
Save both dictionary versions. Put the promptbook under kb/codebooks/ and log each prompt revision in memory/PROMPTLOG.md.
Coming up: Milestone 3: Concept representation (20 points) and Reflection 2: measurement (5 points), due Wednesday, March 10.
Assignments and deliverables · Your project · The book and companion website