Practice guide

How to read a study, work an exercise chapter, and use the in-class reviews.

The book alternates between two kinds of chapter, and so does your week-to-week work. Theory chapters end with a Practice section on published studies, which we use for discussion. Exercise chapters give you numbered exercises to do on your own corpus, which become your milestones. This page explains how to do both, and how the in-class reviews connect them. Nothing described here is graded or uploaded on its own.

All of it sits inside the six-hour weekly outside-class budget. The companion website has the technical walkthroughs, so this page links to them instead of repeating them.

Read and discuss an empirical study

Chapters 3, 5, 7, and 9 each close with a Practice section that asks you to analyze published studies through one design stage. The companion website’s chapter page filters its studies database to that stage, with a short annotation on each study.

Discussion Practice section Choose a study from What to examine
Monday, January 25 Chapter 3: data management The chapter’s annotated studies Critique how the study handled its data in terms of the 3S framework: where the data came from (source), how they were organized (structure), and how the work was made consistent and repeatable (standardization).
Monday, February 15 Chapter 5: concept representation The chapter’s annotated studies Examine how the study turns a core concept into a measure along four dimensions: worldview alignment; a deductive, inductive, or integrative measurement approach; featurization and any auxiliary instrument; and the evidence offered for validity and reliability.
Monday, March 8 Chapter 7: data analysis The chapter’s annotated studies Identify the study’s epistemic purpose (descriptive, explanatory, predictive, or integrative) and the computational capacities its method relies on. Explain how it handles epistemic opacity, the semantics-to-application gap, and temporal dynamics, whether its validation suits its purpose, and what you would do differently.
Monday, April 5 Chapter 9: scientific communication The chapter’s annotated studies Assess how inspectable, reproducible, and interpretable the study is. Check what it shares, place it in the four-category reproducibility framework, read one visualization through Kirk’s four-stage design process, examine how it communicates uncertainty, and say what you would change.

Write a half-page Practice note. Choose one study from the chapter’s list, read the paper itself and not only its annotation, and answer the Practice section’s questions for it. Cite the passage, table, or figure you rely on, and end with one implication for your own project. The book asks for a full page, on two studies in Chapters 3 and 5. We shorten it to fit the weekly budget, and you are welcome to write more.

Open a discussion. Each discussion starts with a 10-minute student-led opening on one study, and we arrange in Week 2 who opens which. If you are opening, prepare that in place of the note: the study’s question, one consequential design choice, the evidence for that choice, and a limitation. End with a question the room can examine using the study’s own evidence. A few slides or an annotated figure are enough.

The database’s coverage is uneven across stages and thins after 2023. Treat that as a property of the collection, not of the field, and add a study from your own area if the list does not reach it.

Chapter 6 has a Practice section too. It asks you to compare a published study’s coding pipeline with the one you built. It is optional in P539, and useful preparation for Reflection 2.

Work an exercise chapter

Chapters 4, 6, 8, and 10 are built from numbered exercises. Each one tells you what you need, the steps to follow, and the deliverable to produce. The assignments page says which exercises P539 requires and which are optional.

For each exercise:

  1. Read it in the book first, including the “You will need” line. Several exercises reuse the output of an earlier one.
  2. Do it on your own corpus. Start with a small sample you can inspect by hand before you run anything on the whole collection.
  3. Keep your first attempt. Save the output before you correct it. A revision is easier to explain, and to grade, when the earlier version still exists.
  4. Write the deliverable the exercise names, at the length it asks for, and put the files where the Chapter 2 project structure expects them.
  5. Log what you decided in memory/DECISIONS.md, and any prompt revision in memory/PROMPTLOG.md.

The companion website carries the code walkthroughs and setup notes for each exercise chapter, the primers to read first if a technique is new to you, and tool guides that are kept current.

Exercise chapter Walkthrough and materials Read first if it is new to you Tools
4. Gathering literature in your field OpenAlex retrieval, paging, filters, and flattening Working with text data Data management
6. Automated coding Corpus preparation, dictionary, prompt, classifier, and audit steps Prompting and its failure modes · Supervised learning, gently Automated coding
8. Applying computational methods Network, simulation, prediction, and embedding notes · network notebook and sample data Supervised learning, gently Computational analysis
10. Communicating your pipeline Environment freezing, report and dashboard scaffolding, and the uncertainty layer Chapter 2 project templates Communication

When your own data are not ready. The companion’s worked notebooks come with saved outputs, and its sample network data, a co-citation network of works, let you practice Chapter 8’s network steps before your own corpus is ready. Use them to learn the step, then return to your own records for the milestone. If troubleshooting would take you past the week’s budget, save the error and bring it to class.

When you use AI to write code. Check that generated code does what you intended, not something plausible but different. Run it on a case small enough to verify by hand, and read the companion’s note on auditing generated code. Using AI explains what to record.

Exercise reviews

Each exercise chapter ends with an ungraded review in class, held before the milestone is due, usually by one week. Bring the work in whatever state it is in. Unfinished work with a specific question gets more from a review than finished work with none.

A review meeting has three parts:

  1. Demonstrate. In a small group, show one artifact and one decision in about three minutes: what you tried, what a check revealed, and what you changed.
  2. Be reviewed. A classmate does the task in the table below with your materials, without coaching. Watch where they hesitate. You do the same for someone else.
  3. Compare. The instructor closes with a worked example of the same exercises, so you can check your reasoning against it.
Review Date Bring What your reviewer does
1. Chapter 4, for Milestone 2 February 10 Your retrieved corpus, source review, entity relationship diagram, codebook, data-quality assessment, and workflow diagram, in whatever state they are in. Your reviewer follows your workflow documentation without asking you anything and marks the first step they could not reproduce.
2. Chapter 6, for Milestone 3 March 3 Your coding corpus, both dictionary versions, the promptbook, both label sets from the stress test, and the audit as far as you have taken it. Your reviewer codes ten of your documents from your construct definition alone, compares the result with your labels, and reads the disagreements with you.
3. Chapter 8, for Milestone 4 March 31 Your purpose statement, the network or predictive analysis with its baseline or comparison, and a draft of the AI-assisted validity statement. Your reviewer states, from your output alone, what claim it supports and what would change that claim. Compare their reading with your purpose statement.
4. Chapter 10, for Milestone 5 April 21 A repository or archive with the pinned environment, the documentation package for your AI-assisted coding, a draft of the integrative artifact, and the uncertainty figure. Your reviewer clones or unpacks the project, runs it, and regenerates a small sample of coded output from your promptbook. Record what they could not reproduce.

Afterward, fix what your reviewer could not follow, and record what you changed and why in memory/DECISIONS.md. Say so too when you decide not to take a suggestion. If you miss a review, arrange an exchange with a classmate before the milestone is due and tell the instructor; see participation expectations.

Use the Reflective Exercises

Every chapter from 1 to 11 ends with Reflective Exercises. We use some of them to open or close a class meeting, and your three graded reflections draw their prompts from them. Chapter 11’s set is also the outline of the ethics review in Milestone 5: it asks you to work back through the pipeline you built, stage by stage.

You do not need to answer every question in writing. Read them when you finish a chapter’s work, and keep a note of the one that changed how you see your project.