Week 4: Data management: gathering your corpus
Can another person understand how you obtained these records?
Design stage: Data management. Examine how a source, its structure, and a standardized workflow shape the evidence you can use, then build and document your own corpus.
Prepare for class
Read: Chapter 4; gathering literature in your field. The chapters are in Canvas, and the companion page has the materials that go with them.
Reading focus: Follow the chapter from source to structure to quality to workflow. It applies Chapter 3’s framework to a corpus you build. Notice where each exercise states its deliverable.
Bring your source review and a draft boundary for your corpus. If JSON or parquet is unfamiliar, work through the companion’s text-data primer first.
Weekly time budget
Outside class: 360 minutes (6 clock hours) maximum planned work per student. These allocations include all required preparation and submissions; they are planning estimates, not time logs for grading.
| Work | Planned minutes |
|---|---|
| Reading and discussion preparation | 90 |
| Exercises and project work, including troubleshooting | 180 |
| Writing, documentation, and reflection | 90 |
| Total | 360 |
Retrieve your corpus in small steps, saving the raw response and the query before you change anything. Draft the entity relationship diagram and the codebook from the records you actually have.
How the weekly cap works. If required work will exceed the budget, save your attempt and contact the instructor to adjust scope or arrange support.
From the book
Exercises 4.2.1 and 4.2.2 give your corpus a structure and a codebook. The chapter’s advice is to serve the research, not the engineering: build the tables your question needs and say what you left out.
- Exercise 4.2.1: Drawing an entity relationship diagram. Required.
- Exercise 4.2.2: Creating a codebook. Required.
The book gives each exercise’s steps. What to submit, and when.
On the companion website: Chapter 4 materials and OpenAlex walkthrough · Working with text data · Data management tools
Monday, February 1: Retrieve a corpus and preserve its origin
12:45–2:00 PM · PV A203
Follow the companion’s OpenAlex walkthrough: a first query, cursor paging, and filters. Save the raw response before touching it, and record the search string, filters, retrieval date, and count. Flatten the records to a table with a stable document ID.
Check the reasoning: A successful response does not show that every requested record arrived. Compare the count the API reports with the number of records you saved, and separate the retrieval date from the publication dates in the data.
Wednesday, February 3: Structure workshop: diagram and codebook
12:45–2:00 PM · PV A203
Do Exercise 4.2.1: divide your records into tables and draw the entity relationship diagram. Then start the codebook for Exercise 4.2.2, one entry per variable you intend to analyze. If you ask a model to draft the tables, audit its draft against your own.
Check the reasoning: For each table, name the unit of analysis and the key that identifies a row. Point to one structure the model proposed, or that you considered, that your research does not need.
Work to carry forward
Finish the diagram and the codebook, and save the codebook under kb/codebooks/. Log the corpus boundary as a decision in memory/DECISIONS.md.
Coming up: Milestone 2: Data management (15 points) and Reflection 1: data and setup (5 points), due Wednesday, February 17.
Assignments and deliverables · Your project · The book and companion website