Assignments

Your grade is based on six project milestones (80 points) and three individual reflections (20 points). Each point represents one percentage point of the course grade; the letter-grade scale converts your total. Submit graded work in Canvas and follow the submission rules.

The milestones follow the required text, Computational & AI-Assisted Methods for Social Sciences: A Research Design Primer. Milestones 2 through 5 each collect the deliverables of the exercises you worked on in the preceding weeks, done on your own corpus. Milestone 1 sets the project up, and Milestone 6 presents it. The book gives the steps for every exercise. This page says which exercises count, what each one produces, and when it is due.

You have 3 late passes of 72 hours each for the interim submissions listed in the late-work policy. The last on-time version is graded when one exists. Work due at the final session follows the final-session and make-up rules.

Semester project

You build one project across the semester, on a topic you choose. Your project explains how to choose a topic and what data to use. The project is worth 80 points across six milestones.

Swipe or scroll tables sideways to read every column. With a keyboard, focus a table and use the arrow keys.

All required work fits the six-hour weekly outside-class budget. The book has 33 exercises in its four exercise chapters. P539 requires a core set of them, gives you a choice among several in Chapter 8, and treats the rest as optional extensions. Optional work is not graded separately and does not raise the expectations for anyone else; include it in your submission if you do it.

Milestones 2 through 5 are each reviewed in class before they are due, usually a week earlier. Bring the work to the exercise review, revise it after a classmate has tried to follow it, and then submit.

How to submit a milestone

For Milestones 1 through 5, submit two things to the Canvas assignment:

1. One PDF report containing the written pieces in the order listed for that milestone, each under its own heading, followed by your AI-use disclosure.
2. Your project folder at a fixed version, as a zip archive or as a link to a specific commit or release. A link to a folder that keeps changing does not show which version you submitted.

Keep files where the Chapter 2 project structure puts them: codebooks and promptbooks under kb/codebooks/, the promptlog and decisions under memory/, raw data under data/raw/, and scripts under analysis/. If a data file is too large or cannot be shared, include the query that retrieves it and a small sample. For Milestone 6, upload your slides or the link you will present from.

Milestone 1: Project setup and direction

5 points · Chapters 1–2 · Due: Week 3 · January 27. Set up the project environment from Chapter 2 and draft the direction of your project.

Submit:

• Project workspace. Your project folder, built from the Chapter 2 skeleton, with the three files of the chapter’s minimal starting point filled in for your project: AGENTS.md (file conventions, verification expectations, and what to do when uncertain), memory/CURRENT.md (research direction, one or two open questions, and next tasks), and kb/README.md (a map of the materials you have and where they are).
• Direction memo. One page: the topic you chose and why it matters to a policy, management, or scholarly audience; a draft research question; the concept you expect to measure; the corpus you expect to build (search terms, years, and source); and what you expect each of the four stages to require.
• Setup record. A short note in your project README.md naming the computing route you chose (browser notebook or local environment), the language you will use, and how you will get access to a language model.

Before you write the memo, try Chapter 1’s three basic moves on your topic: summarize and interrogate one paper, draft a literature search, and ask a model to critique your design. Keep the verification habit that goes with each.

How the points are earned: Workspace files that an agent and a reader can follow 2; question, concept, and feasible corpus 2; setup and model access recorded 1.

On the companion website: Chapter 2 project templates · Getting started

Milestone 2: Data management

15 points · Chapters 3–4 · Due: Week 6 · February 17. Build the corpus your project will use, and document it so a classmate could rebuild it.

Submit:

• The corpus. The recorded query (search terms, filters, retrieval date, and record count), the raw response saved unmodified under data/raw/, and the flattened table with a stable document ID.
• Exercise 4.1.3, Inspecting a data source. A one-page source review that answers the exercise’s prompts on the raw data, preprocessing, content, and retrieval, classifies the source by format, security, and interface, and states your decision about whether to build the corpus from it.
• Exercise 4.2.1, Drawing an entity relationship diagram. An entity relationship diagram of your normalized tables, with a short note naming each table’s unit of analysis and any structure you deliberately did not build.
• Exercise 4.2.2, Creating a codebook. A codebook covering every variable you intend to analyze (name, type, range or categories, coding rules, missing values, and a promptbook entry where an LLM helped build the variable), saved under kb/codebooks/.
• Exercise 4.3, Inspecting data quality in OpenAlex. A one-page data-quality assessment covering source, accuracy and completeness, and consistency, each with a concrete example from your own records and a stated plan for handling it.
• Exercise 4.4, Workflow. A workflow diagram with a short narrative for each stage, and the named and organized project directory it describes.

Optional extensions: Exercise 4.3.1, LLM-assisted screening with an agreement check.

If you have approval to use a source other than OpenAlex, apply each exercise to that source.

How the points are earned: Source review and corpus boundary 4; structure: entity relationship diagram and codebook 4; data-quality assessment with a handling plan 4; workflow, provenance, and an organized directory 3.

On the companion website: Chapter 4 materials and OpenAlex walkthrough · Working with text data · Data management tools

Milestone 3: Concept representation

20 points · Chapters 5–6 · Due: Week 9 · March 10. Turn one concept from your question into coded measures, and audit how far they can be trusted.

Submit:

• Construct definition. Your written definition of the concept, saved under kb/concepts/, with at least two categories and a clear, an ambiguous, and an excluded example.
• Prompt history. The memory/PROMPTLOG.md entry for each substantive prompt revision, with the audit finding that led to it.
• Exercise 6.1.1, Building a coding-ready OpenAlex corpus. The saved corpus file (one document-level table with stable IDs, text, and key metadata) and a one-page corpus memo stating the boundaries, the inclusion and exclusion rules, one difficult boundary decision and how you resolved it, and a provenance note.
• Exercise 6.1.2, Dictionary-based coding. The dictionary file in both versions, the scored corpus, and a half-page revision note recording what changed after you read the high- and low-scoring documents, and why.
• Exercise 6.1.3, Prompt-based coding with LLMs. The promptbook entry, both label sets (careful and deliberately biased), a table comparing their label distributions, and a short audit note on the recurring error patterns and the prompt revisions that addressed them.
• Exercise 6.6, Validity and reliability audit. The criteria assessment for one construct, the results of at least two validation strategies, the marginal-cases audit with class-level metrics, and an honest assessment of the remaining threats to validity.

Optional extensions: Exercise 6.1.4, Supervised coding with labeled data; Exercise 6.2.1, Analyzing topic modeling tools; Exercise 6.2.2, Topic modeling analysis; Exercise 6.3, Triangulating deductive and inductive coding; Exercise 6.4, Diagnosing representation effects; Exercise 6.5, From empirical to theoretical.

Develop the prompt on about 20 documents, stress-test it on a few hundred, and only then code the full corpus. The prompting primer gives the protocol.

Exercise 6.6 asks for the linkage table from the optional Exercise 6.5, From empirical to theoretical. If you did not do that exercise, audit the construct as you wrote it in your construct definition.

How the points are earned: Coding-ready corpus and corpus memo 4; dictionary and its documented revision 4; promptbook, stress test, and reliability check 6; validity and reliability audit 6.

On the companion website: Chapter 6 materials · Prompting and its failure modes · Automated coding tools

Milestone 4: Data analysis

15 points · Chapters 7–8 · Due: Week 13 · April 7. Apply one computational method to your coded corpus for a stated research purpose, and say what its output can support.

Submit:

• Purpose statement. One paragraph naming the epistemic purpose your analysis serves (descriptive, explanatory, predictive, or integrative), the computational capacity it relies on, and the evidence you need to assess the result.

One capacity exercise. Choose one:

• Exercise 8.1.1, Network construction and iterative search. Your network-construction script and a half-page note reporting the basic statistics, the shortest-path finding and its interpretation, and the computational capacity involved.
• Exercise 8.1.3, Non-linear prediction. A performance table (accuracy, precision, and recall for both models on held-out data) and a half-page interpretation of the comparison and of the computational capacity that explains it.

One purpose exercise. Choose one:

• Exercise 8.2.1, Centrality and social capital. A centrality table for your network’s top authors, the two visualizations, and a half-page semantics-gap memo. Builds on Exercise 8.1.1.
• Exercise 8.2.2, Measuring semantic dimensions. A projection table or plot showing where your key terms fall on the dimension, with a paragraph of interpretation and a note on what further evidence an explanatory claim would need. Builds on the optional Exercise 8.1.4.
• Exercise 8.2.3, Community detection and explanatory analysis. A one-page analysis note: the community count and characterization table, the association-test result, and your answers on correlation versus cause and the semantics gap. Builds on Exercise 8.1.1.
• Exercise 8.2.4, Model interpretation and epistemic opacity. The feature-importance summary, one partial dependence plot, the opacity memo, and a brief accuracy comparison for an interpretable alternative. Builds on Exercise 8.1.3.
• Exercise 8.2.5, Integrative prediction and cross-domain evaluation. The merged analysis table or the script that builds it, a table comparing within-period and cross-period performance, and a closing paragraph on integrative modeling. Builds on Exercises 8.1.3, 8.1.4, 8.2.1, and 8.2.3.

Also required:

• Exercise 8.3, Reflecting on AI-assisted validity. A disagreement analysis (which cases diverge and why) and a methods-section validity statement for the AI-assisted parts of your analysis.

Optional extensions: Exercise 8.1.2, Simulation and emergence; Exercise 8.1.4, Word embeddings.

Two routes fit the required work. Network route: Exercise 8.1.1, then 8.2.1 or 8.2.3. Prediction route: Exercise 8.1.3, then 8.2.4. Exercise 8.2.2 builds only on the optional Exercise 8.1.4, and Exercise 8.2.5 combines the outputs of four other exercises, so choose either one only if you plan to do what it depends on.

Exercises 8.1.3 and 8.2.3 ask for topic codes from Chapter 6. If you skipped the optional topic-modeling exercises, use the labels from your Milestone 3 coding.

How the points are earned: Method fits a stated purpose and capacity 3; implementation with a baseline or comparison 5; interpretation of the result and its limits, including opacity or the semantics gap 4; AI-assisted validity statement 3.

On the companion website: Chapter 8 materials · Supervised learning, gently · Worked notebooks and sample network data · Computational analysis tools

Milestone 5: Scientific communication

15 points · Chapters 9–11 · Due: Start of P539’s assigned final session · May 3–7. Make your pipeline inspectable, reproducible, and interpretable for a reader outside your field, and review its ethics.

Submit:

• Ethics review. About one page answering Chapter 11’s Reflective Exercises for your own pipeline: which uses of your corpus preserve the norms under which its records were produced; whose cases your measurement errors fell on; an AI-disclosure paragraph a reader could verify; and where accountability sits when an AI-coded variable is wrong.
• Exercise 10.1.2, Freeze your computational environment. A repository a peer can clone and run, containing the pinned environment specification, the seeded analysis script, and a README stating the Gundersen reproducibility type (R1 to R4) you reached and what the next one still needs.
• Exercise 10.2, Build a Rules/Memory/Knowledge package for your AI-assisted coding. The three-layer package (promptbook under kb/codebooks/, promptlog and run notes under memory/, logging and validation rules in your project rules file), the validation report, and a note on what your peer could not reproduce.
• Exercise 10.3.4, Build an integrative communication artifact. The artifact itself (a link, or the source with instructions to run it), with at least three coordinated views for a named non-specialist audience, and a short note separating the durable analysis from the disposable interface.
• Exercise 10.3.5, Visualize the uncertainty in one of your computational results. The uncertainty-aware figure and a caption stating plainly what the uncertainty represents (resampling, seed variation, or model error) and what it does not cover.

Optional extensions: Exercise 10.1.1, Adapted preregistration for an exploratory CSS analysis; Exercise 10.3.2, Apply the 3-step viz-making framework; Exercise 10.3.3, Apply Kirk’s design process to one figure.

The artifact can be a reproducible report, an interactive dashboard, or an annotated notebook. Choose the form that fits your audience. A dashboard is not required.

Exercise 10.1.2 refers to the analysis you preregistered in the optional Exercise 10.1.1. Freeze the analysis from Milestone 4. If you do not use Git, a zip archive of the project folder serves as the repository.

Exercise 10.3.4 builds on the optional Exercises 10.3.2 and 10.3.3. You can build the three views directly from your analysis table, or do those two exercises first for one figure.

How the points are earned: Frozen environment a peer can run 4; Rules/Memory/Knowledge package and validation report 3; integrative artifact for a named audience 4; uncertainty stated and shown 2; ethics review 2.

On the companion website: Chapter 10 materials and starter scaffolding · Communication tools

Milestone 6: Final presentation

10 points · Chapter 12 · Due: P539’s assigned final session · May 3–7. Present the pipeline and what it found, and answer questions about your choices.

Submit:

• Presentation. Ten minutes, followed by questions, in P539’s assigned final session. Cover your question, one decision at each of the four stages, one finding with its uncertainty, what a choice at one stage ruled in or out at the next, and a limit that remains.
• Slides or artifact link. Upload your slides, or the link you will present from, before the session starts.

How the points are earned: Question and audience 2; pipeline and evidence 4; methodological reasoning and limits 3; responses to questions 1.

What makes strong work

Earn credit through evidence in the files you submit and your explanation of it. A polished output without the checks behind it is incomplete, and partial credit reflects how far each criterion is met.

A careful analysis that produces an unexpected or inconclusive result can meet every criterion. The instructor evaluates the work.

Exercise reviews

Four ungraded reviews close the exercise chapters. In each, you demonstrate your work briefly, a classmate tries to follow or reproduce it, and the instructor ends with a worked example. Nothing is uploaded for a review.

The practice guide explains how a review runs and what to do with the feedback.

Practice notes and discussions

Chapters 3, 5, 7, and 9 each end with a Practice section that asks you to analyze published studies through one design stage. We use those sections for four student-led discussions, on January 25, February 15, March 8, and April 5. Before each one, write a half-page Practice note on one study. Discussion leaders prepare a 10-minute opening in place of the note. The notes are not graded or uploaded. See the practice guide.

Individual reflections

20 points. Three reflections ask what you learned from doing the work. Their prompts draw on the Reflective Exercises that close the book’s chapters. Use a specific decision, artifact, or review as evidence.

Reflection 1: data and setup

5 points · About 500 words · Due: Week 6 · February 17. What did inspecting your data source change about how you read the data section of a published paper? Where did structure serve your research, and where did it become engineering for its own sake? Which of rules, memory, and knowledge did you neglect at first, and what did that cost?

Drawn from the Reflective Exercises in Chapters 2 and 4.

Reflection 2: measurement

5 points · About 500 words · Due: Week 9 · March 10. Where did your dictionary and prompt-based labels diverge, and what did the divergence teach you about your concept? Did your pipeline find the minority cases that matter most to your question? What is the largest remaining threat to the validity of your measure?

Drawn from the Reflective Exercises in Chapter 6.

Final reflection: the whole pipeline

10 points · About 1,000 words · Due: Start of P539’s assigned final session · May 3–7. Trace one decision from an early stage to something it constrained later. Where did showing uncertainty change the story you could tell? If an AI assistant wrote or debugged any of your code, how did you verify that it did what you intended? How has the project changed your judgment and your sense of your responsibilities, and what do you want to learn next?

Drawn from the Reflective Exercises in Chapters 8 and 10, and Chapter 12.

How reflections are evaluated

For each 5-point reflection, specific evidence earns 2 points, analysis of the choice and its consequences 2, and a justified next step 1. For the 10-point final reflection, those shares are 3, 4, and 3. A log of completed tasks does not meet the requirement. You can draw on Practice notes, a changed output, a rejected AI suggestion, or a reviewer’s comment, but explain what the evidence taught you.

Reflection time is included in that week’s writing allocation. Submit each reflection as its own PDF or Markdown file. The reflection guide explains how to write one and which project files to draw on.

What accompanies every submission

Every milestone and reflection ends with a short AI-use disclosure. If you did not use AI, say so. If a model coded, extracted, or drafted anything in the submission, the current promptbook and the matching entries in memory/PROMPTLOG.md must be in your project folder.

Keep credentials and any restricted material out of what you submit. The final submission checklist lists what Milestone 5 should contain.