← Back to all skills

Code Review

Review a development economics project folder against journal replication standards.

Usage

/code-review <path-to-project-folder>

When reviewing a project, first explore the folder structure (list directories, identify scripts, data files, README, and master script). Then evaluate the project against all seven sections below. Adapt checks to whichever languages are present (Stata, R, Python, or a mix).

---

1. Folder Structure & Portability (Gentzkow & Shapiro, Reif, DIME)

• Separate directories by function: /data/, /code/ (or /scripts/), /output/ (or /results/)
• Within /data/, raw data separated from processed/analysis data (e.g. /data/raw/, /data/processed/)
• Within /output/, tables and figures in identifiable locations
• Master script and README in root directory
• Scripts named with execution-order prefixes (01_clean.do, 02_merge.do, 03_analysis.do)
• No spaces in filenames; lowercase with dashes or underscores preferred
• No hardcoded absolute paths anywhere except a single root path definition
• All paths built from that root using globals/macros/variables
• Forward slashes used in file paths (even on Windows)
• All file paths enclosed in double quotes (Stata)
• The package should be runnable by changing only one line (the root directory)

---

2. Master Script (AEA, Reif, DIME)

• A single master script (run.do, main.R, Makefile, etc.) exists in the root directory
• Running it executes the entire pipeline from raw data to final outputs without manual intervention
• Master script sets software version (version 15 in Stata)
• Master script sets random number seed (set seed #)
• Master script installs all required user-written packages (or they are bundled locally)
• Master script defines all global macros / path variables
• Master script disables interactive prompts (set more off, set varabbrev off in Stata)
• Deleting all intermediate and output files and re-running the master script reproduces everything
• Runtime estimate is documented (either in master script comments or README)

---

3. Coding Standards

Stata (DIME Analytics, Reif)

• No abbreviated variable names; all written out completely
• No wildcards (*, ?, -) to reference variable lists in analysis code
• missing(var) used instead of var >= . for missing value checks
• isid used to verify uniqueness before merges and sorts
• stable sort option used, or sorts on unique keys for deterministic ordering
• Floating-point variables used as sort keys cast as double
• Descriptive loop index names (foreach crop not foreach i)
• Loops closed with identifying comments: } // End crop loop
• Lines broken at ~80 characters using ///
• Consistent indentation (4 spaces per nesting level)
• assert statements validate key data properties and results
• No cd commands; absolute paths built from globals
• Tables generated by code (esttab, texsave, regsave), never copy-pasted from output
• qui prefix suppresses unnecessary output in scripts

R

• Reproducible package management (renv lockfile or equivalent)
• No setwd(); paths relative to project root (e.g. via here package)
• Functions used for repeated operations rather than copy-paste
• Random seeds set with set.seed() before stochastic operations
• Package versions documented

Python

• Virtual environment with requirements.txt or pyproject.toml
• No hardcoded paths; pathlib or os.path.join used
• Functions and modules used for repeated operations
• Random seeds set (numpy.random.seed, random.seed) before stochastic operations
• Package versions pinned

All languages

• Every script has a header comment: purpose, author, date, inputs, outputs
• Comments explain why, not what (reasoning and analytical choices, not mechanics)
• Section-level comments delineate logical blocks
• Self-documenting code via descriptive variable and function names

---

4. Data Management (Gentzkow & Shapiro, TIER, J-PAL)

• Raw data files are read-only and never modified by code
• All transformations from raw to analysis data are performed by code, with zero manual steps
• Every derived dataset can be regenerated by running the pipeline from raw data
• Unit of observation / key documented for every dataset
• Merge keys verified with isid or equivalent before merge operations
• Data kept normalised (one observation level per table) as far into the pipeline as possible
• If survey/individual data: PII checks run, de-identification documented
• Confidential data handled separately with access instructions documented
• Missing values handled explicitly and consistently

---

5. Output Reproducibility (Reif, Vilhuber)

• All tables generated by code and saved to files (LaTeX, CSV, or Excel), never manually formatted
• All figures generated by code and saved to files (PDF, PNG, EPS)
• In-text numbers either generated by code or verified with assert statements
• Random seeds set before every stochastic operation (bootstrap, simulation, random sampling, permutation tests)
• Stata version locked (version X) to prevent cross-version differences
• Sort order deterministic throughout (unique sort keys, stable option)
• User-written packages bundled locally or installed by master script with version pinning
• A clean-run test has been performed: copy package to new location, delete all intermediate/output files, run master script, verify all outputs match manuscript

---

6. README & Documentation (AEA Template README, Social Science Data Editors)

• README exists in root directory (PDF or Markdown)
• Data Availability Statement: For each dataset — provenance, access conditions, licensing, whether included in package
• Rights statement: Authors confirm legitimate access and redistribution rights
• Dataset list: Every data file enumerated with filename, source, and format
• Computational requirements: Software with version numbers, add-on packages with versions, OS, hardware requirements, wall-clock runtime estimates
• Code description: Every script listed with its purpose, inputs, and outputs
• Instructions to replicator: Step-by-step procedure, specifying which line to modify (root path)
• Output-to-script mapping: Table mapping each figure, table, and in-text number to the script that produces it
• Data citations: Formal bibliographic citations for all datasets, including DOIs where available
• Data also cited in the manuscript references section (not just footnotes)

---

7. Common Failure Modes (Vilhuber, AEA Data Editor — from 2,400+ package reviews)

Flag any of these top causes of replication failure:

1. Missing packages: User-written commands not installed or bundled; code fails out-of-the-box
2. Broken file paths: Hardcoded absolute paths that only work on the author’s machine
3. Missing data files: Data referenced in code but not included (and no access instructions)
4. Inadequate README: Missing runtime estimates, software versions, or replication instructions
5. Undocumented manual steps: Results depend on steps performed outside of code (manual Excel formatting, hand-editing tables)
6. Nondeterministic results: Missing random seeds, version-dependent sort orders, floating-point instability
7. Code errors: Logic errors, broken references, scripts that error out (~25% of submitted packages)
8. Version incompatibilities: Code written for one software version fails on another
9. Incomplete code: Data cleaning or intermediate processing scripts omitted
10. Missing data citations: Datasets used but never formally cited

---

Output Format

After reviewing all seven sections, provide findings in this format:

Summary: One paragraph overall assessment of replication readiness.

Major Issues (would cause replication failure or journal rejection):

• List each issue with the specific file/location and a concrete fix

Minor Issues (should be fixed but won’t block replication):

• List each issue with the specific file/location and a concrete fix

Strengths (things done well, worth preserving):

• Note good practices to reinforce

Recommended Priority Order: Number the major issues in the order they should be fixed (most impactful first).

Sources

This skill synthesises standards from:

• Gentzkow & Shapiro, “Code and Data for the Social Sciences” (2014)
• DIME Analytics, World Bank Development Research Group coding standards
• Julian Reif, “Stata Coding Guide”
• AEA Data and Code Availability Policy & Lars Vilhuber’s AEA Data Editor guidance
• Social Science Data Editors Template README
• Project TIER Protocol 4.0
• J-PAL coding resources for randomised evaluations
• Christensen, Freese, & Miguel, “Transparent and Reproducible Social Science Research”