Camera ready (#48)

* Add reproducibility package infrastructure

- Makefile with targets for running table/figure generation
- README with usage instructions and data download link
- requirements.txt with dependencies
- download_data.py script for fetching pre-generated graphs

* Add PGD benchmark and MMD table generation scripts

- generate_benchmark_tables.py: Standard PGD metrics with VUN
- generate_mmd_tables.py: Gaussian TV and RBF MMD metrics

Both scripts use the polygraph-benchmark API:
- StandardPGDInterval for PGD computation
- GaussianTVMMD2BenchmarkInterval/RBFMMD2BenchmarkInterval for MMD
- Proper graph format conversion for DIGRESS tensor outputs

* Add GKLR table generation script

Computes PGD metrics using Logistic Regression classifier instead of
TabPFN, with standard descriptors (orbit counts, degree, spectral,
clustering, GIN).

Uses PolyGraphDiscrepancyInterval with sklearn LogisticRegression
for classifier-based evaluation.

* Add concatenation ablation table generation script

Compares standard PGD (max over individual descriptors) vs concatenated
PGD (all descriptors combined into single feature vector).

Features:
- ConcatenatedDescriptor class with PCA dimensionality reduction
- Handles TabPFN 500-feature limit via PCA to 100 components
- Uses LogisticRegression for concatenated features
- Optimized subset mode for faster testing

* Add figure generation scripts for reproducibility

- generate_model_quality_figures.py: Training/denoising curves
- generate_perturbation_figures.py: Metric sensitivity to edge perturbations
- generate_phase_plot.py: PGD vs VUN training dynamics
- generate_subsampling_figures.py: Bias-variance tradeoff analysis

All scripts use StandardPGDInterval from polygraph-benchmark API.
Phase plot gracefully handles missing VUN values (requires graph_tool).

* Update pixi config to new workspace schema and fix package name

Rename [project] to [workspace] per updated pixi schema and correct
the pypi-dependencies package name from polygraph to polygraph-benchmark.

* Relocate data directory to data/polygraph_graphs/

Move the expected data path from polygraph_graphs/ to data/polygraph_graphs/
to keep generated data under the gitignored data/ directory.

* Remove standalone reproducibility README and requirements.txt

Documentation is consolidated into the main README and dependencies
are managed through pyproject.toml extras.

* Add SLURM cluster submission infrastructure

Add submitit-based cluster module for distributing reproducibility
workloads across SLURM nodes. Includes YAML-configurable job parameters,
job metadata tracking, and result collection helpers.

- cluster.py: shared wrapper with SlurmConfig, submit_jobs, collect_results
- configs/: default CPU and GPU SLURM configurations
- pyproject.toml: new [cluster] optional dependency group (submitit, pyyaml)

* Add SLURM cluster support to table generation scripts

Add --slurm-config, --local, and --collect CLI options to all four table
generation scripts for distributing computation across SLURM nodes.
Each script gains a standalone task function suitable for submitit,
result reshaping helpers, and three execution modes (local, submit, collect).

Also updates DATA_DIR paths and adds tables-submit/tables-collect Make targets.

* Add reproducibility section to README

Document the full reproducibility workflow including data download,
script overview, Make targets, hardware requirements, SLURM cluster
submission, and troubleshooting tips.

* Add pre-generated tables and figures for reference

Include LaTeX tables and PDF figures produced by the reproducibility
scripts so reviewers can verify outputs without re-running computation.

* Restructure reproducibility pipeline and regenerate all results

- Replace monolithic generate_*.py scripts with modular 01-08 experiment
  directories, each with compute.py, plot.py, and/or format.py
- Add Hydra configs for all experiments with SLURM launcher support
- Fix sparse feature OOM in GKLR (Bug 12), package name in graph_storage,
  TabPFN CPU limit workaround, and stale cache issues
- Add kernel logistic regression module and async results I/O utility
- Regenerate all tables with correct PGD values, subscores, and GKLR
  graph kernel metrics (PM/SP/WL)
- Regenerate all figures including new subsampling, perturbation,
  model quality, and phase plot visualizations
- Include all JSON result files for full reproducibility

* Fix KernelLogisticRegression float64 precision in kernel normalization

Ensure consistent float64 dtype in kernel diagonal computation and
normalization to prevent precision issues with sparse matrix outputs.

* Fix equal-size graph splits for ego and proteins perturbation

Ego dataset has 757 graphs (odd number), causing unequal reference/
perturbed splits which fails the equal-count requirement. Use
half = len // 2 and slice [half : 2*half] to guarantee equal sizes.
Same fix applied to proteins split for consistency.

* Handle TabPFN NaN errors for near-constant features in PGD

TabPFN v2.0.9 raises ValueError when its encoder produces NaN from
near-constant features after StandardScaler normalization (see
github.com/PriorLabs/TabPFN/issues/108). This caused lobster/GRAN
n=32 PGD subsampling to crash completely.

Wrap classifier fit/predict in try/except in both the CV fold loop
and the refit section. On failure, treat as indistinguishable
distributions (score=0), matching the existing constant-feature
fallback semantics.

* Fix concatenation experiment PCA pipeline and subsample sizes

Match original polygraph CombinedDescriptor behavior: per-descriptor
StandardScaler + PCA, both fit on reference data only. Fix subsample
size calculation to use 50% of min subset capped at 2048, matching
the original experiment configuration.

* Fix GKLR experiment reference graph count and subsample sizes

Increase reference graph count from 512 to 4096 to match original
experiment. Fix subsample size to 50% of min subset capped at 2048,
consistent with the 2x requirement of PolyGraphDiscrepancyInterval.

* Add H100 GPU SLURM launcher configuration

Add submitit launcher config for p.hpcl94g partition with H100 GPUs,
enabling faster TabPFN computation for PGD experiments.

* Regenerate all reproducibility results with bug fixes

Recompute all experiments after fixing:
- KernelLogisticRegression float64 precision
- Ego/proteins unequal graph splits
- TabPFN NaN handling for near-constant features
- Concatenation PCA pipeline
- GKLR reference graph count and subsample sizes

PGD subsampling: 117/120 results (3 ESGG n=4096 infeasible due to
dataset size). All values within bootstrap variance of paper.
Perturbation: 25/25 results including ego dataset.
Benchmark, concatenation, GKLR tables: all 16/16 regenerated.

* Update PGD computation, reproducibility scripts, and dependencies

Includes TabPFN v6 classifier updates, plotting and formatting
improvements across all reproducibility experiments, and added
backoff/tabpfn dependencies.

* Update all figures and tables to TabPFN weights v2.5

Regenerate all reproducibility tables and figures using TabPFN weights
v2.5 for camera-ready preparation. Add --results-suffix support to
03_model_quality/format.py. Include comparison and merge utility scripts.

* Rename PGS to PGD in model quality table headers

* Add pymupdf and pillow dependencies

Needed for PDF-to-image conversion in diff report generation.

* Add parallel VUN computation with isomorphism timeout

Refactor the VUN metric to support multiprocessing for novelty and
validity checks, and add a per-pair SIGALRM timeout on isomorphism to
prevent hangs on pathological graph pairs. Extract shared VUN helpers
into reproducibility/utils/vun.py for reuse across experiments.

* Refactor TabPFN classifier creation to use explicit version map

Replace ad-hoc if/else branching on weights version with a version_map
dict that raises on unknown versions instead of silently falling back.
Applied consistently across all five compute scripts.

* Add VUN computation scripts for model quality and benchmark experiments

Add dedicated scripts to compute VUN (Valid-Unique-Novel) metrics for
denoising-iteration checkpoints and benchmark results. These patch
existing result JSONs with VUN values using parallel isomorphism checking.

* Add SLURM launcher configs and train-test reference experiment

Add CPU-only (hpcl94c) and GPU (hpcl93) SLURM launcher configs for
Hydra multirun. Add experiment 09 that computes train-vs-test reference
PGD values to establish metric baselines per dataset.

* Improve table formatting with per-row ranking and VUN column

Add bold/underline formatting for best/second-best values per row in
correlation and benchmark tables. Scale correlation values by 100 for
readability. Add VUN column support in denoising PGS table. Add subscore
ranking in benchmark table. Rename orbit_pgs to orbit4_pgs.

* Add single-dataset perturbation plotting command

Add a new CLI subcommand for generating perturbation metric-vs-noise
figures for a single dataset (e.g. SBM-only plots), supporting both
single-perturbation and all-perturbation layouts.

* Regenerate all figures and tables for camera-ready

Updated with TabPFN weights v2.5, improved table formatting (bold/
underline ranking, values scaled by 100), new SBM perturbation plots,
and additional versioned table snapshots for comparison.

* Add reproducibility debug utilities and analysis scripts

Add helper scripts used during the camera-ready recomputation:
PGD diff checking, environment validation, pickle inspection,
HTML diff report generation, SLURM recompute wrappers, and
rerun notes documenting the process.

* Move pymupdf and pillow to dev optional dependencies

These are only used by the diff report generator script, not the core
library. Move them from top-level pixi.toml dependencies into the dev
extras in pyproject.toml so they're pulled in via the existing
extras = ["dev", "cluster"] configuration.

* Remove pre-generated figures and tables from tracking

These are generated artifacts that should be reproduced from the
scripts, not tracked in version control.

* remove vscode settings

* Fix ruff lint, ruff format, and pyright type check errors

- Fix 37 ruff lint errors (unused imports, f-strings, ambiguous variable
  names, unused assignments)
- Auto-format all files with ruff
- Fix 31 pyright type errors: numpy.bool return types, scipy sparse
  shape stubs, Literal type annotations, None-safety assertions, and
  conditional weight kwargs

* Add kernel_diag abstraction and refactor KernelLogisticRegression

Extract _resolve_kernel method to eliminate duplicated kernel selection
logic in _compute_kernel_matrix and _compute_kernel_diag. Add abstract
kernel_diag method to DescriptorKernel with concrete implementations
in all subclasses, replacing inline isinstance checks.

* Relax pydantic constraint, remove grakel dev dep, register slow marker

Widen pydantic to >=2.0,<3.0. Remove grakel from dev dependencies
since tests now use frozen reference values. Register the slow pytest
marker to suppress warnings.

* Clean up test fixtures and add requires_import helper

Remove hard imports of rdkit and graph_tool that prevented the test
suite from loading without optional dependencies. Remove autouse=True
from fixtures that are only needed by specific tests. Add
requires_import() skip decorator and --skip-slow marker filtering.

* Replace DGL runtime dependency with frozen reference values

DGL is incompatible with PyTorch>=2.4. Replace runtime DGL comparisons
with reference values precomputed under DGL 2.3.0 / PyTorch 2.3.1,
with regeneration instructions in comments.

* Replace grakel runtime dependency with frozen reference values

grakel is incompatible with numpy>=2. Replace runtime grakel
comparisons with reference gram matrices precomputed under
grakel 0.1.10, with regeneration instructions in comments.

* Use @pytest.mark.slow consistently for slow tests

Replace skipif("config.getoption('--skip-slow')") with the
@pytest.mark.slow decorator for consistency across the test suite.

* Remove one-off debug and comparison scripts from reproducibility/

Remove 15 files that were used during development and paper review
but are not part of the reproducibility pipeline. All unique
configuration they contained is already captured in the Hydra configs
and compute scripts.

Removed: debug utilities (_check_pgd_diffs, check_env, check_pkl),
HTML comparison generators (compare_figures, compare_pgd_v2_vs_v25,
compare_tables, generate_diff_report), one-off recomputation scripts
(recompute_training_pgd, slurm_recompute_*), merge_v2_results,
rerun_notes.md, and the generated rebuttal_vs_camera_ready_diff.html.

* Restore grakel_wl_mmd function with lazy import

grakel is incompatible with numpy>=2 but the reference code should
remain accessible. Restore the function with a lazy import inside the
body and re-reference it in the skipped test_measure_runtime test.

* Speed up test suite and fix parallel test issues

- Mark slow tests (snippets, demo, TabPFN, bootstrap, graph_tool,
  standard PGD) so --skip-slow skips them by default
- Add xdist_group markers to prevent dataset cache races and
  graph_tool concurrency issues under parallel execution
- Add test-all pixi task for running the full suite including slow tests
- Fix MockDescriptorKernel missing kernel_diag abstract method
- Reduce molecule SMILES lists to 10 (sufficient for smoke tests)
- Switch test output from -sv to -v --tb=short for cleaner parallel output
- Use --dist loadgroup to respect xdist_group markers

* Fix ruff and pyright CI failures

- Remove unused unattr_ref variable in test_gin_metrics.py
- Suppress pyright reportOptionalSubscript for csr_array.shape[0]

* Fix ruff format: remove extra blank line in test_mmd.py

* Add pyright to pre-commit hooks and fix whitespace issues

Add pyright as a dev dependency and local pre-commit hook so the
pre-commit workflow mirrors CI (ruff check, ruff format, pyright).
Fix trailing whitespace and missing EOF newlines caught by hooks.

* Remove redundant reproducibility pixi tasks

The reproducibility workflow is fully covered by the Makefile in
reproducibility/. Keep pixi tasks for dev workflow only (test, docs).

* Expand pyright to cover tests/ and reproducibility/

- Add __iter__ to NetworkXView so list() works on dataset views
- Cast np.quantile/mean/std to float in MetricInterval.from_samples
- Replace BinomConfidenceInterval namedtuple with typed class
- Fix Literal type mismatches for split/variant params at call sites
- Add type narrowing assertions in tests for Optional attributes
- Fix matplotlib private imports (use ticker/colors modules directly)
- Exclude third-party test implementations (ggm/gran) from pyright
- Fix pre-commit pyright hook to use pixi run

* Install dev dependencies in pyright CI workflow

Pyright now checks tests/ and reproducibility/ which import pytest
and other dev dependencies.

* Remove --paper-dir CLI option from all reproducibility scripts

This option copied generated outputs to an external paper directory,
which is no longer needed.

* Remove --results-suffix and --mmd-only/--pgd-only CLI options

These options are no longer needed. Output files use fixed paths
directly instead of being parameterized through a suffix.

* Remove results_suffix from compute scripts and Hydra configs

The suffix was always empty. Compute scripts now use fixed result
directory names that match the hardcoded paths in plot/format scripts.
The 09_train_test_reference script now embeds the tabpfn weights
version directly in the directory name.

* Make default classifier explicit via default_classifier() factory

The classifier parameter previously defaulted to None with the actual
TabPFN instantiation hidden deep inside _descriptions_to_classifier_metric.
Now the None sentinel is resolved immediately at the top of the function,
and all docstrings document that the default is TabPFN via
default_classifier().

* Resolve classifier=None at init, not deep in the call chain

Each class now resolves None to default_classifier() in its __init__,
so _classifier is always a concrete ClassifierProtocol. The internal
_descriptions_to_classifier_metric now requires a classifier (keyword-
only) and never sees None.

* Add docstring to _json_default serializer helper

* Simplify io.py: drop maybe_append_reproducibility_jsonl alias

Kept only maybe_append_jsonl as the single function. Added docstrings
to all public functions. Updated all 13 import sites.

* Remove dead compute scripts

- 01_subsampling/compute.py: duplicated compute_pgd.py, unused by
  Makefile and submit scripts
- 09_train_test_reference/: experiment never integrated into the
  reproduction pipeline (no Makefile target, no plot/format scripts,
  no results)

* Remove dead 01_subsampling/compute.py, restore 09_train_test_reference

compute.py in 01_subsampling duplicated compute_pgd.py and was unused.
09_train_test_reference is kept and will be integrated into the
Makefile.

* Integrate 09_train_test_reference into Makefile

Added to compute-tables, submit-tables, and as standalone target 09.

* Remove cluster-specific SLURM configs and hardcoded paths

- Removed 6 cluster-specific launcher variants (slurm_cpu_hpcl94c,
  slurm_cpu_large, slurm_cpu_small, slurm_gpu_fallback, slurm_gpu_h100,
  slurm_gpu_hpcl93). Only generic slurm_cpu and slurm_gpu remain with
  placeholder partitions.
- Replaced hardcoded absolute paths in submit scripts with
  git rev-parse --show-toplevel.
- Replaced cluster-specific partition names with TODO placeholders.
- Cleaned up docstring references to removed launchers.

* Remove review-camera-ready.md

* Fix critical, high, and medium review issues in core library

kernel_lr.py:
- C1: Use K.shape[0] for alpha_init instead of len(X)
- C4: Use np.logaddexp(0, -yf) for numerical stability
- H1: Merge objective/gradient to eliminate redundant K @ alpha
- H2: Avoid double featurization when X2 is None
- L1: Remove unused random_state and project_dim parameters

vun.py:
- C2: Replace signal.SIGALRM timeout with ThreadPoolExecutor
  (works in multiprocessing workers and on Windows)
- H3: Add edges="links" to nx.node_link_data/graph calls
- M7: Remove section separator comments

generic_descriptors.py:
- M2: Replace .get() defensive defaults with direct access
- M3: Remove bare except Exception in PyramidMatchDescriptor

io.py:
- M1: Replace hasattr with isinstance(obj, np.generic)

* Fix remaining review items: performance, style, cleanup

polygraphdiscrepancy.py:
- P3: Vectorize _is_constant sparse check (col min/max vs row loop)

Reproducibility scripts:
- L10: Remove duplicate runtime fields (keep *_perf_seconds only)
- L11: Remove pointless _fmt_pgs/_best_two aliases in format scripts
- M7: Remove section separator comments across all scripts

* Deduplicate TabPFN factory and VUN logic, add sparse eigenvalue path

- H6: 05_benchmark/compute_vun.py now imports compute_vun_parallel
  from utils.vun instead of duplicating ~120 lines of VUN logic
- L5: Extracted make_tabpfn_classifier to utils/data.py, replaced
  6 local copies across compute scripts
- P4: EigenvalueHistogram uses scipy.sparse.linalg.eigsh for graphs
  with >500 nodes, avoiding dense conversion of large Laplacians

* Deduplicate utilities and add kernel size guard

- L4: load_graphs/get_reference_dataset in 01-03 now delegate to
  utils/data.py instead of local copies
- L6: load_results extracted to utils/formatting.py, removed from
  4 format scripts
- P1: Warn when kernel matrix exceeds 10k samples in kernel_lr.py

* Add download hash verification and restore test thresholds

- H4: download_data.py now verifies SHA-256 hash after download,
  before extraction. Uses placeholder hash with TODO for now.
- M8: Restore test thresholds from 0.5 to 0.7 in
  test_polygraphdiscrepancy.py. The test distributions (ER 0.8 vs
  ER 0.1) are clearly distinct; 0.5 was essentially random chance.

* Clean up dead code, deduplicate constants, and fix style issues

Remove unused utilities (to_list, mol2smiles, BOND_STEREO_TYPES,
MetricInterval.__getitem__), extract shared constants
(_DEFAULT_RBF_BANDWIDTHS, _molecule_descriptors, _standard_descriptors),
fix f-string bug in polygraphdiscrepancy, use NamedTuple classes over
namedtuple calls, modernize super() calls, replace assert False with
proper exceptions, use sqeuclidean metric directly, and move tqdm to
core dependencies.

* Split TabPFN tests into slow variants with larger sample sizes

Separate logistic and TabPFN classifier tests so TabPFN variants use
256 samples (up from 128) for stability, and mark them @pytest.mark.slow
instead of using request.applymarker at runtime.

* Fix MoleculePGDInterval test: reduce subsample_size to fit test data

The test only has 10 molecules but subsample_size=8 requires at least
16 reference molecules (2 * subsample_size). Reduce to 4.

* Fix ruff formatting

Author: pjhartout

.github/workflows/pyright.yaml

@@ -21,7 +21,7 @@ jobs:
       - name: Upgrade pip and install project (editable)
         run: |
           python -m pip install --upgrade pip
-          pip install -e .
+          pip install -e ".[dev]"
 
       - name: Install Pyright
         run: |

.gitignore

@@ -9,6 +9,14 @@ experiments/*/figures/*.png
 experiments/*/results/*.csv
 experiments/*/tables/*.tex
 experiments/model_benchmark/benchmark_results.tex
+multirun/
+
+# Reproducibility generated outputs
+reproducibility/figures/**/*.pdf
+reproducibility/figures/**/results/
+reproducibility/figures/**/results_*/
+reproducibility/tables/results/
+reproducibility/figure_comparison/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -793,4 +801,4 @@ TSWLatexianTemp*
 *.vtc
 
 # glossaries
-*.glstex
+*.glstex

.pre-commit-config.yaml

@@ -25,3 +25,12 @@ repos:
     - id: ruff-format
       types_or: [ python, pyi ]
       args: [--config=pyproject.toml ]
+
+- repo: local
+  hooks:
+    - id: pyright
+      name: Pyright Type Check
+      entry: pixi run pyright --project pyproject.toml
+      language: system
+      types: [python]
+      pass_filenames: false

.readthedocs.yaml

@@ -8,7 +8,7 @@ build:
   jobs:
     build:
       html:
-        - python -m mkdocs build --clean --site-dir $READTHEDOCS_OUTPUT/html --config-file mkdocs.yml 
+        - python -m mkdocs build --clean --site-dir $READTHEDOCS_OUTPUT/html --config-file mkdocs.yml
         - python -m mkdocs build --clean --site-dir $READTHEDOCS_OUTPUT/html --config-file mkdocs.yml # Execute twice to make links to /images work
         - ls -la $READTHEDOCS_OUTPUT/html
 

.vscode/settings.json

@@ -25,4 +25,4 @@ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
 OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

LICENSE

@@ -243,19 +243,185 @@ The following results mirror the tables from [our paper](https://arxiv.org/abs/2
 
 <sub>* AutoGraph* denotes a variant that leverages additional training heuristics as described in the [paper](https://arxiv.org/abs/2510.06122).</sub>
 
+## Reproducibility
+
+The [`reproducibility/`](reproducibility/) directory contains scripts to reproduce all tables and figures from the paper.
+
+### Quick Start
+
+```bash
+# 1. Install dependencies
+pixi install
+
+# 2. Download the graph data (~3GB)
+cd reproducibility
+python download_data.py
+
+# 3. Generate all tables and figures
+make all
+```
+
+### Data Download
+
+The generated graph data (~3GB) is hosted on [Proton Drive](https://drive.proton.me/urls/VM4NWYBQD0#3sqmZtmSgWTB). After downloading, extract to `data/polygraph_graphs/` in the repository root.
+
+```bash
+# Full dataset (required for complete reproducibility)
+python download_data.py
+
+# Small subset for testing/CI (~50 graphs per model)
+python download_data.py --subset
+```
+
+Expected data structure after extraction:
+
+```
+data/polygraph_graphs/
+├── AUTOGRAPH/
+│   ├── planar.pkl
+│   ├── lobster.pkl
+│   ├── sbm.pkl
+│   └── proteins.pkl
+├── DIGRESS/
+│   ├── planar.pkl
+│   ├── lobster.pkl
+│   ├── sbm.pkl
+│   ├── proteins.pkl
+│   ├── denoising-iterations/
+│   │   └── {15,30,45,60,75,90}_steps.pkl
+│   └── training-iterations/
+│       └── {119,209,...,3479}_steps.pkl
+├── ESGG/
+│   └── *.pkl
+├── GRAN/
+│   └── *.pkl
+└── molecule_eval/
+    └── *.smiles
+```
+
+### Scripts Overview
+
+#### Table Generation
+
+| Script | Output | Description |
+|--------|--------|-------------|
+| `generate_benchmark_tables.py` | `tables/benchmark_results.tex` | Main PGD benchmark (Table 1) comparing AUTOGRAPH, DiGress, GRAN, ESGG |
+| `generate_mmd_tables.py` | `tables/mmd_gtv.tex`, `tables/mmd_rbf_biased.tex` | MMD² metrics with GTV and RBF kernels |
+| `generate_gklr_tables.py` | `tables/gklr.tex` | PGD with Kernel Logistic Regression using WL and SP kernels |
+| `generate_concatenation_tables.py` | `tables/concatenation.tex` | Ablation comparing individual vs concatenated descriptors |
+
+#### Figure Generation
+
+| Script | Output | Description |
+|--------|--------|-------------|
+| `generate_subsampling_figures.py` | `figures/subsampling/` | Bias-variance tradeoff as function of sample size |
+| `generate_perturbation_figures.py` | `figures/perturbation/` | Metric sensitivity to edge perturbations |
+| `generate_model_quality_figures.py` | `figures/model_quality/` | PGD vs training/denoising steps for DiGress |
+| `generate_phase_plot.py` | `figures/phase_plot/` | Training dynamics showing PGD vs VUN |
+
+Each script can be run independently with `--subset` for quick testing:
+
+```bash
+# Tables (full computation)
+python generate_benchmark_tables.py
+python generate_mmd_tables.py
+python generate_gklr_tables.py
+python generate_concatenation_tables.py
+
+# Tables (quick testing with --subset)
+python generate_benchmark_tables.py --subset
+python generate_mmd_tables.py --subset
+
+# Figures (full computation)
+python generate_subsampling_figures.py
+python generate_perturbation_figures.py
+python generate_model_quality_figures.py
+python generate_phase_plot.py
+
+# Figures (quick testing)
+python generate_subsampling_figures.py --subset
+python generate_perturbation_figures.py --subset
+```
+
+### Make Targets
+
+```bash
+make download        # Download full dataset (manual step required)
+make download-subset # Create small subset for CI testing
+make tables          # Generate all LaTeX tables
+make figures         # Generate all figures
+make all             # Generate everything
+make tables-submit   # Submit table jobs to SLURM cluster
+make tables-collect  # Collect results from completed SLURM jobs
+make clean           # Remove generated outputs
+make help            # Show available targets
+```
+
+### Hardware Requirements
+
+- **Memory:** 16GB RAM recommended for full dataset
+- **Storage:** ~4GB for data + outputs
+- **Time:** Full generation takes ~2-4 hours on a modern CPU
+
+The `--subset` flag uses ~50 graphs per model, runs in minutes, and verifies code correctness (results are not publication-quality).
+
+### Cluster Submission
+
+Table generation scripts support SLURM cluster submission via [submitit](https://github.com/facebookincubator/submitit). Install the cluster extras first:
+
+```bash
+pip install -e ".[cluster]"
+```
+
+SLURM parameters are configured in YAML files (see `reproducibility/configs/slurm_default.yaml`):
+
+```yaml
+slurm:
+  partition: "cpu"
+  timeout_min: 360
+  cpus_per_task: 8
+  mem_gb: 32
+```
+
+Submit jobs, then collect results after completion:
+
+```bash
+cd reproducibility
+
+# Submit all table jobs to SLURM
+python generate_benchmark_tables.py --slurm-config configs/slurm_default.yaml
+
+# After jobs complete, collect results and generate tables
+python generate_benchmark_tables.py --collect
+
+# Or use Make targets
+make tables-submit                                        # submit all
+make tables-submit SLURM_CONFIG=configs/my_cluster.yaml   # custom config
+make tables-collect                                       # collect all
+```
+
+Use `--local` with `--slurm-config` to test the submission pipeline in-process without SLURM.
+
+### Troubleshooting
+
+**Memory issues:** Use `--subset` flag for testing, process one dataset at a time, or increase system swap space.
+
+**Missing data:** Verify `data/polygraph_graphs/` exists in repo root, run `python download_data.py` to check data status, or download manually from Proton Drive.
+
+**TabPFN issues:** TabPFN is pinned to v2.0.0 for reproducibility: `pip install tabpfn==2.0.0`.
 
 ## Citing
 
 To cite our paper:
 
 ```latex
 @misc{krimmel2025polygraph,
-  title={PolyGraph Discrepancy: a classifier-based metric for graph generation}, 
+  title={PolyGraph Discrepancy: a classifier-based metric for graph generation},
   author={Markus Krimmel and Philip Hartout and Karsten Borgwardt and Dexiong Chen},
   year={2025},
   eprint={2510.06122},
   archivePrefix={arXiv},
   primaryClass={cs.LG},
-  url={https://arxiv.org/abs/2510.06122}, 
+  url={https://arxiv.org/abs/2510.06122},
 }
 ```

README.md

@@ -7,4 +7,3 @@ dependencies:
   - graph-tool
   - pip:
       - -e .[dev]
-

environment.yml

@@ -538,5 +538,3 @@
   \node[anchor=base west, text=edgeBColor, inner sep=0pt, font=\LogoTextFont] at ($(benchbbox.east |- 0,\TextBaselineY) + (\BenchTextGapLen,0)$) {enchmark};
 \end{tikzpicture}
 \end{document}
-
-

logo/logo.tex

@@ -571,5 +571,3 @@
   \node[anchor=base west, text=edgeBColor, inner sep=0pt, font=\LogoTextFont] at ($(benchbbox.east |- 0,\TextBaselineY) + (\BenchTextGapLen,0)$) {enchmark};
 \end{tikzpicture}
 \end{document}
-
-