Benchmark Reproduction

This page explains how to regenerate the public benchmark artifacts in this repository. It is meant for software reproducibility: manuscript text and discussion notes live outside the public repository, in the private manuscript workspace until a public preprint or published version exists.

Run commands from the repository root.

cd morseframes

The examples below write raw CSV, Markdown summaries, and diagnostic prose to ../work/. The public repository tracks the scripts and selected LaTeX table fragments, but not the manuscript prose built from them.

Some rendering commands below intentionally write tracked files under docs/. Use those commands when you want to refresh the public table fragments. If you only want to test the workflow on a local machine, redirect the table outputs to ../work/ or restore the tracked table fragments afterward.

Output Policy

Tracked public artifacts:

docs/*_table.tex: LaTeX table fragments used to report benchmark results.
tools/*.py: benchmark, validation, and table-rendering scripts.
benchmarks/benchmark_gudhi_view.cpp: native GUDHI-view benchmark.

Local or private artifacts:

../work/*.csv, ../work/*.md, ../work/*.json: raw benchmark outputs and summaries.
docs/*_prose.tex: generated prose fragments. These are ignored by Git and should be copied into the private notes repository only when needed.
report PDFs and manuscript drafts: private-note material, not public package documentation.

Quick Validation

These checks are the fastest way to confirm that the source tree is usable.

MORSEFRAMES_DISABLE_CPP_BACKEND=1 \
  python3 -m unittest discover -s python/tests -p "test_*.py"

PYTHONPATH=python python3 python/examples/quickstart.py
PYTHONPATH=python python3 python/examples/prime_field_tutorial.py --modulus 3

To include the native C++ backend, install the package in editable mode:

python3 -m pip install -e ".[dev]"
python3 -c "import morseframes as mf; print(mf.__version__, mf.cpp_backend_available())"

The C++ smoke tests are:

cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure

Synthetic Morse vs Standard Benchmarks

The main synthetic runner is tools/benchmark_persistence.py. It can run one strategy, or the default strategy portfolio with --sequence-algorithm portfolio.

Small smoke run:

mkdir -p ../work
PYTHONPATH=python python3 tools/benchmark_persistence.py \
  --preset smoke \
  --sequence-algorithm portfolio \
  --format summary \
  --output ../work/benchmark_smoke_summary.txt

Regenerate the public synthetic scale table:

mkdir -p ../work
PYTHONPATH=python python3 -c "import morseframes as mf; print(mf.cpp_backend_available())"

PYTHONPATH=python python3 tools/benchmark_persistence.py \
  --families lower-star plateau rips \
  --sizes 48 \
  --seeds 0 1 2 \
  --repeats 3 \
  --sequence-algorithm portfolio \
  --validation-mode core \
  --format csv \
  --output ../work/synthetic_scale_size48_portfolio.csv

PYTHONPATH=python python3 tools/render_synthetic_scale_table.py \
  --input ../work/synthetic_scale_size48_portfolio.csv \
  --table-output docs/synthetic_scale_table.tex \
  --prose-output ../work/synthetic_scale_prose.tex

The table reports Std/Morse, so values above 1 mean the Morse pipeline is faster than ordinary full-complex persistence for that row.

The tracked synthetic table is a native-backed core-mode benchmark. Before replacing it, make sure the backend check above prints True; otherwise the CSV will contain cpp_backend=False rows and the timing will describe the pure-Python fallback instead of the optimized C++ backend.

Roadmap and External Data

The benchmark runner also has Roadmap and CAM-style families:

cam-s4-rips
roadmap-rips

Roadmap datasets are cached under ../work/roadmap-data by default. Missing Roadmap files are not downloaded unless requested explicitly:

PYTHONPATH=python python3 tools/benchmark_persistence.py \
  --preset roadmap \
  --sequence-algorithm portfolio \
  --download-roadmap-data \
  --format csv \
  --output ../work/roadmap_portfolio.csv

Use this only when network access is acceptable.

Native GUDHI-View Benchmark

The native GUDHI benchmark compares three in-process paths on the same Gudhi::Simplex_tree input:

Direct: MorseFrames through a read-only Simplex_tree view.
Import: copy into the compact owning MorseFrames complex first.
GUDHI: GUDHI persistent cohomology on the original Simplex_tree.

This benchmark is optional because it needs GUDHI and Boost headers. Configure them explicitly when CMake cannot find them:

cmake -S . -B build-gudhi \
  -DMORSEFRAMES_GUDHI_INCLUDE_DIR=/path/to/gudhi/include \
  -DMORSEFRAMES_BOOST_INCLUDE_DIR=/path/to/boost/include

cmake --build build-gudhi --target morseframes_benchmark_gudhi_view

Quick run:

mkdir -p ../work
./build-gudhi/morseframes_benchmark_gudhi_view \
  --quick \
  --repeats 3 \
  > ../work/native_gudhi_view_quick.csv

PYTHONPATH=python python3 tools/render_native_gudhi_view_table.py \
  --input ../work/native_gudhi_view_quick.csv \
  --output docs/native_gudhi_view_quick_table.tex \
  --summary

PYTHONPATH=python python3 tools/render_native_gudhi_stage_profile.py \
  --input ../work/native_gudhi_view_quick.csv \
  --table-output docs/native_gudhi_stage_profile_quick_table.tex \
  --prose-output ../work/native_gudhi_stage_profile_quick_prose.tex \
  --summary

Default-size repeat run:

./build-gudhi/morseframes_benchmark_gudhi_view \
  --repeats 30 \
  > ../work/native_gudhi_view_default_r30.csv

PYTHONPATH=python python3 tools/render_native_gudhi_view_table.py \
  --input ../work/native_gudhi_view_default_r30.csv \
  --output docs/native_gudhi_view_default_r30_table.tex \
  --caption-title "Native \\texttt{Gudhi::Simplex\\_tree} default benchmark." \
  --label tab:native-gudhi-view-default-r30 \
  --summary

Larger lean run:

./build-gudhi/morseframes_benchmark_gudhi_view \
  --large \
  --lean \
  --repeats 30 \
  > ../work/native_gudhi_large_lean_r30.csv

PYTHONPATH=python python3 tools/render_native_gudhi_view_table.py \
  --input ../work/native_gudhi_large_lean_r30.csv \
  --output docs/native_gudhi_large_lean_r30_table.tex \
  --caption-title "Native \\texttt{Gudhi::Simplex\\_tree} larger lean benchmark." \
  --label tab:native-gudhi-large-lean-r30 \
  --summary

In these tables, GUDHI/Direct < 1 means GUDHI is faster end-to-end, while GUDHI/Reducer > 1 means the Morse reducer kernel alone is faster than GUDHI persistence after the Morse input has already been built.

Prime-Field Overhead

Prime-field coefficient experiments are generated by tools/benchmark_prime_field_overhead.py.

Quick local run:

mkdir -p ../work
PYTHONPATH=python python3 tools/benchmark_prime_field_overhead.py \
  --families lower-star plateau rips \
  --sizes 8 12 16 \
  --seeds 0 1 \
  --algorithms saturated f-max same-level-reduction \
  --primes 3 5 \
  --repeats 5 \
  --output-csv ../work/prime_field_overhead_quick.csv \
  --output-md ../work/prime_field_overhead_quick.md

Composite moduli are intentionally rejected by the barcode API; these reducers work over fields F_p.

Profile-Selection Validation

The profile-selection scripts compare cheap strategy-selection metrics against measured portfolio timings. These runs are more expensive than the smoke tests.

Preview the commands without executing them:

PYTHONPATH=python python3 tools/run_fair_profile_validation.py \
  --validation-preset report \
  --dry-run

Regenerate the public validation table from fresh timings:

mkdir -p ../work
PYTHONPATH=python python3 tools/run_fair_profile_validation.py \
  --validation-preset report \
  --output-dir ../work \
  --table-output docs/profile_metric_fair_validation_table.tex \
  --prose-output ../work/profile_metric_fair_validation_prose.tex \
  --manifest-output ../work/fair_profile_validation_manifest.md

If CSVs already exist in ../work, summaries can be regenerated without rerunning timings:

PYTHONPATH=python python3 tools/run_fair_profile_validation.py \
  --validation-preset report \
  --output-dir ../work \
  --summaries-only \
  --table-output docs/profile_metric_fair_validation_table.tex \
  --prose-output ../work/profile_metric_fair_validation_prose.tex \
  --manifest-output ../work/fair_profile_validation_manifest.md

Selector decision and feature diagnostic tables are rendered from the validation CSVs:

PYTHONPATH=python python3 tools/summarize_selector_decisions.py \
  --table-output ../work/profile_selector_decision_summary.txt \
  --csv-output ../work/profile_selector_decision_summary.csv \
  --latex-output docs/profile_selector_decision_summary_table.tex \
  --prose-output ../work/profile_selector_decision_summary_prose.tex

PYTHONPATH=python python3 tools/analyze_selector_features.py \
  --table-output ../work/selector_feature_diagnostic.txt \
  --csv-output ../work/selector_feature_diagnostic.csv \
  --json-output ../work/selector_feature_diagnostic.json \
  --latex-output docs/selector_feature_diagnostic_table.tex \
  --prose-output ../work/selector_feature_diagnostic_prose.tex

Benchmark Summary Page

The visible compact tables in docs/benchmark_summary.md are generated from the tracked LaTeX table fragments:

python3 tools/render_benchmark_summary.py

CI checks that this generated block is up to date:

python3 tools/render_benchmark_summary.py --check

Before Committing Regenerated Results

Before committing regenerated table fragments, run:

git diff -- docs tools benchmarks
git diff --check
python3 tools/render_benchmark_summary.py --check
MORSEFRAMES_DISABLE_CPP_BACKEND=1 \
  python3 -m unittest discover -s python/tests -p "test_*.py"

Commit only public artifacts that are meant to be reproducible from this repository. Keep manuscript text, discussion packages, generated prose, and PDFs in the private notes repository.