Execution

Local execution

The repository Makefile provides the main development and test entry points:

make help
make integration-test
make unit-test-all

The canonical local integration entry point runs the full-data Collector + Hailing Ducks profile.

Nextflow profiles

The principal profiles are:

testFullCollectorHailingDucks

Full summary-statistics test data using the collector locus breaker and Hailing Ducks LD annotation.

googleCloud / googleCloudTest

Google Cloud Batch execution profiles using Collector and Hailing Ducks.

To run a profile directly:

nextflow run main.nf -profile testFullCollectorHailingDucks -resume

Gentropy alternatives

The pipeline still contains Gentropy-based locus-breaking and LD-annotation modules for comparison and specialized runs. They are not selected by any shipped profile and are discouraged for routine execution because they are slower than the Collector and Hailing Ducks path.

To opt into a Gentropy module, override the relevant method and provide the required Gentropy container and Spark settings in a separate user config. For example:

nextflow run main.nf -c gentropy.local.config \
  --locus_breaker_method gentropy \
  --ld_annotation_method gentropy

The user config must provide the Gentropy process container and, for LD annotation, the appropriate gentropy_spark_uri and Spark configuration.

The -resume flag reuses successful tasks when their process inputs and commands have not changed.

Fine-mapping containers

The fine-mapping workflow currently supports MultiSuSiE. SuSiEx and SuShiE containers are pinned for pending integration work and should not be selected for production runs yet. The schema defaults use pinned images. Override them only when running locally built development images, for example:

params {
    fine_mapping_methods = ['multisusie']
    multisusie_container = 'multisusie:local'
    multisusie_purity_min_r2 = 0.01
    susiex_container = 'susiex:local'
    sushie_container = 'sushie:local'
}

Build and smoke-test the local images before running the non-stub pipeline:

docker build --tag multisusie:local /path/to/MultiSuSiE
docker build --tag susiex:local /path/to/susiex
docker build --tag sushie:local /path/to/sushie
docker run --rm multisusie:local --help
docker run --rm susiex:local --help
docker run --rm sushie:local --help
make integration-test

The nf-test suite uses process stubs and therefore does not require Docker or the method images. CI separately runs Nextflow lint and the complete nf-test suite on pull requests.

multisusie_purity_min_r2 must be strictly between 0 and 1. It defaults to 0.01. MultiSuSiE publishes only credible sets at or above this R-squared threshold; the process always runs with low-memory mode disabled so purity is available for filtering.

Hailing Ducks LD validation

The canonical local integration profile already selects the native DuckDB backend:

make integration-test

To run the same profile directly:

nextflow run main.nf -profile testFullCollectorHailingDucks -resume

The integration target builds and runs the local collector:1.1.0 image for collector-labelled processes and keeps the ordinary stub test suite network-independent. Pairwise LD files remain transient work artifacts for fine-mapping and are not published; only the compact LD statistics are retained. An explicit smoke target still checks the public Pan-UKBB hg38 HT, BlockMatrix, indel resolution, and signed LD extraction:

make -C tools/collector build REGISTRY=local VERSION=hailing-ducks-dev
make -C tools/collector hailing-s3-smoke \
  HAILING_DUCKS_IMAGE=local/collector:hailing-ducks-dev

To compare a Hailing Ducks result with a Gentropy result independent of row order and chr prefixes:

collector ld_parity \
  --hailing hailing.parquet \
  --gentropy gentropy_ld_dataset \
  --hailing_stats hailing.stats.jsonl \
  --gentropy_stats gentropy.stats.jsonl \
  --report parity.json