sdcflows.config module

A Python module to maintain unique, run-wide SDCFlows settings.

This module implements the memory structures to keep a consistent, singleton config. Settings are passed across processes via filesystem, and a copy of the settings for each run and subject is left under <output_dir>/sub-<participant_id>/log/<run_unique_id>/sdcflows.toml. Settings are stored using ToML. The module has a to_filename() function to allow writing out the settings to hard disk in ToML format, which looks like:

Listing 1 Example file representation of SDCFlows settings.

[environment]
cpu_count = 36
exec_env = "posix"
free_mem = 13.7
overcommit_policy = "heuristic"
overcommit_limit = "50%"
nipype_version = "1.8.7.dev0"
templateflow_version = "23.0.0"
total_memory = 62.51573181152344
version = "2.6.1.dev5+gaa86b0b08f.d20231128"

[execution]
ants_float = false
bids_dir = "/data/datasets/hcph"
bids_database_dir = "/data/datasets/hcph/.bids-index"
bids_database_wipe = false
cwd = "/home/oesteban/tmp/sdcflows-hcph"
debug = false
dry_run = false
layout = "BIDS Layout: .../data/datasets/hcph | Subjects: 1 | Sessions: 59 | Runs: 4"
log_dir = "/home/oesteban/tmp/sdcflows-hcph/out/logs"
log_level = 15
notrack = false
output_dir = "/home/oesteban/tmp/sdcflows-hcph/out"
participant_label = [ "001",]
pdb = false
run_uuid = "20231129-112254_e9d6d38e-35ff-463e-9f7e-1ff15ecd54ac"
templateflow_home = "/data/templateflow"
work_dir = "/home/oesteban/tmp/sdcflows-hcph/work"
write_graph = false

[workflow]
analysis_level = [ "participant",]
fmapless = false
species = "human"
template_id = "MNI152NLin2009cAsym"

[nipype]
crashfile_format = "txt"
get_linked_libs = false
local_hash_check = true
nprocs = 8
omp_nthreads = 16
plugin = "MultiProc"
remove_node_directories = false
resource_monitor = false
stop_on_first_crash = true

[execution.bids_filters]

[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false

This config file is used to pass the settings across processes, using the load() function.

Configuration sections

class sdcflows.config.environment

Read-only options regarding the platform and environment.

Crawls runtime descriptive settings (e.g., default FreeSurfer license, execution environment, nipype and SDCFlows versions, etc.). The environment section is not loaded in from file, only written out when settings are exported. This config section is useful when reporting issues, and these variables are tracked whenever the user does not opt-out using the --notrack argument.

class sdcflows.config.execution

Configure run-level settings.

class sdcflows.config.workflow

Configure the particular execution graph of this workflow.

class sdcflows.config.nipype

Nipype settings.

Usage

A config file is used to pass settings and collect information as the execution graph is built across processes.

from sdcflows import config
config_file = mktemp(dir=config.execution.work_dir, prefix='.sdcflows.', suffix='.toml')
config.to_filename(config_file)
# Call build_workflow(config_file, retval) in a subprocess
with Manager() as mgr:
    from .workflow import build_workflow
    retval = mgr.dict()
    p = Process(target=build_workflow, args=(str(config_file), retval))
    p.start()
    p.join()
config.load(config_file)
# Access configs from any code section as:
value = config.section.setting

Logging

class sdcflows.config.loggers

Keep loggers easily accessible (see init()).

Other responsibilities

The config is responsible for other conveniency actions.

• Switching Python’s multiprocessing to forkserver mode.

• Set up a filter for warnings as early as possible.

• Automated I/O magic operations. Some conversions need to happen in the store/load processes (e.g., from/to Path <-> str, BIDSLayout, etc.)

sdcflows.config.dumps()

Format config into toml.

class sdcflows.config.environment

Bases: _Config

Read-only options regarding the platform and environment.

Crawls runtime descriptive settings (e.g., default FreeSurfer license, execution environment, nipype and SDCFlows versions, etc.). The environment section is not loaded in from file, only written out when settings are exported. This config section is useful when reporting issues, and these variables are tracked whenever the user does not opt-out using the --notrack argument.

cpu_count = 4

Number of available CPUs.

exec_docker_version = None

Version of Docker Engine.

exec_env = 'posix'

A string representing the execution platform.

free_mem = 11.1

Free memory at start.

nipype_version = '1.8.6'

Nipype’s current version.

overcommit_limit = '50%'

Linux’s kernel virtual memory overcommit limits.

overcommit_policy = 'heuristic'

Linux’s kernel virtual memory overcommit policy.

templateflow_version = '23.1.0'

The TemplateFlow client version installed.

total_memory = 15.60690689086914

Total memory available, in GB.

version = '0+unknown'

SDCFlows’ version.

class sdcflows.config.execution

Bases: _Config

Configure run-level settings.

ants_float = False

Use float number precision for ANTs computations.

bids_database_dir = None

Path to the directory containing SQLite database indices for the input BIDS dataset.

bids_database_wipe = False

Wipe out previously existing BIDS indexing caches, forcing re-indexing.

bids_description_hash = None

Checksum (SHA256) of the dataset_description.json of the BIDS dataset.

bids_dir = None

An existing path to the dataset, which must be BIDS-compliant.

bids_filters = None

A dictionary describing custom BIDS input filter using PyBIDS.

cwd = '/home/runner/work/sdcflows/sdcflows/docs'

Current working directory.

debug = False

Run in debug mode (allowing debugger, force single-thread).

dry_run = False

Just test, do not run.

classmethod init()

Create a new BIDS Layout accessible with layout.

layout = None

A BIDSLayout object, see init().

log_dir = None

The path to a directory that contains execution logs.

log_level = 25

Output verbosity.

notrack = False

Disable the sharing of usage information with developers.

output_dir = None

Folder where derivatives will be stored.

participant_label = None

List of participant identifiers that are to be preprocessed.

pdb = False

Drop into PDB when exceptions are encountered.

run_uuid = '20240214-164426_8c03cc89-c3b5-4552-9d09-1904ffefc77f'

Unique identifier of this particular run.

session_label = None

Filter input dataset by session identifier.

sloppy = False

Run in sloppy mode (meaning, suboptimal parameters that minimize run-time).

templateflow_home = PosixPath('/home/runner/.cache/templateflow')

The root folder of the TemplateFlow client.

work_dir = PosixPath('/home/runner/work/sdcflows/sdcflows/docs/work')

Path to a working directory where intermediate results will be available.

write_graph = False

Write out the computational graph corresponding to the planned preprocessing.

sdcflows.config.file_path: Path = None

Path to configuration file.

sdcflows.config.from_dict(settings)

Read settings from a flat dictionary.

sdcflows.config.get(flat=False)

Get config as a dict.

sdcflows.config.load(filename)

Load settings from file.

class sdcflows.config.loggers

Bases: object

Keep loggers easily accessible (see init()).

cli = <Logger cli (WARNING)>

Command-line interface logging.

default = <RootLogger root (WARNING)>

The root logger.

classmethod getLogger(name)

Create a new logger.

classmethod init()

Set the log level, initialize all loggers into loggers.

• Add new logger levels (25: IMPORTANT, and 15: VERBOSE).

• Add a new sub-logger (cli).

• Logger configuration.

interface = None

NiPype’s interface logger.

utils = None

NiPype’s utils logger.

workflow = None

NiPype’s workflow logger.

class sdcflows.config.nipype

Bases: _Config

Nipype settings.

crashfile_format = 'txt'

The file format for crashfiles, either text or pickle.

get_linked_libs = False

Run NiPype’s tool to enlist linked libraries for every interface.

classmethod get_plugin()

Format a dictionary for Nipype consumption.

classmethod init()

Set NiPype configurations.

local_hash_check = True

Check if interface is cached locally before executing.

memory_gb = None

Estimation in GB of the RAM this workflow can allocate at any given time.

nprocs = 4

Number of processes (compute tasks) that can be run in parallel (multiprocessing only).

omp_nthreads = 4

Number of CPUs a single process can access for multithreaded execution.

plugin = 'MultiProc'

NiPype’s execution plugin.

plugin_args = {'maxtasksperchild': 1, 'raise_insufficient': False}

Settings for NiPype’s execution plugin.

remove_node_directories = False

Remove directories whose outputs have already been used up.

resource_monitor = False

Enable resource monitor.

stop_on_first_crash = True

Whether the workflow should stop or continue after the first error.

sdcflows.config.restore_env()

Restore the original environment.

sdcflows.config.to_filename(filename)

Write settings to file.

class sdcflows.config.workflow

Bases: _Config

Configure the particular execution graph of this workflow.

analysis_level = ['participant']

Level of analysis.

fmapless = False

Allow fieldmap-less estimation

species = 'human'

Subject species to choose most appropriate template

template_id = 'MNI152NLin2009cAsym'

TemplateFlow ID of template used for the anatomical processing.