Warning: This document is for an old version of dmriprep. The main version is master.

dmriprep.config package

A Python module to maintain unique, run-wide dMRIPrep settings.

This module implements the memory structures to keep a consistent, singleton config. Settings are passed across processes via filesystem, and a copy of the settings for each run and subject is left under <output_dir>/sub-<participant_id>/log/<run_unique_id>/dmriprep.toml. Settings are stored using ToML. The module has a to_filename() function to allow writing out the settings to hard disk in ToML format, which looks like:

Example file representation of dMRIPrep settings.
[environment]
cpu_count = 8
exec_env = "posix"
free_mem = 1.0
overcommit_policy = "heuristic"
overcommit_limit = "50%"
nipype_version = "1.4.2"
templateflow_version = "0.4.2"
version = "0.2.2"

[execution]
bids_dir = "THP002/"
bids_description_hash = "b07ee615a588acf67967d70f5b2402ffbe477b99a316433fa7fdaff4f9dad5c1"
boilerplate_only = false
debug = false
fs_license_file = "/opt/freesurfer/license.txt"
fs_subjects_dir = "opt/freesurfer/subjects"
layout = "BIDS Layout: .../home/oesteban/Data/THP002 | Subjects: 1 | Sessions: 2 | Runs: 2"
log_dir = "/tmp/dmriprep/logs"
log_level = 40
low_mem = false
md_only_boilerplate = false
notrack = true
output_dir = "/tmp"
output_spaces = "run"
reports_only = false
run_uuid = "20200311-121754_aa0b4fa9-6b60-4a11-af7d-02deb54a823f"
participant_label = [ "THP0005",]
templateflow_home = "/usr/share/templateflow"
work_dir = "work/"
write_graph = false

[workflow]
anat_only = false
fmap_bspline = false
force_syn = false
hires = true
ignore = []
longitudinal = false
run_reconall = true
skull_strip_fixed_seed = false
skull_strip_template = "OASIS30ANTs"
spaces = "run"

[nipype]
crashfile_format = "txt"
get_linked_libs = false
nprocs = 8
omp_nthreads = 8
plugin = "MultiProc"
resource_monitor = false
stop_on_first_crash = false

[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false

This config file is used to pass the settings across processes, using the load() function.

Configuration sections

class dmriprep.config.environment

Read-only options regarding the platform and environment.

Crawls runtime descriptive settings (e.g., default FreeSurfer license, execution environment, nipype and dMRIPrep versions, etc.). The environment section is not loaded in from file, only written out when settings are exported. This config section is useful when reporting issues, and these variables are tracked whenever the user does not opt-out using the --notrack argument.

cpu_count = 36

Number of available CPUs.

exec_docker_version = None

Version of Docker Engine.

exec_env = 'posix'

A string representing the execution platform.

free_mem = None

Free memory at start.

nipype_version = '1.5.1'

Nipype’s current version.

overcommit_limit = '50%'

Linux’s kernel virtual memory overcommit limits.

overcommit_policy = 'heuristic'

Linux’s kernel virtual memory overcommit policy.

templateflow_version = '0.7.1'

The TemplateFlow client version installed.

version = '0.5.0+30.g510ed0ea'

dMRIPrep’s version.

class dmriprep.config.execution

Configure run-level settings.

anat_derivatives = None

A path where anatomical derivatives are found to fast-track sMRIPrep.

bids_description_hash = None

Checksum (SHA256) of the dataset_description.json of the BIDS dataset.

bids_dir = None

An existing path to the dataset, which must be BIDS-compliant.

bids_filters = None

A dictionary of BIDS selection filters.

boilerplate_only = False

Only generate a boilerplate.

debug = False

Run in sloppy mode (meaning, suboptimal parameters that minimize run-time).

fs_license_file = None

An existing file containing a FreeSurfer license.

fs_subjects_dir = None

FreeSurfer’s subjects directory.

classmethod init()

Create a new BIDS Layout accessible with layout.

layout = None

A BIDSLayout object, see init().

log_dir = None

The path to a directory that contains execution logs.

log_level = 25

Output verbosity.

low_mem = None

Utilize uncompressed NIfTIs and other tricks to minimize memory allocation.

md_only_boilerplate = False

Do not convert boilerplate from MarkDown to LaTex and HTML.

notrack = False

Do not monitor dMRIPrep using Google Analytics.

output_dir = None

Folder where derivatives will be stored.

output_spaces = None

List of (non)standard spaces designated (with the --output-spaces flag of the command line) as spatial references for outputs.

participant_label = None

List of participant identifiers that are to be preprocessed.

reports_only = False

Only build the reports, based on the reportlets found in a cached working directory.

run_uuid = '20210907-144433_549b76d5-b169-4f63-a9c5-4946105ff607'

Unique identifier of this particular run.

templateflow_home = PosixPath('/root/.cache/templateflow')

The root folder of the TemplateFlow client.

work_dir = PosixPath('/root/project/work')

Path to a working directory where intermediate results will be available.

write_graph = False

Write out the computational graph corresponding to the planned preprocessing.

class dmriprep.config.workflow

Configure the particular execution graph of this workflow.

anat_only = False

Execute the anatomical preprocessing only.

dwi2t1w_init = 'register'

Whether to use standard coregistration (‘register’) or to initialize coregistration from the DWI header (‘header’).

fmap_bspline = None

Regularize fieldmaps with a field of B-Spline basis.

fmap_demean = None

Remove the mean from fieldmaps.

force_syn = None

Run fieldmap-less susceptibility-derived distortions estimation.

hires = None

Run FreeSurfer recon-all with the -hires flag.

ignore = None

Ignore particular steps for dMRIPrep.

longitudinal = False

Run FreeSurfer recon-all with the -logitudinal flag.

run_reconall = True

Run FreeSurfer’s surface reconstruction.

skull_strip_fixed_seed = False

Fix a seed for skull-stripping.

skull_strip_template = 'OASIS30ANTs'

Change default brain extraction template.

spaces = None

Keeps the SpatialReferences instance keeping standard and nonstandard spaces.

use_syn = None

Run fieldmap-less susceptibility-derived distortions estimation in the absence of any alternatives.

class dmriprep.config.nipype

Nipype settings.

crashfile_format = 'txt'

The file format for crashfiles, either text or pickle.

get_linked_libs = False

Run NiPype’s tool to enlist linked libraries for every interface.

classmethod get_plugin()

Format a dictionary for Nipype consumption.

classmethod init()

Set NiPype configurations.

memory_gb = None

Estimation in GB of the RAM this workflow can allocate at any given time.

nprocs = 36

Number of processes (compute tasks) that can be run in parallel (multiprocessing only).

omp_nthreads = 36

Number of CPUs a single process can access for multithreaded execution.

parameterize_dirs = False

The node’s output directory will contain full parameterization of any iterable, otherwise parameterizations over 32 characters will be replaced by their hash.

plugin = 'MultiProc'

NiPype’s execution plugin.

plugin_args = {'maxtasksperchild': 1, 'raise_insufficient': False}

Settings for NiPype’s execution plugin.

resource_monitor = False

Enable resource monitor.

stop_on_first_crash = True

Whether the workflow should stop or continue after the first error.

Usage

A config file is used to pass settings and collect information as the execution graph is built across processes.

from dmriprep import config
config_file = config.execution.work_dir / '.dmriprep.toml'
config.to_filename(config_file)
# Call build_workflow(config_file, retval) in a subprocess
with Manager() as mgr:
    from .workflow import build_workflow
    retval = mgr.dict()
    p = Process(target=build_workflow, args=(str(config_file), retval))
    p.start()
    p.join()
config.load(config_file)
# Access configs from any code section as:
value = config.section.setting

Logging

class dmriprep.config.loggers

Keep loggers easily accessible (see init()).

cli = <Logger cli (DEBUG)>

Command-line interface logging.

default = <RootLogger root (DEBUG)>

The root logger.

classmethod init()

Set the log level, initialize all loggers into loggers.

  • Add new logger levels (25: IMPORTANT, and 15: VERBOSE).

  • Add a new sub-logger (cli).

  • Logger configuration.

interface = <Logger nipype.interface (INFO)>

NiPype’s interface logger.

utils = <Logger nipype.utils (INFO)>

NiPype’s utils logger.

workflow = <Logger nipype.workflow (INFO)>

NiPype’s workflow logger.

Other responsibilities

The config is responsible for other convenience actions.

  • Switching Python’s multiprocessing to forkserver mode.

  • Set up a filter for warnings as early as possible.

  • Automated I/O magic operations. Some conversions need to happen in the store/load processes (e.g., from/to Path <-> str, BIDSLayout, etc.)

dmriprep.config.dumps(flat=False)

Format config into toml.

class dmriprep.config.environment

Bases: dmriprep.config._Config

Read-only options regarding the platform and environment.

Crawls runtime descriptive settings (e.g., default FreeSurfer license, execution environment, nipype and dMRIPrep versions, etc.). The environment section is not loaded in from file, only written out when settings are exported. This config section is useful when reporting issues, and these variables are tracked whenever the user does not opt-out using the --notrack argument.

cpu_count = 36

Number of available CPUs.

exec_docker_version = None

Version of Docker Engine.

exec_env = 'posix'

A string representing the execution platform.

free_mem = None

Free memory at start.

nipype_version = '1.5.1'

Nipype’s current version.

overcommit_limit = '50%'

Linux’s kernel virtual memory overcommit limits.

overcommit_policy = 'heuristic'

Linux’s kernel virtual memory overcommit policy.

templateflow_version = '0.7.1'

The TemplateFlow client version installed.

version = '0.5.0+30.g510ed0ea'

dMRIPrep’s version.

class dmriprep.config.execution

Bases: dmriprep.config._Config

Configure run-level settings.

anat_derivatives = None

A path where anatomical derivatives are found to fast-track sMRIPrep.

bids_description_hash = None

Checksum (SHA256) of the dataset_description.json of the BIDS dataset.

bids_dir = None

An existing path to the dataset, which must be BIDS-compliant.

bids_filters = None

A dictionary of BIDS selection filters.

boilerplate_only = False

Only generate a boilerplate.

debug = False

Run in sloppy mode (meaning, suboptimal parameters that minimize run-time).

fs_license_file = None

An existing file containing a FreeSurfer license.

fs_subjects_dir = None

FreeSurfer’s subjects directory.

classmethod init()

Create a new BIDS Layout accessible with layout.

layout = None

A BIDSLayout object, see init().

log_dir = None

The path to a directory that contains execution logs.

log_level = 25

Output verbosity.

low_mem = None

Utilize uncompressed NIfTIs and other tricks to minimize memory allocation.

md_only_boilerplate = False

Do not convert boilerplate from MarkDown to LaTex and HTML.

notrack = False

Do not monitor dMRIPrep using Google Analytics.

output_dir = None

Folder where derivatives will be stored.

output_spaces = None

List of (non)standard spaces designated (with the --output-spaces flag of the command line) as spatial references for outputs.

participant_label = None

List of participant identifiers that are to be preprocessed.

reports_only = False

Only build the reports, based on the reportlets found in a cached working directory.

run_uuid = '20210907-144433_549b76d5-b169-4f63-a9c5-4946105ff607'

Unique identifier of this particular run.

templateflow_home = PosixPath('/root/.cache/templateflow')

The root folder of the TemplateFlow client.

work_dir = PosixPath('/root/project/work')

Path to a working directory where intermediate results will be available.

write_graph = False

Write out the computational graph corresponding to the planned preprocessing.

dmriprep.config.from_dict(settings)

Read settings from a flat dictionary.

dmriprep.config.get(flat=False)

Get config as a dict.

dmriprep.config.init_spaces(checkpoint=True)

Initialize the spaces setting.

dmriprep.config.load(filename)

Load settings from file.

class dmriprep.config.loggers

Bases: object

Keep loggers easily accessible (see init()).

cli = <Logger cli (DEBUG)>

Command-line interface logging.

default = <RootLogger root (DEBUG)>

The root logger.

classmethod init()

Set the log level, initialize all loggers into loggers.

  • Add new logger levels (25: IMPORTANT, and 15: VERBOSE).

  • Add a new sub-logger (cli).

  • Logger configuration.

interface = <Logger nipype.interface (INFO)>

NiPype’s interface logger.

utils = <Logger nipype.utils (INFO)>

NiPype’s utils logger.

workflow = <Logger nipype.workflow (INFO)>

NiPype’s workflow logger.

class dmriprep.config.nipype

Bases: dmriprep.config._Config

Nipype settings.

crashfile_format = 'txt'

The file format for crashfiles, either text or pickle.

get_linked_libs = False

Run NiPype’s tool to enlist linked libraries for every interface.

classmethod get_plugin()

Format a dictionary for Nipype consumption.

classmethod init()

Set NiPype configurations.

memory_gb = None

Estimation in GB of the RAM this workflow can allocate at any given time.

nprocs = 36

Number of processes (compute tasks) that can be run in parallel (multiprocessing only).

omp_nthreads = 36

Number of CPUs a single process can access for multithreaded execution.

parameterize_dirs = False

The node’s output directory will contain full parameterization of any iterable, otherwise parameterizations over 32 characters will be replaced by their hash.

plugin = 'MultiProc'

NiPype’s execution plugin.

plugin_args = {'maxtasksperchild': 1, 'raise_insufficient': False}

Settings for NiPype’s execution plugin.

resource_monitor = False

Enable resource monitor.

stop_on_first_crash = True

Whether the workflow should stop or continue after the first error.

dmriprep.config.redirect_warnings(message, category, filename, lineno, file=None, line=None)

Redirect other warnings.

dmriprep.config.to_filename(filename)

Write settings to file.

class dmriprep.config.workflow

Bases: dmriprep.config._Config

Configure the particular execution graph of this workflow.

anat_only = False

Execute the anatomical preprocessing only.

dwi2t1w_init = 'register'

Whether to use standard coregistration (‘register’) or to initialize coregistration from the DWI header (‘header’).

fmap_bspline = None

Regularize fieldmaps with a field of B-Spline basis.

fmap_demean = None

Remove the mean from fieldmaps.

force_syn = None

Run fieldmap-less susceptibility-derived distortions estimation.

hires = None

Run FreeSurfer recon-all with the -hires flag.

ignore = None

Ignore particular steps for dMRIPrep.

longitudinal = False

Run FreeSurfer recon-all with the -logitudinal flag.

run_reconall = True

Run FreeSurfer’s surface reconstruction.

skull_strip_fixed_seed = False

Fix a seed for skull-stripping.

skull_strip_template = 'OASIS30ANTs'

Change default brain extraction template.

spaces = None

Keeps the SpatialReferences instance keeping standard and nonstandard spaces.

use_syn = None

Run fieldmap-less susceptibility-derived distortions estimation in the absence of any alternatives.