micro_dl.cli module

Submodules

micro_dl.cli.dataset_pooling module

Pool multiple datasets into a single dataset for training

micro_dl.cli.dataset_pooling.parse_args()

Parse command line arguments

In python namespaces are implemented as dictionaries :return: namespace containing the arguments passed.

micro_dl.cli.dataset_pooling.pool_dataset(config)

Parameters:

args (dict) – dict with input options

Returns:

micro_dl.cli.inference_script module

Model inference on larger images with and w/o stitching

micro_dl.cli.inference_script.parse_args()

Parse command line arguments

In python namespaces are implemented as dictionaries :return: namespace containing the arguments passed.

micro_dl.cli.inference_script.run_inference(config_fname, gpu_ids, gpu_mem_frac=None)

Run inference and optionally also evaluation metrics.

Parameters:

• config_fname (str) – Full path to config yaml file

• gpu_ids (int) – gpu id to use

• gpu_mem_frac (float/None) – gpu memory fraction to use

micro_dl.cli.metrics_script module

micro_dl.cli.metrics_script.compute_metrics(model_dir, image_dir, metrics_list, orientations_list, test_data=True, name_parser='parse_sms_name')

Compute specified metrics for given orientations for predictions, which are assumed to be stored in model_dir/predictions. Targets are stored in image_dir. Writes metrics csv files for each orientation in model_dir/predictions.

Parameters:

• model_dir (str) – Assumed to contain config, split_samples.json and subdirectory predictions/

• image_dir (str) – Directory containing target images with frames_meta.csv

• metrics_list (list) – See inference/evaluation_metrics.py for options

• orientations_list (list) – Any subset of {xy, xz, yz, xyz} (see evaluation_metrics)

• test_data (bool) – Uses test indices in split_samples.json,

otherwise all indices :param str name_parser: Type of name parser (default or parse_idx_from_name)

micro_dl.cli.metrics_script.parse_args()

Parse command line arguments

In python namespaces are implemented as dictionaries :return: namespace containing the arguments passed.

micro_dl.cli.preprocess_script module

Script for preprocessing stack

micro_dl.cli.preprocess_script.flat_field_correct(required_params, block_size, flat_field_channels)

Estimate flat_field_images in given channels.

Parameters:

• required_params (dict) – dict with keys: input_dir, output_dir, time_ids, channel_ids, pos_ids, slice_ids, int2strlen, uniform_struct, num_workers

• block_size (int) – Specify block size if different from default (32 pixels)

• flat_field_channels (list) – Channels in which to estimate flatfields.

Return str flat_field_dir:

full path of dir with flat field correction images

micro_dl.cli.preprocess_script.generate_masks(required_params, mask_from_channel, str_elem_radius, flat_field_dir, mask_type, mask_channel, mask_ext, mask_dir=None)

Generate binary masks from specified channel(s).

Parameters:

• required_params (dict) – dict with keys: input_dir, output_dir, time_ids, channel_ids, pos_ids, slice_ids, int2strlen, uniform_struct, num_workers

• mask_from_channel (int/list) – generate masks from sum of these channels

• str_elem_radius (int) – structuring element size for morphological opening

• flat_field_dir (str/None) – dir with flat field correction images

• mask_type (str) – string to map to masking function. Available options: ‘otsu’, ‘unimodal’, ‘dataset otsu’, ‘borders_weight_loss_map’

• mask_channel (int/None) – channel index assigned to mask channel.

• mask_ext (str) – ‘npy’ or ‘png’. Save the mask as uint8 PNG or NPY files

• mask_dir (str/None) – If creating weight maps from mask directory, specify mask dir

Return str mask_dir:

Directory with created masks

Return int mask_channel:

Channel number assigned to masks

micro_dl.cli.preprocess_script.generate_zscore_table(required_params, norm_dict, mask_dir)

Computes z-score parameters (zscore_median and zscore_iqr) from precomputed intensities sampled from blocks/tiles with a minimum foreground specified by masks (masks with min_fraction specified required). Updates frames_metadata with values for given normalization scheme.

Parameters:

• required_params (dict) – Required preprocessing parameters

• norm_dict (dict) – Normalization scheme (preprocess_config[‘normalization’])

• mask_dir (str) – Directory containing masks

micro_dl.cli.preprocess_script.get_required_params(preprocess_config)

Create a dictionary with required parameters for preprocessing from the preprocessing config. Required parameters are:

‘input_dir’: Directory containing input image data ‘output_dir’: Directory to write preprocessed data ‘slice_ids’: Slice indices ‘time_ids’: Time indices ‘pos_ids’: Position indices ‘channel_ids’ or ‘channel_names’: Channel indices or names (default). ‘uniform_struct’: (bool) If images are uniform ‘int2strlen’: (int) How long of a string to convert integers to ‘normalize_channels’: (list) Containing channel names or bools the length of channels ‘num_workers’: Number of workers for multiprocessing ‘normalize_im’: (str) Normalization scheme

(stack, dataset, slice, volume)

‘zarr_file’: Zarr file name in case of zarr file (as opposed to tiffs)

Parameters:

preprocess_config (dict) – Preprocessing config

Return dict required_params:

Required parameters

micro_dl.cli.preprocess_script.parse_args()

Parse command line arguments In python namespaces are implemented as dictionaries The preprocess_cli script takes as input a path to a configuration yaml file. For examples of preprocessing config files in 2.5D and 2D, see: https://github.com/mehta-lab/microDL/blob/main/config_files/Preprocessing-config_2.5DUnet_regression_phase2membrane.yml https://github.com/mehta-lab/microDL/blob/main/config_files/Preprocessing-config_2DUnet_regression_phase2nucleus.yml

Returns:

namespace containing the arguments passed.

micro_dl.cli.preprocess_script.pre_process(preprocess_config)

Preprocess data. Possible options are:

correct_flat_field: Perform flatfield correction (2D only currently) resample: Resize 2D images (xy-plane) according to a scale factor,

e.g. to match resolution in z. Resize 3d images

create_masks: Generate binary masks from given input channels do_tiling: Split frames (stacked frames if generating 3D tiles) into smaller tiles with tile_size and step_size.

This script will preprocess your dataset, save tiles and associated metadata. Then in the train_script, a dataframe for training data will be assembled based on the inputs and target you specify.

Parameters:

preprocess_config (dict) – dict with key options:

[input_dir, output_dir, slice_ids, time_ids, pos_ids correct_flat_field, use_masks, masks, tile_stack, tile] :param dict required_params: dict with commom params for all tasks :raises: AssertionError: If ‘masks’ in preprocess_config contains both channels and mask_dir (the former is for generating masks from a channel)

micro_dl.cli.preprocess_script.resize_images(required_params, scale_factor, num_slices_subvolume, resize_3d, flat_field_dir)

Resamples images by a scale factor for either 2D or 3D images using bilinear interpolation.

Parameters:

• required_params (dict) – dict with keys: input_dir, output_dir, time_ids, channel_ids, pos_ids, slice_ids, int2strlen, uniform_struct, num_workers

• scale_factor (int/list) – scale factor for each dimension

• num_slices_subvolume (int) – num of slices to be included in each volume. If -1, include all slices in slice_ids

• resize_3d (bool) – indicator for resize 2d or 3d

• flat_field_dir (str/None) – Directory containing flatfields

Return str resize_dir:

dir with resized images

Return int/list slice_ids:

corrected for gaps due to 3d. For ex. slice_ids=[0,1,…8] and num_slices_subvolume=3, returned slice_ids=[0, 2, 4, 6]

micro_dl.cli.preprocess_script.save_config(cur_config, runtime)

Save the current config (cur_config) or append to existing config.

Parameters:

• cur_config (dict) – Current config

• runtime (float) – Run time for preprocessing

micro_dl.cli.preprocess_script.tile_images(required_params, tile_dict, resize_flag, flat_field_dir, tiles_exist=False)

Tile images.

Parameters:

• required_params (dict) – dict with keys: input_dir, output_dir, time_ids, channel_ids, pos_ids, slice_ids, int2strlen, uniform_struct, num_workers

• tile_dict (dict) – dict with tiling related keys: tile_size, step_size, image_format, depths, min_fraction. Optional: mask_channel, mask_dir, mask_depth, tile_3d

• resize_flag (bool) – indicator if resize related params in preprocess_config passed to pre_process()

• flat_field_dir (str/None) – dir with flat field correction images

• tiles_exist (bool) – If tiling weights after other channels, make sure previous tiles are not erased

Return str tile_dir:

dir with tiled images

micro_dl.cli.train_script module

Train neural network models in keras

micro_dl.cli.train_script.create_datasets(df_meta, tile_dir, dataset_config, trainer_config, image_format, masked_loss)

Create train, val and test datasets

Saves val_metadata.csv and test_metadata.csv for checking model performance

Parameters:

• df_meta (pd.DataFrame) – Dataframe containing info on split tiles

• tile_dir (str) – directory containing training image tiles

• dataset_config (dict) – dict with dataset related params

• trainer_config (dict) – dict with params related to training

• image_format (str) – Tile shape order: ‘xyz’ or ‘zyx’

• masked_loss (bool) – Whether or not to use masks

Returns:

Dict containing :BaseDataSet df_train: training dataset :BaseDataSet df_val: validation dataset :BaseDataSet df_test: test dataset :dict split_idx: dict with keys [train, val, test] and list of sample

numbers as values

micro_dl.cli.train_script.create_network(network_config, gpu_id)

Create an instance of the network

Parameters:

• network_config (dict) – dict with network related params

• gpu_id (int) – gpu to use

micro_dl.cli.train_script.get_image_dir_format(dataset_config)

Get dir with input images for generating full path from frames_meta

If the tiled dir is passed as data dir there will be no preprocessing_info.json. If json present use it, else read images from the given dir.

micro_dl.cli.train_script.parse_args()

Parse command line arguments

In python namespaces are implemented as dictionaries :return: namespace containing the arguments passed.

micro_dl.cli.train_script.run_action(action, config, gpu_ids, gpu_mem_frac, model_fname=None)

Performs training or tunes hyper parameters Lambda layers throw errors when converting to yaml! model_yaml = self.model.to_yaml()

Parameters:

• action (str) – Currently the only supported action is ‘train’

• config (dict) – Training config

• gpu_ids (int) – GPU ID

• gpu_mem_frac (float) – Available GPU memory fraction

• model_fname (str) – Full path to model weights if not starting training from scratch

Module contents