Configuration

Every stage reads one YAML file, given with --config. example_config.yaml, reproduced at the end of this page, is the starting point: copy it, set the two output directories, and adjust what the screen requires. Each key is commented there, and the values shown are the defaults, except the channel lists, which are examples.

How the file is read

Section	Read by
top level: data_output_dir, local_output_dir, n_jobs	every stage
schema	every stage
preprocessing_bywell	chart preprocessing; the image locations also by the crops step
preprocessing_allwells, guide_filtering	chart preprocessing
archetypes	chart archetypes, and the channels and removed archetypes by every later stage
annotation, effects, programs	the stage of the same name

Only the two output directories are required. Tables are written under data_output_dir, and reports, plots and filter lists under local_output_dir. A relative path in the file is taken under the corresponding output directory, and image directories under scallops_dir where it is set; absolute paths and URLs such as s3:// are used as given. A key CHART does not recognise is reported with a warning, Unknown config key '...' in ...; it will be ignored, so a misspelt key does not pass silently. Command-line options such as --n-jobs and --parallel-channels override the corresponding keys for one run.

Settings by importance

Required. data_output_dir and local_output_dir.

Describing the screen. These depend on the data, and their defaults fit only the reference dataset. They are described under Your data, and the channels under Archetypes.

• schema: the identifier columns, feature_patterns, the controls control_gene and control_prefix, and the coordinate, bounding-box and area columns that quality control and the crops read

• preprocessing_bywell: the input location and file names, merged_dir, objects_pattern, features_pattern, or premerged and merged_pattern; and the image locations under global and wells

• archetypes: channels, all_channels and image_channels

Decided during the analysis. These are set after inspecting the outputs of an earlier step.

• archetypes.k, and archetypes.npc, after checking the recovery curve and the variance elbow (Archetypes)

• archetypes.archetypes_to_remove, after inspecting the crops (Archetypes)

• annotation.normal_archetypes, before annotating (Annotation)

Analysis parameters. The defaults are those of the published analysis. Changing them changes the results, so they are best changed deliberately and recorded.

• preprocessing_bywell.thresholds and guide_filtering: the quality-control and guide-filtering thresholds (Preprocessing)

• archetypes.params: the metacell graining, gamma and k.knn (Archetypes)

• effects.n_perm and effects.trimmed_alpha (Effects)

• programs.k, programs.n_modules and programs.qval_threshold (Programs)

• annotation.model (Annotation)

Resources. n_jobs, effects.n_jobs and archetypes.parallel_channels set how many cores and channels a run uses, and do not change the results (Running on a cluster).

Rarely changed. The remaining keys have defaults that suit most screens: the output subdirectories (outputs_dir, reports_dir), drop_unassigned and exclude_patterns, the normalisation and outlier settings in preprocessing_allwells, the archetype sample sizes and selected_params, the crop display (top_cells, display_percentiles, display_ranges), the seeds, and the annotation’s similarity_threshold and max_pairs_per_archetype.

The example file

# Example CHART configuration.  Copy this and edit the paths.
#
#   chart preprocessing --config my_screen.yaml --all-wells
#
# Every key below except the two output directories is optional, and the
# values shown are the defaults.  A file holding only those two runs the
# reference dataset: the wells are read off the input filenames, and the
# schema defaults are the names that dataset uses.
#
# Input and output tables are parquet.  Either directory may be a URL
# such as s3://bucket/prefix, which needs fsspec and s3fs installed.

# Bulk tables go here; reports, plots and filter lists stay local.
data_output_dir: /path/to/bulk/outputs
local_output_dir: /path/to/reports

# How many cores the normalize and guides steps, 'chart effects' and
# 'chart programs' use.  -1 is every core the run may use, which under
# SLURM is the job's share.  --n-jobs overrides it.
n_jobs: -1


# Where each role lives in your tables, and what the values mean.  Tables
# are not expected to use CHART's own column names.
schema:
  cell_column: label              # per-object identifier within a well
  guide_column: sgRNA             # guide identifier
  gene_column: gene_symbol        # gene the guide targets

  # A column is a feature when one of these matches its name.  These are
  # unanchored regular expressions, so 'nuclei_' matches anywhere in the
  # name and '^nuclei_' only at the start.  For CellProfiler output,
  # ['^Nuclei_', '^Cells_', '^Cytoplasm_'] is the usual choice.
  #
  # Every column that matches has to hold numbers.  One that does not
  # stops the run rather than being dropped quietly, so narrow these
  # patterns if the table carries text alongside the measurements.
  feature_patterns: [cell_, nuclei_, cytosol_]

  # Which gene values are controls: this name, or this prefix.  Set
  # control_prefix to null for screens with no prefix convention.
  control_gene: NTC
  control_prefix: OR

  # Where the QC and outlier position plots read cell coordinates, y then x.
  centroid_columns: [nuclei_centroid-0, nuclei_centroid-1]

  # Where the archetypes crops step reads each cell's bounding box in the
  # objects tables: first row, first column, then one past the last of each.
  bbox_columns: [cell_bbox-0, cell_bbox-1, cell_bbox-2, cell_bbox-3]

  # The areas the QC size plots and segmentation check read, and each
  # nucleus's bounding box, in the order above, for the pheno QC check.
  nuclei_area_column: nuclei_area
  cell_area_column: cell_area
  cytosol_area_column: cytosol_area
  nuclei_bbox_columns: [nuclei_bbox-0, nuclei_bbox-1, nuclei_bbox-2, nuclei_bbox-3]

  # The well is deliberately absent: it comes from the well being
  # processed, not from a column.  See objects_pattern below.


preprocessing_bywell:
  # Whether to drop cells whose guide or gene is unassigned.  Keeping
  # them lets the QC reports show how many there were.
  drop_unassigned: false

  # Where the input tables are read from.  A relative path is taken
  # under data_output_dir, so set this to an absolute path or a URL when
  # the input lives somewhere other than beside the outputs.
  # merged_dir: /data/screen1/objects/

  # Input filenames.  '{well}' is substituted with the well being
  # processed.  Leave these out for '{well}-objects.parquet' and
  # '{well}-features.parquet'.
  # objects_pattern: 'plate1_{well}.objects.parquet'
  # features_pattern: 'plate1_{well}.features.parquet'

  # Set premerged when one file per well holds objects and features
  # together.  merged_pattern then names it, and the two keys above are
  # unused.  Without a pattern it looks for '{well}.parquet'.
  # premerged: true
  # merged_pattern: 'plate1_{well}.all.parquet'

  # Feature columns to drop after feature_patterns has selected them.
  # Unanchored regular expressions again, but matched case-insensitively,
  # since they pick out position and orientation columns whatever the
  # screen calls them.  Leave this out for the defaults: bbox, centroid,
  # location, centermass, to_origin, _x, _y and orientation.
  # exclude_patterns: [_Location_, _Number_Object_Number]

  # Image directories, which only the QC step and the archetypes crops
  # step read.  Leave this out and the wells come from the input filenames
  # instead, which is all the steps up to normalisation need.  'global'
  # supplies the defaults for every well, and a well may override any of
  # them.  The crops step reads pheno_dir and the two label zarrs, which
  # hold labels/<well>-cell and labels/<well>-nuclei.
  # global:
  #   sbs_dir: /path/to/sbs
  #   sbs_feature_dir: /path/to/sbs/features
  #   dapi_channel_names: [DAPI, GFP, DAPI]
  #   cell_labels_dir: /path/to/cell-seg.zarr
  #   nuclei_labels_dir: /path/to/nuclei-seg.zarr
  # wells:
  #   A1:
  #     pheno_dir: /path/to/pheno/A1
  #     pheno_sbs_registered_dir: /path/to/registered/A1
  #   A2:
  #     pheno_dir: /path/to/pheno/A2
  #     pheno_sbs_registered_dir: /path/to/registered/A2


preprocessing_allwells:
  normalization_type: quantile

  # Outlier detection.  'auto' lets the forest choose how many cells to
  # flag; a float fixes the proportion.
  contamination: auto
  random_state: 42
  n_estimators: 100

  # Prefilters applied before detection.  The missing-value threshold is
  # a count, not a fraction.
  missing_value_threshold: 100000
  rcv_threshold: 0.01


guide_filtering:
  cosine_similarity_threshold: 0.2
  min_cells_per_guide: 50


# Read by 'chart archetypes', not by the preprocessing steps.  Each
# channel's steps work inside data_output_dir/outputs_dir/<channel>/,
# starting from the cell_pca_centered.parquet the pca step writes there
# out of the guide stage's cell table.  Outputs from an earlier run live
# under 'tmp_archetype_outputs/', so name that here to carry on from them.
archetypes:
  outputs_dir: archetypes/

  # The npc elbow, the partition comparison table and the K recovery
  # curves, which nothing downstream reads.  Under local_output_dir.
  reports_dir: archetypes/

  # Channels to analyse.
  channels: [TOM20, Golgin97]

  # Every channel in the feature table.  The pca step builds a channel's
  # space by removing the other channels' columns, which is what drops
  # the colocalisation features naming two of them, so leaving a channel
  # out here keeps its columns in everyone else's space.
  all_channels: [DAPI1, TOM20, Golgin97, CALR, EEA1]

  # gamma and k.knn, run as a grid for every channel.  The default is the
  # single pair below, which is what the published run used for all of its
  # channels.  Add rows to sweep, at the cost of running the metacells and
  # archetypes steps once per row.
  params:
    - [30, 10]

  # How many PCs to keep.  A channel left out gets the elbow of its own
  # variance curve, which is the usual case.
  # npc:
  #   TOM20: 14

  # Which of the swept pairs above the selectk and weights steps work
  # from.  Unset means the first one, which is the only one unless a
  # sweep was added.
  # selected_params: [30, 10]

  # How many archetypes.  A channel left out gets the knee of its
  # recovery curve, which the selectk step writes to k.txt for the
  # weights step to read.  Set it here to overrule that, after looking at
  # the curve the selectk step leaves under reports_dir.
  # k:
  #   TOM20: 12

  # How many metacells the archetypes step samples.  Each of its repeated
  # fits takes fit_sample_size, mostly from a spread-out subset of
  # coverage_sample_size, and every fit is scored on one shared sample of
  # eval_sample_size.  These are the published run's.  A channel with too
  # few metacells uses all of them, and the step warns that its fits then
  # differ only in their random starts.
  # coverage_sample_size: 12000
  # eval_sample_size: 8000
  # fit_sample_size: 12000

  # Archetypes 'chart effects' leaves out, as <channel>_ArchetypeN
  # under any category names.  Optional: leaving one out drops its rows,
  # and also shifts the other archetypes' q-values, since they are
  # corrected together.  A channel with none listed gets a warning.
  # archetypes_to_remove:
  #   staining_artifacts: [TOM20_Archetype4]
  #   seg_err: [TOM20_Archetype1]

  # The crops step, last of all, draws each archetype's most heavily
  # weighted cells from the images, in rows of five, under
  # reports_dir/<channel>/crops/.  It reads the image locations under
  # preprocessing_bywell and the bounding boxes from the across-wells
  # objects table.  image_channels gives each stain's position among the
  # image's channels, counting from 0; a channel it does not list gets a
  # warning and no crops, or an error under --steps crops.  Each well's
  # display range runs between two
  # percentiles of its pixels, 0 and 99.5 unless display_percentiles says
  # otherwise; display_ranges fixes it for particular wells instead.
  # image_channels:
  #   TOM20: 8
  #   Golgin97: 9
  # top_cells: 10
  # display_percentiles:
  #   TOM20: [0, 99.9]
  # display_ranges:
  #   TOM20:
  #     A1: [105, 4000]

  # How many channels run at once, each in its own process logging to
  # reports_dir/<channel>/.  Memory limits it rather than cores: the
  # metacells step needed about 62 GB for one channel of 875,000 cells.
  # --parallel-channels overrides it.
  # parallel_channels: 1


# Read by 'chart annotation', which is optional.  The key comes from the
# environment, never from this file.
annotation:
  reports_dir: annotation/

  # Each channel's normal archetypes, by number; every archetype is
  # compared with them.
  # normal_archetypes:
  #   TOM20: [10]
  #   FAK: [5, 8, 11]

  model: gpt-5.5

  # Summaries closer than this, in cosine distance, are contrasted with
  # each other; max_pairs_per_archetype caps the pairs, 0 for no cap.
  similarity_threshold: 0.08
  max_pairs_per_archetype: 0


# Read by 'chart effects', which runs on the weights the archetypes stage
# left for each of its channels.  It tests each gene against the control
# cells of every well the gene appears in, pooled, with a p-value per
# archetype from shuffling the labels within each well, and writes
# <channel>.parquet and <channel>.json under data_output_dir/outputs_dir/.
# 1000 permutations is what the published run used.  An n_jobs here
# overrides the top-level one for this stage.
effects:
  outputs_dir: effects/
  n_perm: 1000
  trimmed_alpha: 0.02
  seed: 42
  # n_jobs: -1


# Read by 'chart programs', which runs on effects/pooled.parquet.  It keeps
# the genes and archetypes with any q-value below qval_threshold, finds k
# programs among them by sparse PCA, groups the genes into n_modules
# modules and scores every cell, all under data_output_dir/outputs_dir/.
# The values are the published run's.
programs:
  outputs_dir: programs/
  k: 18
  qval_threshold: 0.01
  n_modules: 25
  seed: 42