Content-Aware Computation for Optical Microscopy
NIH.AI Workshop: Optical Microscopy
Matthew Guay, NIH/NIBIB
matthew.guay@nih.gov
leapmanlab.github.io/nihai/jan20/
Content-aware computation
Currently, convolutional neural networks (CNNs) for image restoration and computer vision.
Image restoration: Image denoising, spatial and spectral deconvolution, superresolution, registration, and more!
Computer vision: Object detection and segmentation.
The big picture
Current applications: What problems are solved today?
Computational tools: What algorithms drive solutions?
Trust: Can we rely on data-driven methods?
Current applications
Image denoising
Photon budgets trade off between depth, intensity, and spatial and temporal resolution.
Main idea: Cheat the trade-off by restoring image features from low intensity acquisitions.
Seminal paper: Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy (Weigert et al., 2018) [PDF]
(Weigert et al., 2018). Restoring flatworm stained nuclei from low SNR images.
(Weigert et al., 2018). Medium-zoom comparison between a low-SNR flatworm image, CARE restoration, and high-SNR ground truth.
(Weigert et al., 2018). High-zoom comparison between a low-SNR flatworm image, CARE restoration, and high-SNR ground truth.
(Weigert et al., 2018). Restoring red flour beetle embryo stained nuclei from low SNR images.
Image superresolution
Main idea: Improve image resolution beyond raw acquisition limits.
Resolution artifacts may be due to diffraction limits, hardware limits, sparse axial sampling, others?
Learn statistical superresolution models from synthetic data and alternate superresolution techniques (STORM, PALM).
(Weigert et al., 2018). CARE enhancement of diffraction-limited secretory granules and microtubules in rat INS-1 cells.
(Weigert et al., 2018). CARE enhancement of diffraction-limited secretory granules and microtubules in rat INS-1 cells.
(Weigert et al., 2018). CARE superresolution to correct axial anisotropy in a zebrafish retina image.
Additional superresolution examples from the (Belthangady et al., 2019) review paper.
Computer vision
Image restoration: Learn a statistical model to infer a high-quality image from low quality.
Computer vision: Parse the structural content of an image into constituent objects.
Classic techniques (thresholding, watershedding) work in many cases.
CNNs may improve results in challenging cases, allow novel applications.
Image segmentation - instance
- Instance segmentation: Each distinct object gets a unique label.
Image segmentation - semantic
- Semantic segmentation: Each "type" of object gets a label, label each pixel.
Examples adapted from (von Chamier et al., 2019) [PDF]
Application: blood cell segmentation
- One CNN to segment red blood cells (RBC), another to diagnose malaria presence from segmented cells.
(Rajaraman et al., 2018) [PDF]. RBC slide, its segmentation, and an overlay.
Application: label-free fluorescence imaging
Train a CNN to predict fluorescence targets from bright-field images
Advantage: Predict multiple fluorescence labels at once when that is impractical physically.
Basically, semantic segmentation with fluorescence-derived class labels.
(Ounkomol et al., 2018)[PDF]. Label-free prediction model training on a fluorescence target.
(Ounkomol et al., 2018). Multi-target prediction across a time series by combining individually-trained networks.
Computational tools
Convolutional neural networks
Main idea: Learn convolutional features from image data to solve a classification task.
Sequential convolution operations + nonlinearities (ReLU) + pooling: learn abstract image-scale features to solve, e.g., classification problems.
Idea existed since the 90's, resurgence since the 2012 Alexnet results for natural image classification (Krizhevsky et al., 2012) [PDF]
Source. Example of a small CNN architecture. Convolutional features are learned and transformed into classification predictions about image content.
Convolution for images
Idea: Slide a little window along an image, do some computation in each window.
The computation: dot product between a window-sized kernel and the image.
Example: Gaussian blurring convolves a Gaussian kernel with an image.
CNNs learn kernels. Learned kernels are features, convolution output is a feature map.
Source. 2D convolution example. A sharpening kernel is applied to an "image".
Activation functions
Linear transforms like convolutions are followed by a nonlinear activation function.
Examples: ReLU is
$\max(0, x)$, sigmoid is$1/\left(1+e^{-x}\right)$.Networks with nonlinearities are more expressive: a sequence of linear transforms is just another linear transform.
Spatial pooling
Downsample an image or feature map to produce a "zoomed-out" view.
Convolution ops applied to downsampled data find patterns across larger spatial scales.
Max pooling: replace 2x2 pixel block with the single largest value.
Mean pooling: replace 2x2 pixel block with the average block value.
Source. Visualization of layers inside a small image classification CNN.
U-Nets
Vanilla CNNs provide spatially-coarse classification.
What about spatially-dense image problems? (Segmentation, restoration, etc.)
U-Net: Use convolutional encoder + decoder with skip connections for image-to-image translation.
Seminal paper: (Ronneberger et al., 2015) [PDF]
(Ronneberger et al., 2015). Diagram of the original U-Net architecture.
(Ronneberger et al., 2015). Image segmentation example with the U-Net. Blue input tiles provide spatial context for yellow output tiles. Large images can be segmented tile by tile.
Generative Adversarial Networks
Simultaneously train two networks, a generator (e.g. U-Net) and discriminator.
Generator trained to perform a task (e.g. image-to-image translation).
Discriminator trained to distinguish generator output from real training samples.
When successful, GAN generators produce more realistic output than without adversarial training.
(Belthangady et al., 2019). Comparison of image-to-image translation networks. A U-Net builds on an encoder-decoder by adding skip connections between convolution blocks. A GAN builds on a generator by forcing its output to be indistinguishable from training samples.
Example: thispersondoesnotexist.com
(Belthangady et al., 2019). Two superresolution approaches use GANs to force (fast) trainable network outputs to be indistinguishable from (slow) ground truth high-resolution computations.
Trust
Can content-aware methods be trusted?
Bottom line: No. Not always, not right now.
Neural network image restoration may fail subtly on new data or anomalous structures.
Long term, work on network interpretability will probably make results more trustworthy.
For now, validate like you would validate a human, not an algorithm.
(Belthangady et al., 2019). Demonstration of the pitfalls of data-driven image restoration. An image of the word "Witenagemot" is restored by U-Nets trained on different datasets. Each network produces a visually-plausible answer, but only the one trained on the right dataset produces a correct answer.
Model interpretability
Approach 1: model interpretability.
How does a network reach its prediction? What image features most contribute to it?
Seminal paper: (Olah et al., 2018) [link] for natural image classification.
No rock-solid solutions yet, research is ongoing.
Confidence prediction
Approach 2: predicting network confidence.
Instead of predicting a single pixel value, predict a per-pixel parametric statistical model.
(Weigert et al., 2018) tries predicting mean μ and std σ of Laplace distributions per pixel.
Goal: Low σ = high prediction confidence?
(Weigert et al., 2018). Predicting per-pixel probability distributions allows one to build pseudo-confidence intervals for network predictions. Here, even when ground-truth values differ from predictions, they fall within the predicted confidence interval. Still no guarantees the learned probability distributions converge to anything physical.
Ensemble agreement
Approach 3: measuring ensemble disagreement.
Ensemble: Collection of statistical models (neural nets or otherwise).
Train multiple instances of the same model from initial conditions.
Measure how well the models agree on a prediction.
(Weigert et al., 2018). Using ensemble disagreement to measure prediction confidence.
Bayesian neural networks
More sophisticated approach: Bayesian CNNs.
Learn probability distributions for weights to quantify model uncertainty.
Predict probability distributions for restored pixels to quantify data uncertainty.
See (Xue et al., 2019) for an application to phase imaging microscopy.
(Xue et al., 2019). The authors use ensembles of Bayesian CNNs to quantify uncertainty relative to the data and relative to the model.
Conclusion
Content-aware computation lets microscopists cheat physical limits of sample acquisition.
This enables scientific experiments that are impossible otherwise.
Results cannot be entirely trusted, may fail in subtle ways, trustworthiness will probably improve.
Question: What do you do today when the problem you want to solve requires content-aware tools?