Content-Aware Computation for Optical Microscopy

---

NIH.AI Workshop: Optical Microscopy

Matthew Guay, NIH/NIBIB
matthew.guay@nih.gov
leapmanlab.github.io/nihai/jan20/

### Content-aware computation - Currently, convolutional neural networks (CNNs) for image restoration and computer vision. - **Image restoration**: Image denoising, spatial and spectral deconvolution, superresolution, registration, and more! - ** Computer vision**: Object detection and segmentation.

### The big picture - **Current applications**: What problems are solved today? - **Computational tools**: What algorithms drive solutions? - **Trust**: Can we rely on data-driven methods?

Current applications

### Image denoising - Photon budgets trade off between depth, intensity, and spatial and temporal resolution. - **Main idea**: Cheat the trade-off by restoring image features from low intensity acquisitions. - Seminal paper: Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy (Weigert et al., 2018) [[PDF](https://www.biorxiv.org/content/biorxiv/early/2018/07/03/236463.full.pdf)]

(Weigert et al., 2018). Restoring flatworm stained nuclei from low SNR images.

(Weigert et al., 2018). Medium-zoom comparison between a low-SNR flatworm image, CARE restoration, and high-SNR ground truth.

(Weigert et al., 2018). High-zoom comparison between a low-SNR flatworm image, CARE restoration, and high-SNR ground truth.

(Weigert et al., 2018). Restoring red flour beetle embryo stained nuclei from low SNR images.

### Image superresolution - **Main idea**: Improve image resolution beyond raw acquisition limits. - Resolution artifacts may be due to diffraction limits, hardware limits, sparse axial sampling, others? - Learn statistical superresolution models from synthetic data and alternate superresolution techniques (STORM, PALM).

(Weigert et al., 2018). CARE enhancement of diffraction-limited secretory granules and microtubules in rat INS-1 cells.

(Weigert et al., 2018). CARE enhancement of diffraction-limited secretory granules and microtubules in rat INS-1 cells.

(Weigert et al., 2018). CARE superresolution to correct axial anisotropy in a zebrafish retina image.

Additional superresolution examples from the (Belthangady et al., 2019) review paper.

### Computer vision - Image restoration: Learn a statistical model to infer a high-quality image from low quality. - **Computer vision**: Parse the structural content of an image into constituent objects. - Classic techniques (thresholding, watershedding) work in many cases. - CNNs may improve results in challenging cases, allow novel applications.

### Image segmentation - instance - **Instance segmentation**: Each distinct object gets a unique label. <img class="stretch" src="img/ex_instance.png" alt="Instance segmentation demo">

### Image segmentation - semantic - **Semantic segmentation**: Each "type" of object gets a label, label each pixel. <img class="stretch" src="img/ex_semantic.png" alt="Semantic segmentation demo"> <small>Examples adapted from (von Chamier et al., 2019) [[PDF](https://portlandpress.com/biochemsoctrans/article-pdf/47/4/1029/850822/bst-2018-0391c.pdf)] </small>

### Application: blood cell segmentation - One CNN to segment red blood cells (RBC), another to diagnose malaria presence from segmented cells. <img class="strootch" src="img/malaria.png" alt="Segmentation of red blood cells for malaria detection"> <small>(Rajaraman et al., 2018) [[PDF](https://peerj.com/articles/4568.pdf)]. RBC slide, its segmentation, and an overlay. </small>

### Application: label-free fluorescence imaging - Train a CNN to predict fluorescence targets from bright-field images - Advantage: Predict multiple fluorescence labels at once when that is impractical physically. - Basically, semantic segmentation with fluorescence-derived class labels.

(Ounkomol et al., 2018)[PDF]. Label-free prediction model training on a fluorescence target.

(Ounkomol et al., 2018). Multi-target prediction across a time series by combining individually-trained networks.

Computational tools

### Convolutional neural networks - **Main idea**: Learn convolutional features from image data to solve a classification task. - Sequential convolution operations + nonlinearities (ReLU) + pooling: learn abstract image-scale features to solve, e.g., classification problems. - Idea existed since the 90's, resurgence since the 2012 Alexnet results for natural image classification (Krizhevsky et al., 2012) [[PDF](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)]

Source. Example of a small CNN architecture. Convolutional features are learned and transformed into classification predictions about image content.

### Convolution for images - Idea: Slide a little window along an image, do some computation in each window. - The computation: dot product between a window-sized **kernel** and the image. - Example: Gaussian blurring convolves a Gaussian kernel with an image. - CNNs learn kernels. Learned kernels are features, convolution output is a **feature map**.

Source. 2D convolution example. A sharpening kernel is applied to an "image".

### Activation functions - Linear transforms like convolutions are followed by a nonlinear **activation function**. - Examples: **ReLU** is `$\max(0, x)$`, sigmoid is `$1/\left(1+e^{-x}\right)$`. - Networks with nonlinearities are more expressive: a sequence of linear transforms is just another linear transform.

### Spatial pooling - Downsample an image or feature map to produce a "zoomed-out" view. - Convolution ops applied to downsampled data find patterns across larger spatial scales. - **Max pooling**: replace 2x2 pixel block with the single largest value. - Mean pooling: replace 2x2 pixel block with the average block value.

Source. Visualization of layers inside a small image classification CNN.

### U-Nets - Vanilla CNNs provide spatially-coarse classification. - What about spatially-dense image problems? (Segmentation, restoration, etc.) - **U-Net**: Use convolutional encoder + decoder with skip connections for _image-to-image translation_. - Seminal paper: (Ronneberger et al., 2015) [[PDF](https://arxiv.org/pdf/1505.04597.pdf)]

(Ronneberger et al., 2015). Diagram of the original U-Net architecture.

(Ronneberger et al., 2015). Image segmentation example with the U-Net. Blue input tiles provide spatial context for yellow output tiles. Large images can be segmented tile by tile.

### Generative Adversarial Networks - Simultaneously train two networks, a generator (e.g. U-Net) and discriminator. - Generator trained to perform a task (e.g. image-to-image translation). - Discriminator trained to distinguish generator output from real training samples. - When successful, GAN generators produce more realistic output than without adversarial training.

(Belthangady et al., 2019). Comparison of image-to-image translation networks. A U-Net builds on an encoder-decoder by adding skip connections between convolution blocks. A GAN builds on a generator by forcing its output to be indistinguishable from training samples.

Example: thispersondoesnotexist.com

(Belthangady et al., 2019). Two superresolution approaches use GANs to force (fast) trainable network outputs to be indistinguishable from (slow) ground truth high-resolution computations.

Trust

### Can content-aware methods be trusted? - **Bottom line**: **No**. Not always, not right now. - Neural network image restoration may fail subtly on new data or anomalous structures. - Long term, work on network interpretability will probably make results more trustworthy. - For now, validate like you would validate a human, not an algorithm.

(Belthangady et al., 2019). Demonstration of the pitfalls of data-driven image restoration. An image of the word "Witenagemot" is restored by U-Nets trained on different datasets. Each network produces a visually-plausible answer, but only the one trained on the right dataset produces a correct answer.

### Model interpretability - Approach 1: **model interpretability**. - How does a network reach its prediction? What image features most contribute to it? - Seminal paper: (Olah et al., 2018) [[link](https://distill.pub/2018/building-blocks/)] for natural image classification. - No rock-solid solutions yet, research is ongoing.

### Confidence prediction - Approach 2: **predicting network confidence**. - Instead of predicting a single pixel value, predict a per-pixel parametric statistical model. - [(Weigert et al., 2018)](https://www.biorxiv.org/content/biorxiv/early/2018/07/03/236463.full.pdf) tries predicting mean μ and std σ of Laplace distributions per pixel. - Goal: Low σ = high prediction confidence?

(Weigert et al., 2018). Predicting per-pixel probability distributions allows one to build pseudo-confidence intervals for network predictions. Here, even when ground-truth values differ from predictions, they fall within the predicted confidence interval. Still no guarantees the learned probability distributions converge to anything physical.

### Ensemble agreement - Approach 3: **measuring ensemble disagreement**. - Ensemble: Collection of statistical models (neural nets or otherwise). - Train multiple instances of the same model from initial conditions. - Measure how well the models agree on a prediction.

(Weigert et al., 2018). Using ensemble disagreement to measure prediction confidence.

### Bayesian neural networks - More sophisticated approach: **Bayesian CNNs**. - Learn probability distributions for weights to quantify _model uncertainty_. - Predict probability distributions for restored pixels to quantify _data uncertainty_. - See ([Xue et al., 2019](https://www.osapublishing.org/DirectPDFAccess/DF2983E0-90AF-0DDB-EBACECC4047414E8_412113/optica-6-5-618.pdf)) for an application to phase imaging microscopy.

(Xue et al., 2019). The authors use ensembles of Bayesian CNNs to quantify uncertainty relative to the data and relative to the model.

### Conclusion - Content-aware computation lets microscopists cheat physical limits of sample acquisition. - This enables scientific experiments that are impossible otherwise. - Results cannot be entirely trusted, may fail in subtle ways, trustworthiness will probably improve. - **Question**: What do you do today when the problem you want to solve requires content-aware tools?

Thank you!

Please send questions or comments to matthew.guay@nih.gov