This article guides users through the implementation of StarDist segmentation in arivis Vision4D.
Introduction
StarDist is a deep learning-based method for 2D and 3D nucleus detection, developed & published by Martin Weigert and Uwe Schmidt: github.com/stardist/. StarDist uses a cell detection method that predicts a shape representation with star-convex polygons that is well-suited to approximate the typically roundish shapes of cell nuclei in microscopy images.The 3D shape of a single object (cell nucleus) is described using a star-convex polyhedron instead of polygons.
StarDist is reported to run well in multiple open source environments such as Fiji/ImageJ, Qupath or Python (Using Python editor like Jupyter Notebook, PyCham and Spider), but what are the benefits of integrating StarDist directly inside of your arivis Vision4D imaging software? In this document, we will highlight the advantages of integrating open source analysis tools such as StarDist directly in Vision4D.
Preliminary Remarks
Vision4D is able to run deep learning applications such as StarDist using external and arivis-independent Python libraries and tools produced by third parties.
These tools must be installed by the user under his or her own responsibility, strictly following the instructions in this document. arivis has tested the setup protocol on several computers, however, due to the different and unpredictable hardware and software configurations of any given computer system, the results may vary on a case-by-case basis. Therefore, arivis declines any responsibility concerning the correct tools, installation, and set up on the individual user’s workstation. arivis cannot be made responsible for any malfunctioning or failure of the deep learning environment setup. arivis does not guarantee technical support on the setup task or on any deep learning application. Furthermore, arivis also declines any responsibility regarding the validity of the scientific results gathered from the deep learning application.
How does it work?
The StarDist workflow is based on three main steps:
- Object Annotation
- Network creation and training
- Inference (image analysis)
Objects annotation
This task consists of manually drawing the shape of the object over a set of representative images (2D or 3D). The reference objects should describe all their possible variation within the reference samples. The annotations are then used to create a binary masked image (Ground-truth). Both the annotations and the related binary masks are used afterward by the training task to build the Neuronal Network (training).
The annotation task is a manual activity and therefore requires a lot of time to be performed. The correct number of annotations must be estimated in advance in order to get reliable training results. During the training, the annotations can be increased if required. The annotations range starts from a minimum number of 200 samples, spread over 10 - 20 different images, up to a thousand (or even tens of thousands) in the more complex cases.
In order to increase the number of samples without the need to acquire new images, it is also possible to re-use the already existing images after applying some operations like rotations, random flips, or intensity shifts of the original.
Creating and training a neural network
This task takes either the sample images or the related masked images to build the Neuronal Network. The training is a loop in which, in any cycle, 2 parameters are computed: the training loss and the validation loss.
The progress of training can be evaluated by comparing the training loss with the validation loss. During training, both values should decrease before reaching the minimal value, which should not change significantly with further cycles. Comparing the validation loss development with the training loss can give insights into the model’s performance. If both training and validation loss values are still decreasing, this indicates that training is still necessary. If the validation loss suddenly increases again, while the training loss decreases towards zero, it usually means that the network is overfitting to the training data.
Basically, the training is based on math operations. These operations are repetitive and time-consuming and can easily be parallelized. The usage of GPU resources improves the training performance in reducing the total time. Working with the CPU only, a complex training can take 7 to 10 days of work, while using the GPU the total time may reduce to mere hours (10 to 12).
Image analysis
Once the Neuronal Network is trained, it can be used to analyze sample images.
Why use StarDist within Vision4D?
arivis Vision4D (V4D) is a modular software for working with multi-channel 2D, 3D, and 4D images of almost unlimited size, independent of available RAM. Many imaging systems, such as high-speed confocal, light sheet / SPIM, and 2-photon microscope systems can produce vast amounts of multi-channel data, which Vision4D handles without constraints.
V4D allows the user to execute complex analysis tasks in automatic or batch mode. This includes sophisticated pre-processing algorithms, multiple segmentation approaches (including machine learning segmentation tools), and powerful data handling. Dataset from Mega- to Terabytes in size can be quantified by Vision4D with a single pipeline on virtually any computer that is capable of holding the data (see system requirements).
StarDist represents an advanced method to detect roundish objects such as cells and nuclei, especially in crowded fields where the objects are overlapping, but it is limited to these cases. The new frontiers of image analysis in life science need the capability to analyze the complex iterations between biological structures. Vision4D has the tools to satisfy these requirements. StarDist can be integrated into the V4D analysis workflow and directly contribute to better detect its target structures.
StarDist can already be currently executed as a python script but, in the near future, it will be available as a V4D pipeline operator, making its usage even more flexible and powerful.
Application examples:
Nuclei Tracking
Stardist is used to segment the nuclei over multiple timepoints while further operations take the result of the segmentation to create tracks and measure their properties.
Distribution Analysis
Stardist segments the nuclei, then a pipeline operation can import Atlas regions from a labelled image and compartmentalize the nuclei to specific regions and report numerical density.
Measuring Cell Volumes
StarDist is used to segment the cell nuclei and a region growing operation can be used to identify the boundaries of the cell based on a membrane staining.
How to get StarDist working with Vision4D
The StarDist python package is required to get it to work with Vision4D and must be added to an existing python 3.x enviroment. We tested & strongly recommend the Anaconda 3.x python package for the scope of this application.
Once the StarDist package has been correctly set up, Vision4D must also be configured accordingly by changing the scripting preferences.
Setting up Vision4D preferences
With Vision4D open, go to Extras> Preferences.
In the preferences window, select the Scripting tab on the left and select "Anaconda environment".
You will then need to browse your computer for the Anaconda installation folder and select the Stardist environment previously created.
By default, the new enviroments are stored under the \envs folder located in the Anaconda installation folder e.g. C:\Anaconda3\envs\stardist
Having selected the correct environemtn folder we can then run “Install arivis package” and "Test Environment" for compatibility:
arivis also provides a free python script to run the StarDist algorithm inside of V4D.
The script allows the user to select the active channel, as well as the time points and Z planes (full-range or sub-selection thereof), from which to segment objects. A new channel will be created to store the labeled objects found by StarDist.
Loading the script
A startup package including the python script, the technical instructions and the test image is available on request. Contact your arivis local area sales manager to get more information about how to get the python script mentioned here or use the contact form on our website.
In the Script Editor, the script can be opened by dragging. the .py file into an open window, of by going to File > Open... and navigating to the script file.
Once loaded some parameters will need to be changed for your specific images.
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ USER SETTINGS @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# -----------------------------------------------------------------------------
# Stardist 2D - Script
# INPUT_CHANNEL channel with the target objects
# is only used to read/write the voxels
# ---> imageset.get_channeldata / imageset.set_channeldata
# -----------------------------------------------------------------------------
INPUT_CHANNEL = 1 # <---- Count start from 0 (Ch#1 == 0)
# name of the new channel (skeleton storage)
OUTPUT_CH_NAME = "Stardist_" # Net_Skel_
#
#MODEL3D_PATH = "D:/Arivis_Dataset/Carlo Stardist/2021-05-10/3D/2021-04-22/models"
MODEL2D_PATH = "D:/Arivis_Dataset/Carlo Stardist/2021-05-10/2D/2021-05-03/models"
MODEL_NAME = "stardist"
# PARAMETERS YOU CAN CHANGE?
CURRENT_TIME_POINT = True
FIRST_PLANE = -1 # -1 == BOTTOM
LAST_PLANE = -1 # -1 TOP
#
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ END USER SETTINGS @@@@@@@@@@@@@@@@@@@@@@@@@@@@
Of specific importance are:
- Input Channel number (remember the first channel is channel "0"). This is to specify which channel to do the segmentation on
- Model Path and name. This will depend on your installation and the model you are using.
- Input range.
CURRENT_TIME_POINT, if true only the current time point is analyzed, if false the entire time series.
FIRST_PLANE, the first plane in the stack to be used for segmentation
LAST_PLANE, the last plane in the stack to be used for segmentation
Note that is both the first and last plane parameters are set to "-1" the script will use the full stack.
Once the parameters have been set the script can be saved and run.
Download Full "How to run StarDist within arivis Vision4D» PDF