How to create, share and import pipelines that use Deep Learning segmentation
TL;DR
Link your arivis Pro installation to you arivis Cloud account
- Log in to your arivis Cloud account and create an access token
- In arivis Pro, open the arivis Cloud Model Store and enter your access token
Creating a pipeline that includes ZEISS arivis AI segmentation
- Add the DL Segmenter to your pipeline and select your model
- Build the rest of your pipeline as needed
Exporting pipelines with DL instance segmentation
- Export your pipeline from the arivis Pro analysis panel
- Share your your ONNX/CZANN file or share your model using your arivis Cloud account
Importing pipelines with DL segmentation
- Connect to your arivis Cloud account and accept the shared model invitation, if necessary, create an access token
- import the pipeline in the arivis Pro analysis panel and point the segmentation operation to your downloaded model
- Run your pipeline
Introduction
Since arivis Vision4D 3.6 arivis has included Deep Learning inference operations in the analysis panel. As of the release of arivis Pro 4.2, pipelines can run DL Instance segmentation. Like all pipelines, those that include DL Instance segmentation can be shared to use on other systems and with other images, and can also be run in batch. However, DL pipelines must be linked to the specific model on which they are based and these models must be exported as well as the pipeline and linked back to the pipeline before execution.
There are two ways to select a model for use in a pipeline. Whether we are using the Deep Learning Reconstruction or the Deep Learning Segmenter, once it is added to the pipeline we need to select the model to be used:
The model can either be loaded from a file as either and ONNX or CZANN file, or selected from the arivis Cloud model store.
ONNX or CZANN models typically only allow semantic segmentation. If the model has been created outside of the Zeiss ecosystem, other formats are commonly used but these can usually be converted to ONNX and we provide some scripts to do this, including for PyTorch and Cellpose.
Access to models from arivis Cloud in arivis Pro is managed through access tokens. These access tokens give an arivis Pro installation access to every model in an arivis Cloud account. Therefore, rather than sharing an access token, which would give access to every model linked to an account, it is better to share the specific model with collaborators so that they can access the model by creating their own access token.
Arivis Cloud models can be trained for either instance or semantic segmentation. Semantic models can be exported as ONNX or CZANN directly from your arivis Cloud account. Instance models are only accessible through Access Tokens as mentioned above. Also, please note that DL segmentation in arivis Cloud instance models requires that you have installed and configured Docker on your system.
In either case, it is also a good idea to install and use the GPU acceleration package if you haven't done so already as this can significantly speed up DL and ML tasks.
Linking arivis Pro to your arivis Cloud account
As mentioned above, arivis Pro can use ONNX or CZANN files with DL operations, and these files are relatively simple to create, share and use. However, arivis Cloud instance segmentation creates models with additional dependencies which would ordinarily require the installation of additional software libraries and could cause compatibility issues. To avoid such issues, arivis uses Docker containers to store both the model and the dependencies. To facilitate management of these containers we access our models through the arivis Cloud Model Store.
Creating access tokens
As mentioned above, access to arivis Cloud models in arivis Pro is managed through access tokens. An access token gives access to every model available to an arivis Cloud account for the duration of the token validity. To use a token in arivis Pro we must first generate this token in arivis Cloud. To do this, first we log in to our arivis Cloud account, then we can click the More icon next to the user account name and select My Access Tokens. Then, click New Access Token, set a name and an expiration date then click Next. Finally, we can copy the access token.
Configuring arivis Pro using access tokens
To give arivis Pro access to your models, we need to provide it the access token generated previously. We find the arivis Cloud AI Model Store under the Analysis menu.
The first time we access the Model Store we'll be prompted to enter an access token and select a destination folder for the downloaded models.
When we click OK the Model store will automatically populate with all the models linked to your arivis Cloud account.
Note that some models may be incompatible do to versioning issues. If preferred, we can simply hide all incompatible models.
Also, the list of available models is dynamically updated after each restart of the application, so a new model is added there is no need for a new access token.
Creating Pipelines with DL Segmentation
In many ways, creating a pipeline that uses DL Instance or Semantic Segmentation is no different to creating any other pipelines. There is nothing special in the way the objects created by DL segmentation are handled compared to any other pipeline created segments. The Features available are the same, including Custom Features, and they can be used for any downstream segment processing operations, including tracking, parent-child analysis, and segment morphology operations to mention just a few. This ability to do both DL segmentation and traditional segmentation, and use the resulting segments all in the same pipeline, with the ability to batch process, is one of the key strengths of the arivis approach.
To create an analysis pipeline that uses DL we start the same way we always do to create pipelines, that is to say we open the Analysis panel, either from the Analysis menu or from the Shortcuts toolbar.
Then, in the Analysis panel we can create a new pipeline by using + New Pipeline, or choose an existing pipeline to modify.
With the pipeline open, we can set up the Input ROI and any other operations as needed, and add the Deep Learning Segmenter to the pipeline using + Add Operation.
Note that there are two ways to use DL in pipelines.
The Deep Learning Reconstruction can use a model to create new images of the probability maps from that model. These probability maps can be used like any channel in the pipeline. This includes filtering (denoising, morphology, image math etc), and segmentation. We can, for example, use a Blob Finder on a probability map of a semantic model to obtain an instance segmentation result. Dell Learning Reconstruction only support ONNX or CZANN models.
However, the majority of cases will call for the Deep Learning Segmenter which uses the model to generate objects form the image.
Once we've added the Deep Learning Segmenter to our pipeline, all we need to do is select which model we want to use. If we use ONNX or CZANN file option we then click the browse button and select our model file.
If we use arivis Cloud models, we can either select from previously downloaded models, or open the Model Store to download models as needed.
Once we've selected the model, the operation works like any other segmentation operation. We can preview the results, choose an output tag and colour, and the segmented objects can be used in downstream pipeline operations like any other pipeline objects.
Exporting Pipelines with DL Instance Segmentation
Exporting the pipeline
As with any pipeline created in arivis, we can use the Analysis Panel menu to open the export window...
...choose a save location...
...and send the .pipeline file to our collaborator.
But if the pipeline requires additional files, like DL models, these need to be shared as well.
An ONNX or CZANN file can just be shared together with the pipeline like any other file. The most practical way to do this may be to copy both the exported pipeline and the model file to a new folder, compress this folder into a ZIP file, and share the ZIP using your preferred file sharing method. Email should work fine, though some email clients have file size limits so other cloud-sharing methods might work better (OneDrive, Dropbox, etc.).
Sharing an arivis Cloud model
For arivis Cloud models, the person we need to share it with must have their own access token. We can create access tokens directly from our account, but this is not recommended. Instead, it is recommended that they use their own arivis Cloud account to create their own access tokens. In that case we must also share the model from our account to their account.
In arivis Cloud, we can go to the My Models page, click on the More icon next to the model we want to share, and then click the Share icon. Then, we type the name of the person you wish to share the model with, select them from the search results then click Save.
Note that only the model owner can share a model.
Once the model is shared, the collaborators will see the model in their model library. They can then create their own token as described above.
Importing the pipeline and model
If we receive a pipeline that includes a DL model from a collaborator, first we must save the pipeline to our workstation.
If the model was shared as an ONNX or CZANN file together with the pipeline, we simply save that file to the workstation along with the pipeline.
If the model was shared using an arivis cloud account, we can log in to our arivis Cloud account, accept the invitation to the shared model, and if necessary link the arivis installation to our Cloud account as described above. Once the model appears in the Model Store, we can click the Download link on the right:
Importing the analysis pipeline
Once the models have been added to the store, we can import the pipeline into the analysis panel. Then, we simply click the Menu icon at the top of the analysis panel, select the Import option, and select the pipeline file.
Pipelines using CZANN or ONNX models will need to be re-pointed to the new file location. Arivis Cloud pipeline should load the model automatically and we can then run the pipeline like any other.