Protocol for applying Machine Learning models for the transformation of conventional fluorescence images to super-resolution

doi:10.21203/rs.3.pex-1721/v1

Method Article

Protocol for applying Machine Learning models for the transformation of conventional fluorescence images to super-resolution

https://doi.org/10.21203/rs.3.pex-1721/v1

This work is licensed under a CC BY 4.0 License

This protocol has been posted on Protocol Exchange, an open repository of community-contributed protocols sponsored by Nature Portfolio. These protocols are posted directly on the Protocol Exchange by authors and are made freely available to the scientific community for use and comment.

Version 1

posted

You are reading this latest protocol version

Machine Learning offers the opportunity to visualize the invisible in conventional fluorescence microscopy images by improving their resolution while preserving and enhancing image details. This protocol describes the application of GAN-based Machine Learning models to transform the resolution of conventional fluorescence microscopy images to a resolution comparable with super-resolution. It provides a flexible environment using a modern app functioning on both desktop and mobile computers. This approach can be extended for use on other types of microscopy images empowering life science researchers with modern analytical tools.

Computational biology and bioinformatics

machine learning

fluorescence microscopy

super-resolution

tablet computer

The majority of studies evaluating fluorescently-labeled proteins are conducted using conventional microscopes with resolution determined by the physical properties of light at approximately 250 nm. This diffraction limit makes conventional microscopes unusable for analytical studies, since they are unable to distinguish many multi-protein complexes¹.

In recent years, this limit has been overcome with the help of super-resolution microscopy (SRM). These sophisticated microscopes can achieve a resolution of approximately 100 nm and, more recently, even 25-50 nm, which makes them the tools of choice for various analytical applications^2,3. However, SRMs utilize highly specialized hardware, can be too daunting to operate and, since images are acquired at multiple frames, require significant computational resources. The occurrence of artifacts in super-resolution images was also reported^4,5. Importantly, SRMs are usually orders of magnitude more expensive than conventional microscopes, putting them out of reach for many research labs.

The latest advances in Machine Learning (ML) provided another way to achieve super-resolution in microscopy images^6,7. With the help of ML models based on convolutional neural networks (CNN), it became possible to use images obtained using conventional microscopes and transform them into images with resolutions comparable to SRMs⁸. The biggest issues when creating traditional ML models are the vast amounts of images required for model training and the need for their annotation. Large sets are required for supervised learning to properly learn the mapping between a source domain and a target domain and annotation of images in them serves to identify pairs of sources and targets⁹. These issues obstruct the model development, since large sets of highly specific images are frequently impossible to obtain and proper annotation of them is very laborious and error-prone.

In this protocol, we describe the application of Generative Adversarial Networks (GAN)-based ML models for the transformation of conventional microscopy images to images comparable with super-resolution. GAN-based models proved to be a highly efficient architecture capable of precise image transformation and reconstruction¹⁰. GAN networks are unsupervised. Importantly, they can perform with significantly less model training data and do not require paired training sets. GAN networks train a generative model and an adversarial model simultaneously, learn to optimize a loss function, and produce more realistic results. GAN-based models were used with great success for bright-field holography¹¹, virtual histological staining¹², denoising of optical diffraction tomography images¹³, and super-resolution reconstruction^14,15. We employed a model architecture similar to a so-called Multi-Scale GAN (MSGAN)¹⁶. This architecture, Super-Resolution GAN (SRGAN) generates a low-resolution version of the image first followed by the generation of patches of constant sizes but successively growing resolutions before generating the final image (Figure 1).

Our model combines three separate models dealing with independent image transformation tasks: Denoising, Axial Resolution Restoration, and Super-Resolution Reconstruction (Figure 2). The protocol itself consists of two workflows: workflow A and workflow B for conventional and super-resolution images, respectively (Figure 3). We enabled the use of the protocol on both desktop and mobile computing platforms¹⁷. Workflow A uses the SRGAN ML combined model to reduce the image noise and increase resolution and is applied to conventional images. Workflow B uses the SRGAN ML BC model to reduce the image noise and is applied to original super-resolution images.

Hardware

· A conventional fluorescence (Workflow A) or super-resolution (Workflow B) microscope for newly acquired images.

· A desktop or laptop Mac computer with at least 8 GB of GPU RAM (16GB is recommended) running macOS 10.15 and newer.

· A tablet computer running iPadOS 13 and newer.

Critical: The microscope should be able to save images as RGB files readable by the CoLocalizer app.

Software

· Desktop: CoLocalizer Pro for Mac 7.0 and newer (CoLocalization Research Software) (https://colocalizer.com/mac/), Fiji (https://fiji.sc).

· Tablet: CoLocalizer for iPad 2.0 and newer (CoLocalization Research Software) (https://geo.itunes.apple.com/app/colocalizer/id1116017542?mt=8).

· Desktop and tablet: Microsoft Excel (https://products.office.com/en/excel).

Before you begin

Before capturing any photos, careful consideration should be given to the image file type and storage choices. Many microscopes save acquired images in lossy compression formats, such as JPEG, due to their smaller file size set by default. However, lossy formats “discard” important image color data and are not suitable for quantitative analysis. Other popular file formats saving options, such as TIFF and PNG, are lossless. Natively, the CoLocalizer app used to transform images in this protocol, opens a limited number of lossy and lossless image file formats, including JPEG, PNG, and TIFF. Other proprietary microscopy image formats (ICS, LIF, LSM, ND2, etc) can only be opened following conversion via 3^rd party software tools and are not always optimal storage-wise, eventually limiting their use. Therefore, saving captured images as lossless TIFFs and PNGs is highly recommended. We strongly discourage using previously acquired images in lossy formats in this protocol.

Launching CoLocalizer app and opening images

Timing: 1 min

1. Launch the CoLocalizer app either on a desktop (Mac) or a tablet (iPad) computer (Figure 4).

2. Open your image file:

On a desktop:

i. Open the image by selecting the File > Open path from the application menu bar or using the Command+O shortcut or dragging and dropping the file on a CoLocalizer app icon in the dock (Figure 4a).

ii. If you are using single-channel images, merge them on a desktop before opening for transformation. The Merge functionality is not available on a tablet. To merge, select Tools > Merge from the application menu bar or use the Shift+Command+M shortcut to open the Merge window.

iii. Drag-and-drop single-channel images onto the respective image wells of the Merge window and click the Merge button to merge the selected channels (Figure 4b).

On a tablet:

i. Open the image by tapping the image thumbnail on the File Browser screen (Figure 4c).

ii. Once tapped, the image will open on a new Open Image screen (Figure 4d).

Critical: Due to system RAM limitations, the protocol currently works on SISR only. If you need to apply it to the image of stacks, split it into single images (slices) and apply the protocol to the slices one-by-one.

Applying ML models

Timing: 10-30 min per image

3. Once the image is opened, apply ML models to transform it. For conventional images, use the Conventional image approach (Workflow A) (Figure 5). For original super-resolution images, use the Super-Resolution approach (Workflow B) (Figure 5). The Conventional approach will apply the SRGAN ML combined model consisting of denoising, axial restoration, and super-resolution reconstruction models. The Super-Resolution approach will apply the SRGAN ML BC model consisting of the denoising model.

Workflow A

On a desktop

i. Select the model by going to Image > ML Super Resolution in the application menu bar or use the Option+Command+U shortcut (Figure 5a).

ii. Wait for SRGAN ML combined model to be applied (Figure 5b).

iii. Observe the transformed image (Figure 5c).

On a tablet:

i. Select the model by tapping the ML Super Resolution icon in the navigation bar (Figure 5d).

ii. Wait for the SRGAN ML combined model to be applied (Figure 5e).

iii. Observe the transformed image (Figure 5f).

Workflow B

On a desktop:

i. Access the model either by clicking the Background icon in the application toolbar or going to Tools > Background Correction in the application menu bar or using the Shift+Command+B shortcut (Figure 5g).

ii. In the opened Background Correction view, select the color channels used for staining antigens in the image (Figure 5h).

iii. Click the ML Correct button to apply the SRGAN ML BC model (Figure 5i).

On a tablet:

i. Access the model by tapping the Tools icon in the navigation bar and then selecting the Correct Background option from the Tools popover (Figure 5j).

ii. On the opened Background Correction screen, select the color channels used for staining antigens in the image (Figure 5k).

iii. Tap the ML Correct button to apply the SRGAN ML BC model (Figure 5l).

Critical: If a sample is stained for three different antigens and you wish to change a channel pair to view them all, change the Channel Pair using the selector at the top of the view. No need to click or tap the ML Correct button again.

Application of the models shows significantly improved quality of transformed images verified by several controls (Figure 6). After selecting a channel pair, the image will be synced between devices, i.e., it will be possible to switch between a desktop and a tablet and continue the image analysis on one device where it is left off on another device using the Handoff functionality.

Comparing results

Timing: 1-3 min per image

Workflow A and B

4. Once the models were applied, compare results in original vs transformed images. Comparison serves as a control of successful use of the models, since images are expected to change following their application.

On a desktop:

i. Return to the main window by clicking the Inspector icon in the application toolbar or going to Tools > Inspector in the application menu bar or using the Shift+Command+I shortcut.

ii. Choose Undo > ML Super Resolution in the application menu bar to return to the original image. Alternatively, go to File > Revert To and find the original image to revert to.

iii. Access the Colocalization view again by clicking the Colocalization icon in the application toolbar or going to Tools > Quantify Colocalization in the application menu bar or using the Shift+Command+C shortcut as described above.

iv. Compare pixel information data on transformed vs original images.

v. Compare original vs transformed images at higher magnification. To access the Zoom functionality, do one of the following: (a) Click the selector field in the Zoom icon in the application toolbar and select a higher zoom option or (b) Select View > Zoom In from the application menu bar or (c) Use the Command+Plus shortcut.

On a tablet:

i. Return to the Open Image screen by tapping Done at the top right of the navigation bar.

ii. Tap Undo at the top left of the navigation bar to return to the original image.

iii. Compare original vs transformed at higher magnification. To zoom the image, tap on it with two fingers and pinch open to zoom in. While pinching, you will see a zoom label indicating the current image size.

Exporting data

Timing: 1-3 min per image

Workflow A and B

5. The final step of the protocol is exporting obtained data.

On a desktop:

i. In the Colocalization view, click the Export Results… button at the bottom (Figure 7a).

ii. In the opened Export window, fill the Comments field and select the Data, Images, and Report options (Figure 7b). The file formats for Data (XLSX and Text) and Images (JPEG, PNG, and TIFF) can be set in the Preferences of the app. You can export the following features of analyzed images: the whole image, selected ROI, revealed pixels, and scattergram (scatter plot). Once selected, they will be exported as single files as well as included in the Report file.

iii. The Report option will allow the saving of all data in PDF and HTML file formats.

iv. Click the Save… button at the bottom of the window to save the selected options (Figure 7b).

v. To finish the export, select the destination where you would like to save the exported data, either locally or on iCloud Drive.

Caution: Saving the whole image in JPEG format will make it unusable for colocalization analysis. We recommend saving it as lossless TIFF.

Critical: If you wish to export only the transformed image, go to the application menu bar and select File > Export As… or use the Command+E shortcut. Then, save it in the file format of your choice. If you plan to reuse this image, we recommend saving it in native COLOCALIZER format.

On a tablet:

i. On the Open Image screen, tap the Export icon in the navigation bar at the right to access the Export popover (Figure 7c).

ii. In the popover, select the type of data to export, Data or Images. Only one option can be selected at a time.

iii. In the appeared pop-up screen, select the Data or Images exporting options. Only one option can be selected at a time. Data can be saved in either PDF or XLSX formats. Images can be saved in either JPEG, PNG, or TIFF formats (Figure 7d). In addition to coefficients numbers, a PDF file of Data will include the analyzed image, selected ROI, and scattergram (scatter plot) too.

iv. To finish the export, select the destination where you would like to save your data, either on On My iPad or on iCloud Drive.

Step 2 (image opening)

Problem:

The image file does not open

Possible reason:

Native image file formats of CoLocalizer apps are limited to JPEG, PNG, and TIFF

Solution:

Check the list of currently supported image file formats (https://colocalizer.com/support/). If your format is not on the list, convert it using a 3^rd party software, like Fiji (https://imagej.net/plugins/importing-image-files), before opening it in CoLocalizer

Step 3 (applying models)

Problem:

ML Super Resolution option is grayed out on a desktop and an alert is shown on a tablet when trying to apply it on image stacks

Possible reason:

Due to GPU RAM limitations, the use of models on stacks of images is not supported

Solution:

Split stacks of images into single images (SI) and apply SRGAN ML models to them (SISR)

Problem:

The transformed image is the same or lower quality than the original image, image artifacts are visible

Possible reason:

Lossy file formats, such as JPEG, employ compression algorithms to minimize the size of the file. This results in the loss of image data and produces compression artifacts

Solution:

Use lossless data compression formats, like lossless TIFF and PNG

Problem:

Transformed images are not synchronized between devices

Possible reason 1:

No Internet connection

Solution 1:

Make sure you are connected to the Internet

Possible reason 2:

Not signed with Apple ID

Solution 2:

Sign with Apple ID on all devices you are using CoLocalizer with

Possible reason 3:

iCloud is not set up and/or iCloud Drive is not turned on

Solution 3:

Set up iCloud Drive and/or turn on iCloud Drive on all devices you are using CoLocalizer with

Problem:

Synchronization of images between devices is very slow

Possible reason:

Internet connection is insufficient. The image is too big

Solution:

Make sure the speed of your Internet connection is adequate

Problem:

Deleted images on one device disappear on other devices

Possible reason:

This is an expected pattern

Solution:

To delete images only on one of your devices, sign off from iCloud on that device. Keep in mind that this action will cancel all synchronization steps

Step 4 (comparing results)

Problem:

The total pixel count according to the selected channel pair of transformed images increased less than three times compared to the original ones

Possible reason:

This is possible because image transformation can change the color values of pixels in the images

Solution:

Nothing needs to be done

Problem:

Cannot revert to the original image

Possible reason:

This is possible if the app did not save a transformed image automatically

Solution:

Make sure to save the image after opening. The app will take care of the rest

Step 5 (exporting data)

Problem:

Not all images have been exported

Possible reason:

Certain image exporting options in the export sheet were left unchecked

Solution:

Make sure to check boxes for all the desired options

The time to complete the protocol will depend on the size of the original image files.

Steps 1 and 2, launching CoLocalizer app and opening the image file either on a desktop or a tablet: 1 min

Step 3, applying SRGAN ML models: 10-30 min per average (1024 x 1024 pixels) image

Step 4, comparing results: 1-3 min per image

Step 5, exporting data: 1-3 min per image

The application of SRGAN ML models should result in a noticeable visual improvement by revealing more details in the images (Figure 6). These changes will be more pronounced in conventional images. The extent of improvement, as well as the presence of artifacts, will depend on the image quality – the higher is the quality of the original image the better and more reliably it can be transformed. Original super-resolution images will be likely visually unchanged.

Technically, the image files will become significantly bigger and their pixel dimensions will increase approximately three times.

1. Pawley, J. Handbook of biological confocal microscopy. 3 edn, (Springer, Boston, MA, 2006).

2. Demmerle, J., Wegel, E., Schermelleh, L. & Dobbie, I. M. Assessing resolution in super-resolution imaging. Methods 88, 3-10, doi.org/10.1016/j.ymeth.2015.07.001 (2015).

3. Schermelleh, L. et al. Super-resolution microscopy demystified. Nat Cell Biol 21, 72-84, doi:10.1038/s41556-018-0251-8 (2019).

4. Culley, S. et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat Methods 15, 263-266, doi:10.1038/nmeth.4605 (2018).

5. Culley, S. et al. Author Correction: Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat Methods 17, 1167, doi:10.1038/s41592-020-00983-7 (2020).

6. Rivenson, Y. G., Z. Günaydin, H. Zhang, Y. Wang, H. Ozcan, A. Deep learning microscopy. Optica 4, 1437-1443 (2017).

7. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat Methods 15, 1090-1097, doi:10.1038/s41592-018-0216-7 (2018).

8. Zinchuk, V. & Grossenbacher-Zinchuk, O. Machine learning for analysis of microscopy images: a practical guide. Curr Protoc Cell Biol 86, e101, doi:10.1002/cpcb.101 (2020).

9. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436-444, doi:10.1038/nature14539 (2015).

10. Goodfellow, I. J. P.-A., J. Mirza, M. Xu, B. Warde-Farley, D. Ozair, S. Courville, A. Bengio, Y. Generative adversarial nets. in Adv Neur Inf Proc Syst. Vol. 3 2672-2680.

11. Wu, Y. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light Sci Appl 8, 25, doi:10.1038/s41377-019-0139-9 (2019).

12. Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat Biomed Eng 3, 466-477, doi:10.1038/s41551-019-0362-y (2019).

13. Choi, G. et al. Cycle-consistent deep learning approach to coherent noise reduction in optical diffraction tomography. Opt Express 27, 4927-4943, doi:10.1364/OE.27.004927 (2019).

14. Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat Methods 16, 103-110, doi:10.1038/s41592-018-0239-0 (2019).

15. Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat Biotechnol 36, 460-468, doi:10.1038/nbt.4106 (2018).

16. Uzunova, H., Ehrhardt, J. & Handels, H. Memory-efficient GAN-based domain translation of high resolution 3D medical images. Comput Med Imaging Graph 86, 101801, doi:10.1016/j.compmedimag.2020.101801 (2020).

17. Zinchuk, V. & Grossenbacher-Zinchuk, O. Mobile quantitative colocalization analysis of fluorescence microscopy images. Curr Protoc Cell Biol 78, 4 37 31-34 37 12, doi:10.1002/cpcb.43 (2018).

18. Zinchuk, V., Wu, Y., Grossenbacher-Zinchuk, O. & Stefani, E. Quantifying spatial correlations of fluorescent markers using enhanced background reduction with protein proximity index and correlation coefficient estimations. Nat Protoc 6, 1554-1567, doi:10.1038/nprot.2011.384 (2011).

19. Zinchuk, V. & Grossenbacher-Zinchuk, O. Quantitative colocalization analysis of fluorescence microscopy images. Curr Protoc Cell Biol 62, Unit 4 19 11-14, doi:10.1002/0471143030.cb0419s62 (2014).

Training image datasets for creating the models were taken from publicly available databases by Weigert et al.: https://publications.mpi-cbg.de/publications-sites/7207/ and Zhang at al.: https://arxiv.org/abs/1812.10366. Test images were taken from a publicly available database provided by Cell Image Library: http://cellimagelibrary.org/home.

Download PDF

Version 1

posted

You are reading this latest protocol version

Protocol for applying Machine Learning models for the transformation of conventional fluorescence images to super-resolution

Status:

Version 1

Abstract

Figures

Introduction

Equipment

Procedure

Troubleshooting

Time Taken

Anticipated Results

References

Acknowledgements

Associated Publications

Status:

Version 1

Privacy Policy

Terms of Service

Protocol for applying Machine Learning models for the transformation of conventional fluorescence images to super-resolution

Status:

Version 1

Abstract

Figures

Introduction

Equipment

Procedure

Troubleshooting

Time Taken

Anticipated Results

References

Acknowledgements

Associated Publications

Status:

Version 1

Privacy Policy

Terms of Service

Manage Cookie Preferences