Skip to content

Official PyTorch implementation of the "SN2N" -v0.3.5

License

Notifications You must be signed in to change notification settings

WeisongZhao/SN2N

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

65 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Github commit Github All Releases License paper releases

Twitter Twitter GitHub stars

SN2N

Self-inspired learning to denoise for live-cell super-resolution microscopy.
v0.3.5



This repository is for our developed self-inspired Noise2Noise (SN2N) learning-to-denoise engine, and it will be in continued development. It is distributed as accompanying software for publication: Liying Qu et al. Self-inspired learning to denoise for live-cell super-resolution microscopy, bioRxiv (2024). Please cite SN2N in your publications, if it helps your research.




Introduction | 🔧 Installation | 🚀 Overall| 🎨 Dataset | 💻 Training |Inference | 🚩 Execution | 🏆 Models |🌈 Resources


Introduction

Our SN2N is fully competitive with the supervised learning methods and overcomes the need for large dataset and clean ground-truth. First, we create a self-supervised data generation strategy based on super-resolution images' spatial redundancy, using a diagonal resampling step followed by a Fourier interpolation for single-frame Noise2Noise. Second, we have taken a step further by ushering in a self-constrained learning process to enhance the performance and data-efficiency. Finally, we provide a Patch2Patch data augmentation (random patch transformations in multiple dimensions) to further improve the data efficiency.

🔧 Installation

Tested platform

  • Python = 3.7.6, Pytorch = 1.12.0 (Win 10, 128 GB RAM, NVIDIA RTX 4090 24 GB, CUDA 11.6)

Dependencies

  • Python >= 3.6
  • PyTorch >= 1.10

Instruction

  1. Clone the repository.

    git clone https://github.com/WeisongZhao/SN2N.git
    cd SN2N    
  2. Create a virtual environment and install PyTorch and other dependencies. Please select the correct Pytorch version that matches your CUDA version from https://pytorch.org/get-started/previous-versions/.

    Users can set up the environment directly by installing the packages listed in the (requirements.txt) file. The packages required by the environment have also been uploaded to the requirements.

    $ conda create -n SN2N python=3.7.6
    $ conda activate SN2N
    
    $ pip install -r requirements.txt

🚀 Overall

We have two execution modes for SN2N: The first one is step by step, which involves (dataset generation, training, and then inference.) The second one is directly invoking (SN2Nexecute.)

We have provided two examples of denoising in 2D and 3D, along with the datasets used for training and validation of SN2N. The 2D dataset consists of synthenic microtubules, represented by 2048 x 2048 pixels with a 32.5 nm pixel size. The 2D dataset can be found at 'examples/denoising2D/data' path. The 3D dataset consists of outer mitochondrial membrane (OMM) network imaging of live COS-7 cells labeled with Tom20-mCherry on SD-SIM sysytem. The 3D dataset has a size of 101 * 1478 * 1137 pixels, with a pixel size of 38.23 nm, and is avaliable at https://zenodo.org/records/12697244/files/c12_SR_w1L-561_t1.tif?download=1.

🎨 Dataset

0. Percentile normalization for data under ultralow SNR (optional)

For the ultralow SNR data with ultrahigh baseline signal and a number of hot pixels, we adapted the routinely used percentile normalization before the data generation step to remove the smooth background or hot pixels.

There are two ways to execute percentile normalization. One option is to utilize the percentile normalization Fiji plugin, available for easy access: https://github.com/WeisongZhao/percentile_normalization.imagej. Users can directly remove the ultra-strong baseline signal before training using this ready-to-use plugin. Alternatively, users can employ the following Python scripts for percentile normalization.

---------import package--------
import tifffile
import numpy as np
from SN2N.utils import *

---------parameter------------
## pmin and pmax represent mapping the data range to a specified percentage range.
## In the most applications, the pmin and pmax were assigned as 0% and 99.999%.
## For ultralow SNR data, we set pmin and pmax as 20% and 99.9%, respectively.

pmin = 30
pmax = 99.8
imgpath = 'raw.tif'
save_path = 'raw_per.tif'

image_data = tifffile.imread(imgpath)

## 3D percentile normalization
try:
    [t, x, y] = image_data.shape
    for tt in range(t):
        image_data_single = image_data[tt, :, :] 
        image_data_single = np.squeeze(image_data_single)
        image_data_single1 = normalize_percentage(x = image_data_single, pmin = pmin, pmax = pmax, axis=None, clip=True, eps=1e-20, dtype=np.float32)
        image_data_single2 = 255*image_data_single1
        tifffile.imsave(save_path, image_data_single2.astype('uint8'))
        
## 2D percentile normalization        
except ValueError:
        image_data = tifffile.imread(imgpath)
        image_data = normalize_percentage(x = image_data, pmin = pmin, pmax = pmax, axis=None, clip=True, eps=1e-20, dtype=np.float32)
        image_data = 255*image_data
        tifffile.imsave(save_path, image_data)        
    
print("Data preprocessing is completed.")

1. Self-supervised data generation

Our SN2N is adaptable to both 2D (xy) and 3D (xyz) datasets. You can use your own data or our uploaded to generate 2D / 3D SN2N data pairs.

Users can run the script after customizing parameters in Script_SN2N_datagen_2D.py (Script_SN2N_datagen_3D.py) or run directly from the command line.

2D data generation

python Script_SN2N_datagen_2D.py --img_path "Users own path/data/raw_data" --P2Pmode "1" --P2Pup "1" --BAmode "1" --SWsize "64"        

3D data generation

python Script_SN2N_datagen_3D.py --img_path "Users own path/data/raw_data" --P2Pmode "1" --P2Pup "1" --BAmode "1" --SWsize "64"    

Parameters instructions

The key parameters for 2D data generation and 3D data generation are consistent. There are also other parameters that do not require user modification. Detailed explanations can be found in the SN2N.get_options.datagen2D / SN2N.get_options.datagen3D function.

    -----Parameters------
    =====Important==========
    img_path:
        Path of raw images to train.
    P2Pmode(0 ~ 3):
        Augmentation mode for Patch2Patch.
        0: NONE; 
        1: Direct interchange in t;
        2: Interchange in single frame;
        3: Interchange in multiple frame but in different regions;
        {default: 0}
    P2Pup:
        Increase the dataset to its (1 + P2Pup) times size.
        {default: 0}
    BAmode(0 ~ 2):
        Basic augmentation mode.
        0: NONE; 
        1: double the dataset with random rotate&flip;
        2: eightfold the dataset with random rotate&flip;
        {default: 0} 
    SWsize:
        Interval pixel of sliding window for generating image pathes.
        {default: 64}
        
    ======Other parameters do not require modification; ======
    ======for details, refer to SN2N.get_options.========
    

💻 Training

1. Prepare the data

You can use SN2N data pairs for SN2N learning

2. Start training

Users can run the script after customizing parameters in Script_SN2N_trainer_2D.py (Script_SN2N_trainer_3D.py) or run directly from the command line.

2D data training

python Script_SN2N_trainer_2D.py --img_path "Users own path/data/raw_data" --sn2n_loss "1" --bs "32" --lr "2e-4" --epochs "100"

3D data training

python Script_SN2N_trainer_3D.py --img_path "Users own path/data/raw_data" --sn2n_loss "1" --bs "4" --lr "2e-4" --epochs "100" 

Parameters instructions

The key parameters for 2D trainer and 3D trainer are consistent. There are also other parameters that do not require user modification. Detailed explanations can be found in the SN2N.get_options.trainer2D/ SN2N.get_options.trainer3D function.

    -----Parameters------
    =====Important==========
    img_path:
        Path of raw images to train.
    sn2n_loss:
        Weight of self-constrained loss.
        {default: 1}
    bs:
        Training batch size.
        {default: 32}
    lr:
        Learning rate
        {default: 2e-4}.
    epochs:
        Total number of training epochs.
        {default: 100}.
        
    ======Other parameters do not require modification; ======
    ======for details, refer to SN2N.get_options.========
    

⚡ Inference

1. Prepare models

Before inference, you should have trained your own model

2. Test models

Users can run the script after customizing parameters in Script_SN2N_inference_2D.py (Script_SN2N_inference_3D.py) or run directly from the command line.

2D inference

python Script_SN2N_inference_2D.py --img_path "Users own path/data/raw_data" --model_path "Users own path/data/model" --infer_mode "0"

3D inference

In 3D prediction tasks, we use the method of stitching predictions to avoid issues of memory overflow

python Script_SN2N_inference_3D.py --img_path "Users own path/data/raw_data" --model_path "Users own path/data/model" --infer_mode "0" --overlap_shape "2,256,256"

Parameters instructions

The key parameters for 2D inference and 3D inference are nearly consistent except for'overlap_shape'. There are also other parameters that do not require user modification. Detailed explanations can be found in the SN2N.get_options.Predict2D/ SN2N.get_options.Predict3D function.

    -----2D/3D inference Parameters------
    img_path:
        Path of raw images to inference
    model_path:
        Path of model for inference
    infer_mode:
        Prediction Mode
        0: Predict the results of all models generated during training 
        under the default "models" directory on the img_path.                
        1: Predict the results of the models provided by the user under 
        the given model_path on the Img_path provided by the user.
        
    -----3D execution unique Parameters------
    overlap_shape:
        Overlap shape in 3D stitching prediction.
        {default: '2, 256, 256'}    

🚩 Execution

1. Self-supervised data generation

Our SN2N denoiser has been integrated into the SN2Nexecute.py function, allowing users to input their allowing users to input their own parameters for denoising with just one click.

Users can run the script after customizing parameters in Script_SN2Nexecute_2D.py (Script_SN2Nexecute_3D.py).

2D execution

python Script_SN2Nexecute_2D.py --img_path "Users own path/data/raw_data" --P2Pmode "1" --P2Pup "1" --BAmode "1" --SWsize "64" --sn2n_loss "1" --bs "32" --lr "2e-4" --epochs "100" --model_path "Users own path/data/model" --infer_mode "0"

3D execution

python Script_SN2Nexecute_3D.py --img_path "Users own path/data/raw_data" --P2Pmode "1" --P2Pup "1" --BAmode "1" --SWsize "64" --sn2n_loss "1" --bs "32" --lr "2e-4" --epochs "100" --model_path "Users own path/data/model" --infer_mode "0" --overlap_shape "2,256,256"

Parameters instructions

The key parameters for 2D execute and 3D execution are nearly consistent execept for 'overlap_shape'. There are also other parameters that do not require user modification. Detailed explanations can be found in the SN2N.get_options.execute2D / SN2N.get_options.execute 3D function.

    -----2D/3D execution Parameters------
    =====Important==========
    img_path:
        Path of raw images to train.
    P2Pmode(0 ~ 3):
        Augmentation mode for Patch2Patch.
        0: NONE; 
        1: Direct interchange in t;
        2: Interchange in single frame;
        3: Interchange in multiple frame but in different regions;
        {default: 0}
    P2Pup:
        Increase the dataset to its (1 + P2Pup) times size.
        {default: 0}
    BAmode(0 ~ 2):
        Basic augmentation mode.
        0: NONE; 
        1: double the dataset with random rotate&flip;
        2: eightfold the dataset with random rotate&flip;
        {default: 0} 
    SWsize:
        Interval pixel of sliding window for generating image pathes.
        {default: 64}
        sn2n_loss:
        Weight of self-constrained loss.
        {default: 1}
    bs:
        Training batch size.
        {default: 32}
    lr:
        Learning rate
        {default: 2e-4}.
    epochs:
        Total number of training epochs.
        {default: 100}.
    model_path:
        Path of model for inference
    infer_mode:
        Prediction Mode
        0: Predict the results of all models generated during training 
        under the default "models" directory on the img_path.                
        1: Predict the results of the models provided by the user under 
        the given model_path on the Img_path provided by the user.
        
    -----3D execution unique Parameters------
    overlap_shape:
        Overlap shape in 3D stitching prediction.
        {default: '2, 256, 256'} 
        
    ======Other parameters do not require modification; ======
    ======for details, refer to SN2N.get_options.========
    

🏆 Models

We have provided 12 generalized pre-trained models for specfic tasks and hope this will be helpful to the community.

No. Data Microscopy system Pixel size Network
1 (Fig. 2l) Fixed COS-7 cells of lysosomes labeled with LAMP1-EGFP. SD-SIM 32.5 nm 2D U-Net
2 (Fig. 2m) Fixed COS-7 cells of mitochondria labeled with Tom20-mGold1. SD-SIM 32.5 nm 2D U-Net
3 (Fig. 4b) Live COS-7 cells of mitochondria labeled with Tom20–mCherry. SD-SIM 38.23 nm 3D U-Net
4 (Fig. 4e) Live COS-7 cells of mitochondria labeled with Tom20–mCherry. SD-SIM 38.23 nm 3D U-Net
5 (Fig. 4g) Live COS-7 cells, mitochondria labeled with mGold-Mito-N-7, ER labeled with DsRed-ER, and the nucleus labeled with SPY650-DNA. SD-SIM 38.23 nm 3D U-Net
6 (Fig. 5g) Live COS-7 cells, MT labeled with SiR-Tubulin, Actin labeled with Lifeact-EGFP, and the ER labeled with Sec61β–EGFP. STED 15 nm 2D U-Net
7 (Extended Data Fig. 6a) Live COS-7 cells, ER labeled with Hoechst and the mitochondira labeled with MitoTracker Deep Red. EMCCD SD-SIM 94 nm 3D U-Net

Version

  • v0.3.1 add examples for both 2D denoising and 3D denoising, and integrate them into the SN2Nexecute function.
  • v0.2.8 reorder the core code
  • v0.1.0 initial version

🌈 Resources:

Open source SN2N

This software and corresponding methods can only be used for non-commercial use, and they are under Open Data Commons Open Database License v1.0.

About

Official PyTorch implementation of the "SN2N" -v0.3.5

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages