Chainner integration

[zelenooki87]

Hi, @mahmoudnafifi
I'm very impressed with your project! I asked the creators of the Chainner application to integrate your models, but unfortunately, they're NumPy models (.npy) if I understand correctly. Would there be any chance you could convert them to PyTorch (.pth) or ONNX format? Thank you very much and best regards, sir.

[zelenooki87]

@mahmoudnafifi
I have converted gpu+ model in matlab to 3 files: S_net.onnx, T_net.onnx, AWB_net.onnx
But I do not know how to merge them and use together. please someone suggest?

[zelenooki87]

@mahmoudnafifi
Could you please answer to me?
This code extract 3 files instead of 1
load(fullfile('models','WB_model+_gpu.mat'));
% Konverzija AWB_net
exportONNXNetwork(net.AWB_net, 'AWB_net.onnx');

% Konverzija T_net
exportONNXNetwork(net.T_net, 'T_net.onnx');

% Konverzija S_net
exportONNXNetwork(net.S_net, 'S_net.onnx');

[mahmoudnafifi]

Hi,

Thank you for your interest in the work.

First, I would like to emphasize that this work and the deep white balance work are patented, and the source code license only allows for academic use. Therefore, if you would like to integrate this work or deep white balance into Chainner, you need to obtain written permission from Samsung (for deep white balance) and from York University/Adobe for this project.

Second, I believe the paper and the Python version of the code for this project and deep white balance clearly show how the models are used. Please let me know if you still have any questions after reviewing the code.

Thanks!

[zelenooki87]

Thank you for your answer. I am a "regular" user, an individual.
It would be really interesting for me to convert the model to ONNX because it works quite well with video files and I could use the model, for example, in a Vapoursynth plugin for processing video clips.
And it has proven even better for photos, so I even suggested to the authors of the chainner that they integrate it into their project.
As the chainner integration is not an option due licensing terms could you help me to convert model to onnx, I would contact you via email.
Thank you very much!

[mahmoudnafifi]

For non-commercial use, you can use this code, especially for academic purposes. If you are referring to this project (not the deep white balance editing work), there is no need to convert to ONNX because it is a simple KNN method, first histogram of input image is computed then encoded by a PCA matrix and then nearest neighbour search is performed on training samples (that what is stored in mat models in the matlab code). The polynomial correction of nearest examples are then used to fix the input image.

[zelenooki87]

Certainly for personal/non-commercial purposes. I understand how to use WB_SRGB with MATLAB or Python, but I gave an example for ONNX conversion because it would really mean a lot to me. The Vapoursynth plugin vsmlrt supports almost all ONNX format models, and when I extracted the models, all three work through the vsmlrt plugin for Vapoursynth, but they give different results and I can't figure out how to transfer them to one file so that it gives identical results as the basic, upgraded WB_SRGB model. I also tried using Python and the npy library to convert to one ONNX file but I can't seem to succeed. If you can help me with the ONNX format because I need it the most for video processing, because I would use it with the vsmlrt plugin and plugins like scunet for denoise, so the temporal changes in the video image probably wouldn't be so noticeable. I would be very grateful to you. And as for deep white balance, there are already cpp+ ONNX ports but I don't know if it's up to me but I get much better results with WB_SRGB. Best regards

[zelenooki87]

there is no need to convert to ONNX because it is a simple KNN method, first histogram of input image is computed then encoded by a PCA matrix and then nearest neighbour search is performed on training samples (that what is stored in mat models in the matlab code). The polynomial correction of nearest examples are then used to fix the input image.

If I understood you correctly, even if I extracted the weights, i.e., the model, I would not get an identical image as with preprocessing and additional code. So I wouldn't be able to use it as a generic model as I intended. Too bad... Is it possible to somehow add a function to MATLAB that works with video files, extracts images using FFmpeg, and then saves the output video file? I'm really new to the MATLAB environment and everything is "unknown" to me.