Problem in building and synthesizing resnet50 for Alveo U280

[Adib6637]

Hi,

I'm currently try to synthesis resnet50 model for Alveo U280. I created the model using brevitas and qonnx library (from dev and main brunch, I merge them ) and based on resnet18, resnet50 (already in qonnx format) and resnet 50 from pytorch. Both of the are from finn example. The synthesis of resnet50 (from the example) process for Alveo U280 is successful but not for my model with the same flow (with auto configuration for folding).
During synthesis, the intermediate model (model after cleanup and streamline) look different from the intermediate model during the synthesis process of the example model. For the blocks number >= 2 on each layer, the input for Relu (MultiTreshold) is change cause defect on the structure of the model ( as in image below ).

My questions are:

1. Is the notebook for explaining the process to create the resnet50 available?
2. What is the best method to compute FIFO depth so that I could try to apply it manually?
3. How can I add “add” to activation function as in the example? (as image below)
4. Is there any reasons why the position of the maxpool is change in the example compare to standard pytorch resnet50 model?
5. Why input for Relu (MultiTreshold) is change for my model but not for example model even though the model structure at that part is the same?

Link to my resnet code: resnet50

Thank you!

[fpjentzsch]

Hi,

1. I don't think the exact model/training code for the ResNet-50 from finn-examples is available. I can give you this piece of code that I used as a reference for building ResNets in the past. However, I don't have a full custom build flow matching this exact model definition either.
2. Computing optimal FIFO depths manually is difficult, especially for ResNets. I suggest that you let FINN's automatic FIFO sizing (auto_fifo_depths=True) run once for each folding config (SIMD/PE) and then set these FIFO depths manually in the folding config for further builds. This can save you time because FINN does not have to simulate the whole accelerator for every build. You can also try to set the auto_fifo_strategy to AutoFIFOSizingMethod.CHARACTERIZE for faster FIFO sizing.
3. You mean streamlining the Add node to absorb it into the MultiThreshold? The AbsorbSignBiasIntoMultiThreshold transformation will do this, but it should be called by default.
4. I don't know why the MaxPool might be in a different position. Maybe someone else can chime in?
5. What do you mean with "the input for the MultiThreshold changed"? Can you give an example as a screenshot or .onnx file?

I know @bwintermann is also working with ResNet-50 (at least the finn-example) on the Alveo U280. Maybe you can talk to him.
@iksnagreb is also working with ResNets, he can have a quick look over your code if he finds the time.

In general, there have been quite a few issues/discussions that deal with ResNet streamlining problems. Perhaps we should aggregate them into a list or guide. An end-to-end ResNet example (including model definition, training code, and pre-built finn-example) would also be helpful.

[Adib6637]

Thank you for the answers. Now Im able to solve the problem regarding the change in relus input. For building the model, which version/branch of brevitas are you using because there is no QuantAvgPool2d in main and dev branch?

[iksnagreb]

Hm, this seems to have been renamed to TruncAvgPool2d "recently" (see this commit). Either you stick to a brevitas version before this change to keep the name QuantAvgPool2d (if you stick to the official releases this should be any version before (not including) v0.9.0), or you keep using a more recent version and rename all occurrences in your code to TruncAvgPool2d. The model behavior should not be change much as I see this: Apparently the default changed from truncation to rounding, but I do not know the details/reasoning behind this change.

[iksnagreb]

Please check again, whether the code you have linked above is indeed the code you used to export the model shown above: It seems you are not using the shared quantizer for adding the residual branches here, but according to your comment here it seems like you are aware of this issue or even had a shared quantizer in some earlier version of your code? Your model, as it is right now, will not make it past the export to ONNX, due to RuntimeError: Scaling factors are different.

Could you then please share the exported ONNX model file (or at least some code I could use to quickly export a model with the right input sizes)? I need to see the whole model graph and the order of streamlining steps you are trying to apply to give you more detailed hints at what might be going wrong here.

Are you using the build.py and custom_steps.py from the resnet50 finn-examples repository? Or did you build your own custom build configuration? Which version/branch of finn and finn-examples are you using?

[Adib6637]

You are right, I didnt use the shared quantizer. I already change them accordingly. You can get the model here.
I usebuild.py and custom_steps.py from the resnet50 finn-examples repository. Thanks a lot

[fpjentzsch]

Maybe the "feature/rn-18" branch in finn-examples can also help you: https://github.com/Xilinx/finn-examples/tree/feature/rn-18/build/resnet18

It should contain a working FINN flow for the ResNet-18 example from Brevitas, but was not merged because the folding configuration is sub-optimal.