"nodes-to-exclude" in quantization doesnt work

[marziye-A]

hi
why "nodes-to-exclude" in quantization doesn't work?
my code is as below:

from optimum.onnxruntime.configuration import QuantizationConfig

dqconfig = QuantizationConfig(is_static=False,format=QuantFormat.QDQ,mode = QuantizationMode.IntegerOps, per_channel=True ,weights_dtype = QuantType.QUInt8,nodes_to_exclude=["Wav2Vec2FeatureEncoder","Wav2Vec2FeatureProjection","Wav2Vec2EncoderStableLayerNorm"])

does any one know the reason?
any help is really appreciated!

[baskrahmer]

Hi @marziye-A

I took a look and I think the issue is that your nodes_to_exclude list does not correspond to valid ONNX graph nodes but instead to model submodules. You can run the below code to get all ONNX nodes for a model:

import tempfile
from pathlib import Path

from onnx import load as onnx_load
from optimum.onnxruntime import ORTModelForAudioClassification, ORTQuantizer
from optimum.onnxruntime.configuration import QuantizationConfig, QuantFormat, QuantizationMode, QuantType

qconfig = QuantizationConfig(
    is_static=False,
    format=QuantFormat.QDQ,
    mode=QuantizationMode.IntegerOps,
    per_channel=True,
    weights_dtype=QuantType.QUInt8,
    nodes_to_exclude=["/wav2vec2/feature_extractor/conv_layers.0/conv/Conv"]  # <-- Node from ONNX graph
)

with tempfile.TemporaryDirectory() as tmp_dir:
    output_dir = Path(tmp_dir)
    model = ORTModelForAudioClassification.from_pretrained("hf-internal-testing/tiny-random-wav2vec2", export=True)

    quantizer = ORTQuantizer.from_pretrained(model)
    quantizer.quantize(
        save_dir=output_dir,
        quantization_config=qconfig,
    )
    quantized_model = onnx_load(output_dir.joinpath("model_quantized.onnx"))
    node_list = [node.name for node in quantized_model.graph.node]

    print(node_list)

I don't know what your exact use-case is, but if you want to exclude the Wav2Vec2FeatureEncoder graph from quantization, you would have to find the corresponding ONNX nodes for those computations and add them to the nodes_to_exclude argument. You can use a regex to filter those nodes from the full list or maybe implement something clever to get all nodes of the submodule ;)

[fxmarty]

Hi @marziye-A, apology for the late reply and thank you @baskrahmer for the correct answer!

I will improve the documentation in this regard.

An alternative to make it easier to exclude part of a model from quantization may be using export_modules_as_functions during the export as suggested here huggingface/transformers#26307 (comment)

[marziye-A]

thank you very much for your answer,
do we have any limitation in number of nodes that we can to exclude?

[baskrahmer]

do we have any limitation in number of nodes that we can to exclude?

As far as I know, you can add all nodes of the model.