HierarchicalModel can't be instantiated easily with AutoModel.from_pretrained

[angus-lherrou]

Since HierarchicalModel uses a separate config class for its encoder heads, it does not play nice with AutoModel.from_pretrained.

>>> model = AutoModel.from_pretrained(<path/to/pretrained/model>, config=<CnlpConfig>)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../site-packages/transformers/models/auto/auto_factory.py", line 463, in from_pretrained
    return model_class.from_pretrained(
  File ".../site-packages/transformers/modeling_utils.py", line 2156, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: HierarchicalModel.__init__() missing 1 required positional argument: 'transformer_head_config'

This config needs to be the same as it is in training, so it needs to be part of what gets exported when saving the model. The most straightforward approach would be to incorporate it into CnlpConfig as an optional dict attribute.

[angus-lherrou]

The ideal invocation is as above; as a workaround with the current implementation, we can manually construct and pass in a HierarchicalTransformerConfig object with the same parameters as the model (which we will just have to know):

>>> transformer_head_config = HierarchicalTransformerConfig(n_layers=2, d_model=256 ,d_inner=2048, n_head=8, d_k=8, d_v=96)
>>> model = AutoModel.from_pretrained(encoder_name, config=config, transformer_head_config=transformer_head_config)

[tmills]

is this done?