fix inference_step

[Fireblossom]

inference_step passes inference=True to model_engine.
However, the __forward__ of the Magma model does not accept this parameter, which will cause an error during training.
I fix it by simply copying the inference code from example_inference.py.

[CoEich]

I don't like the hardcoded instruction here, since not all images are paintings. For the purpose of this codepath I would prefer not to use any instruction.

[Fireblossom]

I write this just because the example :)

[CoEich]

I think 6 sampling steps might be a bit short in general.

[Fireblossom]

I would suggest putting hardcoded asks, max_steps, temperature, and top_k into config.py as a dictionary.
do you think it would be better?

[CoEich]

Hi,

thanks for your efforts to contribute to the open source MAGMA code. The codepath for inference during training should indeed be fixed and your help is appreciated :-)

In addition to the comments I added, I would in general prefer not to overload the forward function of the model too much. Maybe you could try to just change the inference_step method to invoke model.generate instead of changing the forward pass, see

magma/magma/train_loop.py

Line 85 in 4d01e51

def inference_step(config, eval_loader, model_engine):

Thanks again and let me know what you think.

Best,

Constantin

[Fireblossom]

Hi Constantin,

thank you for your advice. I was going to do the same.

But in practice, I found that deepspeed's model_engine cannot call methods other than forward.
(I'm a beginner in deepspeed, as this is my first time with it, so please point out if I'm wrong)
But if I call the model directly, it may lead to some unexpected errors, like device mismatch.

For the above reasons, I had to add the code into forward.

Best,

Changxu

[CoEich]

Ok, I think you can just access the model by model_engine.model, not 100% sure if that always works but maybe give it a try.

Best,

Constantin