Greedy search and beam search

[KexinFeng]

Description

This PR succeeds PR #2547 and #2509. The model tracing is shown therein.

Benchmarked with huggingface transnformers' output.

Ref. https://huggingface.co/blog/how-to-generate

Demo output

In the demo TestLMSearch.java, we feed in batch sequence input, using right padding with the space token ' ' (id = 220).

["DeepMind Company is",  
 "Memories follow me left and right. I can"]

Output of beam search (numBeam=3, maxLength=50):

'DeepMind Company is      \xa0 a company that has been around for a long time and has been around for a long time. They have been around for a long time and have been around for a long time. They have been'
'DeepMind Company is      \xa0 a company that has been around for a long time. It has been around for a long time and has been around for a long time. It has been around for a long time and has been'
'DeepMind Company is      \xa0 a company that has been around for a long time and has been around for a long time. They have been around for a long time and have been around for a long time. They are a'

"Memories follow me left and right. I can't tell you how many times I've been told that I'm not a good person. I'm not a good person. I'm not a good person. I'm not a good person. I"
"Memories follow me left and right. I can't tell you how many times I've been told that I'm not a good person. I'm not a good person. I'm not a good person. I'm not a good person.\n"
'Memories follow me left and right. I can\'t tell you how many times I\'ve been told that I\'m not a good person. I\'m not a good person. I\'m not a good person. I\'m not a good person."\n'

Output of greedy search (maxLength=50):

'DeepMind Company is      \xa0 a company that has been around for over 20 years. We have been around for over 20 years and have been around for over 20 years. We have been around for over 20 years and have been'

"Memories follow me left and right. I can't remember the last time I saw a girl in a dress. I can't remember the last time I saw a girl in a dress. I can't remember the last time I saw a girl in"

Notes about the GPT2's behaviour with padding and attention mask:

This notes shows that for GPT2, the right padding and left padding can behave very differently, either with attention_mask or without. (The situation with the attention_mask is the most intuitive method). The reason is not totally interpretible yet. But this result will guide the next batching solution.

1. With attention mask which is set 0 on the padded tokens, and 1 everywhere else, for the input_ids that correpsonds to the above input (the space token ' ' id is 220).

A. Right padding:

input_ids = torch.tensor([[29744, 28478, 5834, 318, 220, 220, 220, 220, 220, 220],
                   [13579, 1749, 1061, 502, 1364, 290, 826, 13, 314, 460]])

Output:

"DeepMind Company is      \xa0 a company that has been around for a long time and has been around for a long time. They have been around for a long time and have been around for a long time. They have been"

"Memories follow me left and right. I can't tell you how many times I've been told that I'm not a good person. I'm not a good person. I'm not a good person. I'm not a good person. I"

B. Left padding

input_ids = torch.tensor([[220, 220, 220, 220, 220, 220, 29744, 28478, 5834, 318],
                   [13579, 1749, 1061, 502, 1364, 290, 826, 13, 314, 460]])

"      DeepMind Company is                                        "

"Memories follow me left and right. I can't tell you how many times I've been told that I'm not a good person. I'm not a good person. I'm not a good person. I'm not a good person. I"

2. Without any attention mask (i.e. all set to 1), for the input_ids that correpsonds to the above input (the space token ' ' id is 220).

A. Right padding
output:

"DeepMind Company is                                                                                  "

"Memories follow me left and right. I can't tell you how many times I've been told that I'm not a good person. I'm not a good person. I'm not a good person. I'm not a good person. I"

B. Left padding
output =

'      DeepMind Company is a subsidiary of DeepMind Technologies, Inc. DeepMind Technologies, Inc. is a subsidiary of DeepMind Technologies, Inc. is a subsidiary of DeepMind Technologies, Inc. is a subsidiary of Deep'

Memories follow me left and right. I can't tell you how many times I've been told that I'm not a good person. I'm not a good person. I'm not a good person. I'm not a good person. I.

[KexinFeng]

Merged in #2637