Add intersection of input, reference to overlap #1817
Conversation
andyzorigin
changed the title
| @@ -106,6 +107,20 @@ def create_ngram_index( | |||
| ngram_index[n][reference_ngram].add( | |||
| EntryDataOverlapKey(stats_key=stats_key, instance_id=id, part=PART_REF) | |||
| ) | |||
|
|
|||
| # compute intersection ngrams [defined as n-grams that occur between instance and reference] | |||
There was a problem hiding this comment.
Not clear what intersection means exactly or "between instance and reference"...can this be defined a bit more precisely in natural language?
There was a problem hiding this comment.
Yep the example I give in the PR description is:
input = ["is 2+2 4 true or false"] reference = ["true"], the intersection is the 5-gram ["4 true or false true"] (which is formed from the input 4-gram [4 true or false] and the reference 1-gram [true]) I'll reword the comment
| @@ -110,7 +110,7 @@ def create_ngram_index( | |||
|
|
|||
| # concatenate the last n-1 tokens of input and the first n-1 tokens of reference and compute n-grams on this "interesection token" sequence | |||
| # for instance: input = ["is 2+2 4 true or false"] reference = ["true"] | |||
| # the intersection is the 5-gram ["4 true or false true"] | |||
| # the intersection is the 5-gram ["4 true or false true"] | |||
There was a problem hiding this comment.
I think this is good (as we discussed), but (i) do we have to get providers to rerun with this new code (non-trivial cost) and (ii) I wonder how often the question and answer will be juxtaposed. If there's any token that separates the Q and A, then we won't detect overlap.
There was a problem hiding this comment.
Yeah I think for now we can just run this for the pile and see what sort of insights we get; this is simply an additional metric that doesn't affect existing metrics. If we're concerned for a token gap, we can allow for a skip token budget, esp for this metric, though it may be premature to add at this point
|
Closing stale PR. Not deleting the branch, in case it is still being used. |
Previously, we computed ngrams for input and references, but did not calculate ngrams for those on the boundary of input and reference. e.g. input = ["is 2+2 4 true or false"] reference = ["true"], the 5-gram ["4 true or false true"] is not captured; this PR adds this in