ALTO export

[cneud]

The way the NER tags are currently exported in ALTO creates considerable overhead in terms of file size, parsing etc. as every occurrence of a named entity will result in a separate NamedEntityTag and TAGREF.

Consider, for example, the following text in ALTO:

<TextLine HEIGHT="71" HPOS="309" VPOS="1004" WIDTH="779">
    <String CONTENT="Berlin" [...] />
    <String CONTENT="is" [...] />
    <String CONTENT="a" [...] />
    <String CONTENT="city" [...] />
    <String CONTENT="but" [...] />
    <String CONTENT="Berlin" [...] />
    <String CONTENT="is" [...] />
    <String CONTENT="also" [...] />
    <String CONTENT="a" [...] />
    <String CONTENT="state" [...] />
    <String CONTENT="of" [...] />
    <String CONTENT="Germany" [...] />
    <String CONTENT="." [...] />
</TextLine>

Tagging this line will result in two TAGREFS and two NamedEntityTag elements for the named entity "Berlin" being added to the ALTO like this:

<TextLine HEIGHT="71" HPOS="309" VPOS="1004" WIDTH="779">
    <String CONTENT="Berlin" [...] TAGREFS="Tag15"/>
    <String CONTENT="is" [...] />
    <String CONTENT="a" [...] />
    <String CONTENT="city" [...] />
    <String CONTENT="but" [...] />
    <String CONTENT="Berlin" [...] TAGREFS="Tag16"/>
    <String CONTENT="is" [...] />
    <String CONTENT="also" [...] />
    <String CONTENT="a" [...] />
    <String CONTENT="state" [...] />
    <String CONTENT="of" [...] />
    <String CONTENT="Germany" TAGREFS="Tag17"/>
    <String CONTENT="." [...] />
</TextLine>
[...]
<NamedEntityTag DESCRIPTION="Berlin" ID="Tag15" LABEL="LOC"/>    
<NamedEntityTag DESCRIPTION="Berlin" ID="Tag16" LABEL="LOC"/>
<NamedEntityTag DESCRIPTION="Germany" ID="Tag17" LABEL="LOC"/>

It would be preferable not to repeat NamedEntityTag for identical references and instead write this like:

<TextLine HEIGHT="71" HPOS="309" VPOS="1004" WIDTH="779">
    <String CONTENT="Berlin" [...] TAGREFS="Tag15"/>
    <String CONTENT="is" [...] />
    <String CONTENT="a" [...] />
    <String CONTENT="city" [...] />
    <String CONTENT="but" [...] />
    <String CONTENT="Berlin" [...] TAGREFS="Tag15"/>
    <String CONTENT="is" [...] />
    <String CONTENT="also" [...] />
    <String CONTENT="a" [...] />
    <String CONTENT="state" [...] />
    <String CONTENT="of" [...] />
    <String CONTENT="Germany" TAGREFS="Tag16"/>
    <String CONTENT="." [...] />
</TextLine>
[...]
<NamedEntityTag DESCRIPTION="Berlin" ID="Tag15" LABEL="LOC"/>
<NamedEntityTag DESCRIPTION="Germany" ID="Tag16" LABEL="LOC"/>

[TuulaP]

Is there any plan if this is going to be implemented at some point? This would be handy also in our case.

[cneud]

This is definitely on our to-do list for 1st half of 2018, though I cannot make any promises yet as to when it will be implemented exactly.
PR welcome ;-)