Repetition, retracing, reformulation correction

[JeltevanBoheemen]

In the current parses, the following CHAT annotated constructs are ignored:

• repetition [/]
• retracing [//]
• reformulation [///]

The following steps should be implemented and executed:

• identify parse files with these constructs. This is already done by @JanOdijk, paths are provided in error check Excel file.
• extract the part of the original utterance that is ignored
• parse the above part separately, using the same Alpino version as the original parse
• take everything under the top/top node, and prepend it to the original top/top node
• fix node attributes for new tree:
  • @begin and @end
    • count new number of leaf nodes
    • add the count to original nodes
  • @id
    • presumed implementation is a depth-first sequence 0...n
    • check this with existing files
    • fix reparses
  • add metadata concerning false starts. @JanOdijk developed a script for this purpose.

[JanOdijk]

The metadata can be added by using the function cleantext from the module cleanCHILDEStokens

def cleantext(utt: str, repkeep: bool, tokenoutput: bool = False) -> Tuple[CleanedText, Metadata]:

where utt is a string with CHAT-annotations (to be taken from the existing metadata associated with the parse tree), the output is a tuple consisting of a cleaned text (all CHAT-annotations applied) and a list of Metadata .

For the cases at hand, it will turn the CHAT codes [/] and [//] into appropriate metadata.

There is no need to run this specifically for this set, because metadata for all AnnCor utterances must be generated by using this function