Building a golden collection of parallel Multi-Language Word Alignment

Transcrição

1 Building a golden collection of parallel Multi-Language Word Alignment João Graça, Joana Paulo Pardal, Luísa Coheur and Diamantino Caseiro

2 Resume Needed resources for Machine Translation Manual Word Alignments for Portuguese Comparable Word Alignments between different languages

3 Resume Needed resources for Machine Translation Manual Word Alignments for Portuguese Comparable Word Alignments between different languages Created and made available 100 sentences manually aligned between 6 language pairs Detailed guidelines for Multi-language Word Alignment

4 Outline Manual Word Alignment Corpus Alignment Process Evaluation Future Work

5 Manual Word Alignment i 1 did 2 receive 3 the 4 request 5 you 6 sent. dirigiu me que pedido 7 me 8. de recebi o facto

6 Corpus European Parliament proceedings (Koehn 2006) First 100 sentence from common test set ( ) Sentences 100 English Portuguese French Spanish Words Types Avg. Size

7 Alignment Process Team Four annotators (h 1, h 2, h 3, h 4 ) Proficient speakers of annotated languages

8 Alignment Process Team Four annotators (h 1, h 2, h 3, h 4 ) Proficient speakers of annotated languages Starting point Guidelines for Spanish-English (EPPS) (Mariño 2005)

9 Annotation Procedure Part I EN-PT h 1 EN-ES h 1 EN-FR h 2 PT-ES h 1 PT-FR h 2 ES-FR h 2 New alignment guidelines created Annotated all languages a sentence at a time Leave guidelines as unambiguous as possible

10 Annotation Procedure Part II EN-PT h 1 h 1 &h 3 EN-ES h 1 h 1 &h 4 EN-FR h 2 h 2 &h 3 PT-ES h 1 h 1 &h 4 PT-FR h 2 h 2 &h 4 ES-FR h 2 h 2 &h 4 First contact of new annotators with new guidelines First evaluation of the guidelines Disagreements were discussed and guidelines updated

11 Annotation Procedure Part III EN-PT h 1 h 1 &h 3 h 1 &h 3 EN-ES h 1 h 1 &h 4 h 1 &h 4 EN-FR h 2 h 2 &h 3 h 2 &h 3 PT-ES h 1 h 1 &h 4 h 1 &h 4 PT-FR h 2 h 2 &h 4 h 2 &h 4 ES-FR h 2 h 2 &h 4 h 2 &h 4 Final evaluation of the guidelines Guidelines refinement

12 Annotation Procedure Part IV EN-PT h 1 h 1 &h 3 h 1 &h 3 h 1 h 3 EN-ES h 1 h 1 &h 4 h 1 &h 4 h 1 h 4 EN-FR h 2 h 2 &h 3 h 2 &h 3 h 2 h 3 PT-ES h 1 h 1 &h 4 h 1 &h 4 h 1 h 4 PT-FR h 2 h 2 &h 4 h 2 &h 4 h 2 h 4 ES-FR h 2 h 2 &h 4 h 2 &h 4 h 2 h 4 All sentences were reviewed one at a time for all languages Current version of the guidelines

13 Evaluation Structures Output S1 1 S2 2 S3 T1 T2T S1 1 S2 2 S3 T1 T2T3

14 Evaluation Structures Output How to count misaligned points S1 1 S2 2 S S1 1 S2 2 S3 T1 T2T3 T1 T2T3 I 1 2 A1 + A2 50 %

15 Evaluation Structures Output How to count misaligned points S1 1 S2 2 S S1 1 S2 2 S3 T1 T2T3 T1 T2T3 2 I 1 2 A1 + A2 66 % Melamed (1998), Kruijff-Korbayová (2006)

16 Evaluation Structures Output How to count misaligned points Sure and Possible Alignment S1 1 S2 2 S3 T1 T2T S1 1 S2 2 S3 T1 T2T3

17 Inter Annotator Agreement Distinguish between Sure and Possible alignment mistakes Strong Agreement - 2 ( I s s + I p p ) A 1 + A 2 Weak Agreement - 2 ( I p s + I s p ) A 1 + A 2 Weak Disagreement - 2 ( I p 0 + I 0 p ) A 1 + A 2 Strong Disagreement - 2 ( I s 0 + I 0 s ) A 1 + A 2

18 First Evaluation: A 1 A 2 SA WA WD SD EN-PT (h 1,h 4 ) EN-ES (h 1,h 3 ) EN-FR (h 2,h 4 ) PT-ES (h 1,h 3 ) PT-FR (h 2,h 4 ) ES-FR (h 2,h 3 ) Average Undifferentiated Average Errors due to different interpretation of the guidelines

19 Second Evaluation: A 1 A 2 SA WA WD SD EN-PT (h 1,h 4 ) EN-ES (h 1,h 3 ) EN-FR (h 2,h 4 ) PT-ES (h 1,h 3 ) PT-FR (h 2,h 4 ) ES-FR (h 2,h 3 ) Average Undifferentiated Average

20 Future Work Increase each alignment set External contributions Add data from new domains Add other types of annotations

21