Inter-annotator agreement in CoNLL-2012

Main references:

  • 2011: Pradhan et al. (2011)[1]
  • 2012: Pradhan et al. (2012)[2]

The data can be downloaded from (2011) and (2012).

Notice! The data is not ready after downloaded. Follow the instructions to create *.v2_auto_conll and *.v2_gold_conll files.
Difference between .*_auto_conll and .*_gold_conll: from Xiao Cheng:
auto_conll uses parser's parse tree and gold_conll uses human annotated parse tree. Both have the same gold mentions ( thus auto_conll might have some gold mention not being a NP due to parser error )
About singletons: from Durrett and Klein (2013)[3]: "Singletons are always removed before evaluation because the OntoNotes corpus does not annotate them" (CoNLL-2011 and 2012 are part of OntoNotes).

References Edit

  1. Pradhan, S., Ramshaw, L., Marcus, M., Palmer, M., Weischedel, R., & Xue, N. (2011). CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task (pp. 1–27). Association for Computational Linguistics.
  2. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., Zhang, Y.: CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In: Joint Con- ference on EMNLP and CoNLL-Shared Task, pp. 1–40. Association for Computa- tional Linguistics (2012)
  3. Durrett, G., & Klein, D. (2013). Easy victories and uphill battles in coreference resolution. EMNLP ’13, (October), 1971–1982.