Research Article

Keyphrase based Evaluation of Automatic Text Summarization

by  Fatma Elghannam, Tarek El-Shishtawy
journal cover
International Journal of Computer Applications
Foundation of Computer Science (FCS), NY, USA
Volume 117 - Issue 7
Published: May 2015
Authors: Fatma Elghannam, Tarek El-Shishtawy
10.5120/20564-2953
PDF

Fatma Elghannam, Tarek El-Shishtawy . Keyphrase based Evaluation of Automatic Text Summarization. International Journal of Computer Applications. 117, 7 (May 2015), 5-8. DOI=10.5120/20564-2953

                        @article{ 10.5120/20564-2953,
                        author  = { Fatma Elghannam,Tarek El-Shishtawy },
                        title   = { Keyphrase based Evaluation of Automatic Text Summarization },
                        journal = { International Journal of Computer Applications },
                        year    = { 2015 },
                        volume  = { 117 },
                        number  = { 7 },
                        pages   = { 5-8 },
                        doi     = { 10.5120/20564-2953 },
                        publisher = { Foundation of Computer Science (FCS), NY, USA }
                        }
                        %0 Journal Article
                        %D 2015
                        %A Fatma Elghannam
                        %A Tarek El-Shishtawy
                        %T Keyphrase based Evaluation of Automatic Text Summarization%T 
                        %J International Journal of Computer Applications
                        %V 117
                        %N 7
                        %P 5-8
                        %R 10.5120/20564-2953
                        %I Foundation of Computer Science (FCS), NY, USA
Abstract

The development of methods to deal with the informative contents of the text units in the matching process is a major challenge in automatic summary evaluation systems that use fixed n-gram matching. The limitation causes inaccurate matching between units in a peer and reference summaries. The present study introduces a new Keyphrase based Summary Evaluator (KpEval) for evaluating automatic summaries. The KpEval relies on the keyphrases since they convey the most important concepts of a text. In the evaluation process, the keyphrases are used in their lemma form as the matching text unit. The system was applied to evaluate different summaries of Arabic multi-document data set presented at TAC2011. The results showed that the new evaluation technique correlates well with the known evaluation systems: Rouge-1, Rouge-2, Rouge-SU4, and AutoSummENG–MeMoG. KpEval has the strongest correlation with AutoSummENG–MeMoG, Pearson and spearman correlation coefficient measures are 0. 8840, 0. 9667 respectively.

References
  • El-Shishtawy, T. , & El-Ghannam, F. 2012. An Accurate Arabic Root-Based Lemmatizer for Information Retrieval Purposes. International Journal of Computer Science Issues (IJCSI), 9(1). ?
  • El-Shishtawy, T. , & El-Ghannam, F. 2012, May. Keyphrase based Arabic summarizer (KPAS). Informatics and Systems (INFOS), 2012 8th International Conference. (pp. NLP-7. IEEE). ?
  • Giannakopoulos, G. , Karkaletsis, V. , Vouros, G. , & Stamatopoulos, P. 2008. Summarization system evaluation revisited: N-gram graphs. ACM Transactions on Speech and Language Processing (TSLP), 5(3), 5. ?
  • Hovy, E. , Lin, C. Y. , & Zhou, L. 2005. Evaluating duc 2005 using basic elements. In Proceedings of DUC (Vol. 2005). ?
  • Hovy, E. , Lin, C. Y. , Zhou, L. , & Fukumoto, J. 2006, May. Automated summarization evaluation with basic elements. In Proceedings of the Fifth Conference on Language Resources and Evaluation (LREC 2006) (pp. 604-611). ?
  • Jones, K. S. , & Galliers, J. R. (Eds. ). 1996. Evaluating natural language processing systems: An analysis and review (Vol. 1083). Springer. ?
  • Lin, C. Y. 2004, July. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop (pp. 74-81). ?
  • Lin, C. Y. , & Hovy, E. 2003, May. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1 (pp. 71-78). Association for Computational Linguistics. ?
  • Papineni, K. , Roukos, S. , Ward, T. , & Zhu, W. J. 2002, July. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (pp. 311-318). Association for Computational Linguistics. ?
  • Tratz, S. , & Hovy, E. 2008. Summarization evaluation using transformed basic elements. In Proceedings of the 1st Text Analysis Conference. ?
Index Terms
Computer Science
Information Sciences
No index terms available.
Keywords

Evaluating automatic text summarization keyphrase-based summary evaluation Summarization keyphrase extraction Arabic summary evaluation.

Powered by PhDFocusTM