Abstract

News search has now become an important internet activity as users are switching from hard copies to online news reading. Many modern news search engines like: Google News or Bing News are available for this purpose. We propose a theoretical framework for evaluating the retrieval effectiveness of news search systems. The framework exploits supervised machine learning approach for evaluating therefore we performed retrieval effectiveness tests on a small data set consisting relevancy features- Tfidf and Latent Semantic Indexing (LSI) as well as freshness feature-publication time, extracted from 1120 query-document pairs collected from search results of Google News, to evaluate the performance of various machine learned learning to rank algorithms on NDCG and ERR metric at different cut-offs. The motive behind this work is to conduct large-scale retrieval effectiveness studies for news search engines.

References

• Lewandowski, D., 2013. Evaluating the retrieval effectiveness of web search engines using a representative query sample. Journal of the Association for Information Science and Technology, Vol 66, issue 9, pages-1763-1775.
• Can, F., Nuray, R. and Sevdic, A. B., 2003. Automatic performance evaluation of Web search engines. Information Processing and Management 40 (2004) 495–514.
• Lewandowski, D., (2008) "The retrieval effectiveness of web search engines: considering results descriptions", Journal of Documentation, Vol. 64 Issue: 6, pp.915-937, https://doi.org/10.1108/00220410810912451
• Ali, R and Beg, M. M. S., 2011. An overview of Web search evaluation methods. Computers and Electrical Engineering. 37 (2011) 835–848.
• Leighton, H. V. and Srivastava, J., 1999. First 20 Precision among World Wide Web Search Services (Search Engines). Journal of the American Society for Information Science. 50(10):870–881.
• Clough, P. And Sanderson, M., 2013. Evaluating the performance of information retrieval systems using test collections. Information research. Vol 18(2).
• Vaughan, L., 2004. New measurements for search engine evaluation proposed and tested. Information Processing and Management 40 (2004) 677–691.
• Ali, R. and Beg, M., M. S., 2009. Modified rough set based aggregation for effective evaluation of web search engines. Fuzzy Information Processing Society, 2009. NAFIPS 2009. Annual Meeting of the North American.
• Bokhari, M. U. and Adhami, M. K., 2015. Article: A New Criterion for Evaluating News Search Systems. Communications on Applied Electronics 2(7):28-35, August 2015. Published by Foundation of Computer Science (FCS), NY, USA.
• Bokhari, M. U. and Adhami, M. K., 2016. How well they retrieve fresh news items: News search engine perspective. Perspective in Science. Elsevier, Volume 8, September 2016, Pages 469-471.
• Liu., K. L., Meng, W., Qiu, J., Yu, C., Raghavan, V., Wu., Z., Lu, Y., He, H. and Zhao, H., 2007. AllInOneNews: Development and Evaluation of a Large-Scale News Metasearch Engine. SIGMOD’07 Proceedings of the 2007 ACM SIGMOD international conference on Management of data Pages 1017-1028.
• Lewandowski, D. and Sunkler, S., 2013. Designinh search engine retrieval effectiveness tests with RAT. Information Services & Use 33 (2013) 53–59.
• Ali, R. and Naim, I., 2011. Neural Network based Supervised Rank Aggregation. In international Conference on Multimedia, Signal Processing and Communication Technologies, pages 72-75, IEEE 978-1-4577-1105-3.
• Freund, Y., Iyer, R., Schapire, R. E. and m Singer , Y. 2003. An Efficient Boosting Algorithm for Combining Preferences. Journal of Machine Learning Research 4 (2003) 933-969.
• Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N. and Hullender, G., 2005. Learning to rank using gradient descent. In ICML’05 Proceedings of the 22nd international conference on Machine learning, Pages 89 – 96, Bonn, Germany.
• Xu, J. and Li, Hang., 2007. AdaRank: a boosting algorithm for information retrieval. In SIGIR’07, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 391-398.
• Schapire, R., E. and Singer, Y., 1999. Improved Boosting Algorithms Using Confidence-rated Predictions. Machine Learning, Volume 37, issue 3, pages 297-336.
• Metzler, D. and Croft, W. B., 2006. Linear feature-based models for information retrieval. In Information Retrieval, Kluwer Academic Publishers, Netherlands.
• Wu, Q., Burges, C. J., Svore, K. M. and Gao, J., 2010. Adapting boosting for information retrieval measures. Information Retrieval, Volume 13, issue 3, pages 254-270.
• Burges, C. J., Ragno, R. and Le, Q. V., 2006. Learning to rank with non-smooth cost functions. In Advances in Neural Information Processing Systems 18, 2006.
• Cao, Z., Qin, T., Liu, T. Y., Tsai, M. F. and Li, H., 2007. Learning to rank from pairwise approach to listwise approach. In ICML’07 Proceedings of International Conference on Machine Learning.
• Kristofer, T., 2015. Learning to rank, a supervised approach for ranking of documents. Master Thesis in Computer Science - Algorithms, Languages and Logic Chalmers University of Technology, Sweden.
• Dong, A., Chang, Y., Zheng, Z., Mishne, G., Bai, J., Zhang, R., Buchner, K., Liao, C. and Diaz, F., 2010. Towars recency ranking in web search. In WSDM’10 Proceedings of the third ACM international conference on Web search and data mining, pages 11-20, New York, USA.