Abstract

The goal of this research is to develop a text similarity program using word frequency and cosine similarity in Python. The purpose of text similarity is to measure the similarity between texts. The word frequency is used to measure the word importance in the text, and cosine similarity is used to measure the similarity between texts. The basic steps of text similarity are explained: preprocessing text, creating list of words, creating bag of words, creating word frequency, calculating cosine similarity, and printing similarity score. The developed program was tested on an experimental text from Wikipedia. The program successfully performed the basic steps of text similarity and provided the required results.

References

• Sammut, C., & Webb, G. I. (2011). "Encyclopedia of Machine Learning". Springer.
• Aggarwal, C. (2015). "Data Mining: The Textbook". New York: Springer.
• Aggarwal, C. (2018). "Machine Learning for Text". New York: Springer.
• Hotho, A., Nürnberger, A., & Paass, G. (2005). "A Brief Survey of Text Mining". LDV Forum - GLDV Journal for Computational Linguistics and Language Technology. 20, 19-62.
• Gomaa, W. H., & Fahmy, A. A. (2013). "A Survey of Text Similarity Approaches". International Journal of Computer Applications, 68(13), 13-18.
• Breitinger, C., Gipp, B., Langer, S. (2015). "Research-Paper Recommender Systems: A Literature Survey". International Journal on Digital Libraries, 17(4), 305-338.
• Vijaymeena, M. K., & Kavitha, K. (2016). "A Survey on Similarity Measures in Text Mining". Machine Learning and Applications: An International Journal, 3(2), 19-28.
• Gunawan, D., Sembiring, C. A., & Budiman, M. A. (2018). "The Implementation of Cosine Similarity to Calculate Text Relevance between Two Documents". In Journal of Physics: Conference Series (Vol. 978, p. 012120). IOP Publishing.
• Prasetya, D. D., Wibawa, A. P., & Hirashima, T. (2018). "The Performance of Text Similarity Algorithms". International Journal of Advances in Intelligent Informatics, 4(1), 63-69.
• Shahmirzadi, O., Lugowski, A., & Younge, K. (2019). "Text Similarity in Vector Space Models: A Comparative Study". In 2019 18th IEEE international conference on machine learning and applications (ICMLA) (pp. 659-666). IEEE.
• Wang, J., & Dong, Y. (2020). "Measurement of Text Similarity: A Survey". Information, 11(9), 421.
• Luhn, H. (1958). "The Automatic Creation of Literature Abstracts". IBM Journal of Research and Development, 2(2), 159-165.
• Salton, G. & Lesk, M. E. (1965). "The SMART Automatic Document Retrieval Systems: An Illustration". Communications of the ACM. 8 (6): 391-398.
• Salton, G. (1971). "The SMART Retrieval System: Experiments in Automatic Document Retrieval". Englewood Cliffs, N.J.: Prentice Hall Inc.
• Salton, G., Wong, A., & Yang, C. S. (1975). "A Vector Space Model for Automatic Indexing". Communications of the ACM, 18(11), 613-620.
• Salton, G., Yang, C. S., & Yu, C. T. (1975). "A Theory of Term Importance in Automatic Text Analysis". Journal of the American Society for Information Science, 26(1), 33-44.
• Salton, G. & McGill, M. (1983). "Introduction to Modern Information Retrieval". McGraw Hill Book Co, New York.
• Salton, G., & Buckley, C. (1988). "Term-Weighting Approaches in Automatic Text Retrieval". Information Processing and Management, 24(5), 513-523.
• Salton, G. (1989). "Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer". Addison- Wesley Publishing Company, USA.
• Salton, G., Allan, J., & Buckley, C. (1994). "Automatic Structuring and Retrieval of Large Text Files". Communications of the ACM, 37(2), 97-108.
• Salton, G., Singhal, A., Mitra, M., & Buckley, C. (1997). "Automatic Text Structuring and Summarization". Information Processing & Management, 33(2), 193-207.
• Sparck Jones, K. (1972). "A Statistical Interpretation of Term Specificity and Its Application in Retrieval". Journal of Documentation. 28(1), 11–21.
• Sparck Jones, K. (2004). "IDF Term Weighting and IR Research Lessons". Journal of Documentation, 60(5), 521-523.
• Robertson, S. (1972). "Term Specificity". Journal of Documentation, 28(1), 164-165.
• Robertson, S. (1974). "Documentation Note: Specificity and Weighted Retrieval". Journal of Documentation, 30(1), 41-46.
• Robertson, S. (2004). "Understanding Inverse Document Frequency: On Theoretical Arguments for IDF". Journal of Documentation, 60(5), 503-520.
• Python: https://www.python.org
• Numpy: https://www.numpy.org
• Pandas: https:// pandas.pydata.org
• Matplotlib: https://www. matplotlib.org
• NLTK: https://www.nltk.org
• SK Learn: https://scikit-learn.org
• Wikipedia: https://en.wikipedia.org