Performance Evaluation of Sentiment Analysis Using Machine Learning Models

Alexander Ntino, Souley Boukari, Sani Abba, Monday Simon

Abstract


Sentiment analysis is the process of using natural language processing and machine learning techniques to determine the emotional tone or sentiment expressed in a piece of text, such as positive, negative, or neutral. Many techniques were used to evaluate similar models but the developed model utilizes natural language preprocessing, text mining and sentiment analysis techniques to analyze Twitter datasets using deep learning algorithms.  Pre-trained word embedding’s, such as Word2Vec were utilized to capture semantic relationships, while TF-IDF vectorization was employed to convert text into numerical features. The dataset was split into training and testing sets, and machine learning models, LSTM and Naive Bayes are trained and evaluated based on accuracy metrics. The performance of LSTM, and Naïve Bayes were compared. LSTM has a performance of 0.937, and Naïve Bayes 0.795. The results indicate that, with an accuracy of 0.937 in predicting sentiment analysis on Twitter, the LSTM model outperforms Naïve Bayes that was developed and tested using the same dataset and system configuration. The developed Model demonstrates  superior  performance  compared  to  the  baseline  model  in  terms  of  accuracy, precision and Recall with the performance improvement of 0.142 in accuracy. We recommend this model to be deployed on social media for sentiment analysis. 


Full Text:

PDF

References


Arar, Ö. F., & Ayan, K. (2017). A feature dependent Naive Bayes approach and its application to the software defect prediction problem. Applied Soft Computing, 59, 197-209.

Ashi, M. M., Siddiqui, M. A., & Nadeem, F. (2018). Pre-trained word embeddings for Arabic aspect-based sentiment analysis of airline tweets. Paper presented at the International Conference on Advanced Intelligent Systems and Informatics.

Carneiro, D., Novais, P., Andrade, F., Zeleznikow, J., & Neves, J. (2014). Online dispute resolution: an artificial intelligence perspective. Artificial Intelligence Review, 41(2), 211- 240.

Cheong, M., & Lee, V. C. (2011). A microblogging-based approach to terrorism informatics: Exploration and chronicling civilian sentiment and response to terrorism events via Twitter. Information Systems Frontiers, 13(1), 45-59.

Chouhan, K., Yadav, M., Rout, R. K., Sahoo, K. S., Jhanjhi, N., Masud, M., & Aljahdali, S.

(2023). Sentiment Analysis with Tweets Behaviour in Twitter Streaming API. Comput. Syst. Sci. Eng., 45(2), 1113-1128.

Derbentsev, V. D., Bezkorovainyi, V. S., Matviychuk, A. V., Pomazun, O. M., Hrabariev, A. V., Hostryk, A. M. (2023). A comparative study of deep learning models for sentiment analysis of social media texts. Paper presented at the CEUR Workshop Proceedings.

El Barachi, M., AlKhatib, M., Mathew, S., & Oroumchian, F. (2021). A novel sentiment analysis framework for monitoring the evolving public opinion in real-time: Case study on climate change. Journal of Cleaner Production, 312, 127820.

Ferdiansyah, I. F. (2022). Comparative Study of Hoax Detection using Support Vector Machine Algorithm, Naive Bayes, Random Forest and K-Nearest Neighbor. Paper presented at the 2022 13th International Conference on Information and Communication Technology Convergence (ICTC).

Gandhi, U. D., Malarvizhi Kumar, P., Chandra Babu, G., & Karthick, G. (2021). Sentiment

Analysis on Twitter Data by Using Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM). Wireless Personal Communications. doi:10.1007/s11277-021- 08580-3

Godara, J., Aron, R., & Shabaz, M. (2022). Sentiment analysis and sarcasm detection from social network to train health-care professionals. World Journal of Engineering, 19(1), 124-133. doi:10.1108/WJE-02-2021-0108

Hazarika, D., Konwar, G., Deb, S., & Bora, D. J. (2020). Sentiment Analysis on Twitter by Using TextBlob for Natural Language Processing. ICRMAT, 24, 63-67.

Huang, X., Zhang, W., Tang, X., Zhang, M., Surbiryala, J., Iosifidis, V., Zhang, J. (2021). Lstm based sentiment analysis for cryptocurrency prediction. Paper presented at the Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 11–14, 2021, Proceedings, Part III 26.

Irmayani, D., Edi, F., Harahap, J. M., Rangkuti, R. K., Ulya, B., & Watrianthos, R. (2021). Naives Bayes algorithm for twitter sentiment analysis. Paper presented at the Journal of Physics: Conference Series.

Jazib, A., Tariq, W., & Mahmood, M. (2022). Sentiment Analysis using Ensemble Classifier for Entrepreneurs based on Twitter Analytics. Paper presented at the 2022 19th International Bhurban Conference on Applied Sciences and Technology (IBCAST).

Kamyab, M., Tao, R., Mohammadi, M. H., & Rasool, A. (2018). Sentiment analysis on Twitter: A text mining approach to the Afghanistan status reviews. Paper presented at the Proceedings of the 2018 International Conference on Artificial Intelligence and Virtual Reality.

Kanojia, D., & Joshi, A. (2023). Applications and Challenges of Sentiment Analysis in Real-life Scenarios. arXiv preprint arXiv:2301.09912.

Kausar, M. A., Soosaimanickam, A., & Nasar, M. (2021). Public sentiment analysis on Twitter data during COVID-19 outbreak. International Journal of Advanced Computer Science and Applications, 12(2).

Khrais, L. T. (2020). Role of artificial intelligence in shaping consumer demand in E-commerce. Future Internet, 12(12), 226.

Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023). Natural language processing: State of the art, current trends and challenges. Multimedia tools and applications, 82(3), 3713-3744. Lee, E., Rustam, F., Washington, P. B., El Barakaz, F., Aljedaani, W., & Ashraf, I. (2022). Racism detection by analyzing differential opinions through sentiment analysis of tweets using stacked ensemble gcr-nn model. IEEE Access, 10, 9717-9728.

Neri, F., Aliprandi, C., Capeci, F., Cuadros, M., & By, T. (2012). Sentiment analysis on social media. Paper presented at the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining.

Ogudo, K. A., & Nestor, D. M. J. (2019). Sentiment analysis application and natural language processing for mobile network operators’ support on social media. Paper presented at the 2019 International Conference on Advances in Big Data, Computing and Data Communication Systems (icABCD).

Philander, K., & Zhong, Y. (2016). Twitter sentiment analysis: Capturing sentiment from integrated resort tweets. International Journal of Hospitality Management, 55(2016), 16-24.

Radiuk, P., Pavlova, O., & Hrypynska, N. (2022). An ensemble machine learning approach for Twitter sentiment analysis.

Rahmaniar, W., & Hernawan, A. (2021). Real-time human detection using deep learning on embedded platforms: A review. Journal of Robotics and Control (JRC), 2(6), 462-468. Rajput, A. (2020). Natural language processing, sentiment analysis, and clinical analytics. In Innovation in Health Informatics (pp. 79-97): Elsevier.

Revathy, G., Alghamdi, S. A., Alahmari, S. M., Yonbawi, S. R., Kumar, A., & Haq, M. A. (2022). Sentiment analysis using machine learning: Progress in the machine intelligence for data science. Sustainable Energy Technologies and Assessments, 53, 102557.

Rodrigues, A. P., Fernandes, R., A, A., B, A., Shetty, A., K, A., Shafi, R. M. (2022). Real-Time Twitter Spam Detection and Sentiment Analysis using Machine Learning and Deep Learning Techniques. Computational Intelligence and Neuroscience, 2022, 5211949. doi:10.1155/2022/5211949

Rosenthal, S., Farra, N., & Nakov, P. (2019). SemEval-2017 task 4: Sentiment analysis in Twitter. arXiv preprint arXiv:1912.00741.

Sasikala, P., & Sheela, L. M. I. (2018). Sentiment analysis of online food reviews using customer ratings. International Journal of Pure and Applied Mathematics, 119(15), 3509-3514. Services, D. W. (2019). What is Artificial Neural Network – Structure, Working, Applications. Retrieved from https://data-flair.training/blogs/artificial-neural-network/

Shanmugavadivel, K., Sathishkumar, V., Raja, S., Lingaiah, T. B., Neelakandan, S., & Subramanian, M. (2022). Deep learning based sentiment analysis and offensive language identification on multilingual code-mixed data. Scientific Reports, 12(1), 21557.

Sisodia, D. S., Vishwakarma, S., & Pujahari, A. (2017). Evaluation of machine learning models for employee churn prediction. Paper presented at the 2017 International Conference on Inventive Computing and Informatics (ICICI).

Sivakumar, S., Videla, L. S., Kumar, T. R., Nagaraj, J., Itnal, S., & Haritha, D. (2020). Review on word2vec word embedding neural net. Paper presented at the 2020 international conference on smart electronics and communication (ICOSEC).

Srinivas, A. C. M. V., Satyanarayana, C., Divakar, C., & Sirisha, K. P. (2021). Sentiment analysis using neural network and LSTM. Paper presented at the IOP conference series: materials science and engineering.

Sugumaran, P., & Uma, A. (2022). Real-time twitter data analytics of mental illness in COVID-19: sentiment analysis using deep neural network. Indonesian Journal of Electrical Engineering and Computer Science, 26(1), 560-567.

Trupthi, M., Pabboju, S., & Narasimha, G. (2017). Sentiment analysis on twitter using streaming API. Paper presented at the 2017 IEEE 7th International Advance Computing Conference (IACC).

Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731-5780.

Zimbra, D., Abbasi, A., Zeng, D., & Chen, H. (2018). The state-of-the-art in Twitter sentiment analysis: A review and benchmark evaluation. ACM Transactions on Management Information Systems (TMIS), 9(2), 1-29.


Refbacks

  • There are currently no refbacks.