Developing a Service for Collecting and Analyzing Electives Reviews

This article provides a review of publications on the analysis of students’ satisfaction with the educational process based on natural language processing methods. 197 student feedback on 129 elective disciplines at University of Tyumen was collected. A comparative analysis of keyword extraction methods was conducted: statistical TF-IDF, RAKE and YAKE; contextual KeyBERT; graph-based TextRank. On the collected reviews, grouped by elective disciplines, the RAKE method had the highest F1 BERTScore with 79 %. By parsing open sources, a dataset with 2210 Russian-language reviews for courses of different educational platforms was formed. Machine learning methods for sentiment analysis were described: support vector machines, logistic regression and based on Transformers, comparison on the manually marked part of the collected reviews. After fine-tuning on the rubert-base-cased model macro-averaged F1- score became 71.6 %. Classification into three classes (negative, neutral, positive) is not performed for the whole text of the review, but separately for each sentence from that text. The implementation of a database and information system for collecting and analyzing student feedback on the studied elective courses are presented. The model for sentiment analysis of the feedback is put into a separate microservice, which is communicated through an interface of the freely distributed Python framework FastAPI. The information system is designed to help students choose electives based on more qualitative data, and teachers and university administration ‑ to draw conclusions for further transformation of the educational space, taking into account students’ opinions.

1. Fedorova N. K. Individualizatsiya obrazovaniya: model’ Tyumenskogo gosudarstvennogo universiteta / N.K. Fedorova // EdCrunch Tomsk : Materialy mezhdunarodnoi konferentsii po novym obrazovatel’nym tekhnologiyam, 29-31 maya 2019 goda. – Tomsk: Izdatel’skii Dom Tomskogo gosudarstvennogo universiteta, 2019. – p. 301-305 (in Russ).

2. Zakharova I. G., Vorobeva M. S., Boganyuk Yu. V. Support of individual educational trajectories based on the concept of explainable artificial intelligence. The Education and Science Journal. 2022; 24 (1): p.163–190. (In Russ.) DOI: 10.17853/1994-5639-2022-1-163-190

3. Gottipati S., Shankararaman V., Lin J. R. Text analytics approach to extract course improvement suggestions from students’ feedback. Res Pract Technol Enhanc Learn. 2018;13(1):6. DOI: 10.1186/s41039-018-0073-0.

4. Shejwal S., Deokar T., Dumbre B. Analysis of Student Feedback using Deep Learning. International Journal of Computer Applications Technology and Research, 2019. 8. p.161–164. DOI: 10.7753/IJCATR0805.1004.

5. Kirina M. A. Avtomaticheskaya otsenka vpechatlenii obuchayushchikhsya metodami analiza tonal’nosti (na materiale otzyvov na onlain-kursy na russkom i angliiskom) / M. A. Kirina, L. D. Tel’nina // Tsifrovaya gumanitaristika i tekhnologii v obrazovanii (DHTE 2022) : Sbornik statei III Vserossiiskoi nauchno-prakticheskoi konferentsii s mezhdunarodnym uchastiem, Moskva, 17–18 noyabrya 2022 goda / Pod redaktsiei V.V. Rubtsova, M.G. Sorokovoi, N.P. Radchikovoi. Moskva: Moskovskii gosudarstvennyi psikhologo-pedagogicheskii universitet, 2022. Р. 355–374. EDN VJVKLU. (In Russ.)

6. Dyulicheva Yu. Tu. Dataset dlya analiza russkoyazychnykh otzyvov na MOOK, izvlechennykh s platformy Stepik [Dataset for Analysisof Russian-Language Reviews on MOOCs Expacted from Stepik]. Voprosy obrazovaniya / Educational Studies Moscow, no. 4. Р. 298–321. (In Russ.) DOI: 10.17323/1814-9545-2022-4-298-321

7. Jones K. S. A statistical interpretation of term specificity and its application in retrieval // Journal of Documentation : journal. MCB University: MCB University Press, 1972. Vol. 28, no. 1. Р. 11–21. DOI: 10.1108/00220410410560573

8. Stuart R., Dave E., Nick C., Wendy Cowley, Automatic Keyword Extraction from Individual Documents. March 2010. DOI:10.1002/9780470689646.ch1. In book: Text Mining: Applications and Theory, p. 1–20. DOI: 10.1002/9780470689646.ch1

9. Campos R., Mangaravite V., Pasquali A., Alípio Mário Jorge, Célia Nunes, Adam Jatowt, YAKE! Collection-Independent Automatic Keyword Extractor. Conference paper. DOI: 10.1007/978-3-319-76941-7_80

10. Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd. The PageRank Citation Ranking: Bringing Order to the Web. 1998.

11. Rada Mihalcea, Paul Tarau. TextRank: Bringing Order into Texts. Department of Computer Science University of North Texas.

12. Sharma, P., Li, Y. (2019). Self-Supervised Contextual Keyword and Keyphrase Retrieval with Self-Labelling. Aug. 2019, DOI: 10.20944/preprints201908.0073.v1

13. Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 11 October 2018

14. Vaswani, A. Attention is All You Need / Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorei, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin // NIPS, 2017

15. Lin, Chin-Yew. 2004. ROUGE: a Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain, July 25 - 26, 2004.

16. Papineni, Kishore; Roukos, Salim; Ward, Todd; Zhu, Wei-Jing (2001). “BLEU”. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ‘02. Morristown, NJ, USA: Association for Computational Linguistics. DOI:10.3115/1073083.1073135

17. Tianyi Zhang. BERTScore: Evaluating Text Generation with BERT. // Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi.

18. Kuratov Y., Arkhipov M. Adaptation of deep bidirectional multilingual transformers for russian language. 2019.

19. Smetanin S., Komarov M. Sentiment Analysis of Product Reviews in Russian using Convolutional Neural Networks, 2019 IEEE 21st Conference on Business Informatics (CBI), 2019, p. 482-486, DOI: 10.1109/CBI.2019.00062