intechnguВестник НГУ. Серия: Информационные технологииVestnik NSU. Series: Information Technologies1818-79002410-0420НГУ10.25205/1818-7900-2023-21-3-56-71intechngu-239Research ArticleСтатьиМодульная архитектура системы помощи водителю для эффективного распознавания дорожных знаковModular Architecture of Advanced Driver Assistance Systems for Effective Traffic Sign RecognitionХарченкоИ. К.KharchenkoI. K.

Харченко Игорь Константинович, аспирант кафедры автоматизированных систем управления

Томск

Kharchenko Igor Konstantinovich, Postgraduate Student

Tomsk

igor.k.kharchenko@tusur.ruhttps://orcid.org/0000-0003-2707-7203БоровскойИ. Г.BorovskoyI. G.

Боровской Игорь Георгиевич, доктор физико-математических наук, профессор, заведующийкафедрой экономической математики, информатики и статистики

Томск

Borovskoy Igor Georgievich, Doctor of Physics and Mathematics

Tomsk

igor.g.borovskoi@tusur.ruhttps://orcid.org/0000-0002-6278-5961ШельминаЕ. А.ShelminaE. А.

Шельмина Елена Александровна, кандидат физико-математических наук, доцент кафедры экономической математики, информатики и статистики

Томск

Shelmina Elena Aleksandrovna, Candidate of Physics and Mathematics, Associate Professor

Tomsk

 

elena.a.shelmina@tusur.ruТомский государственный университет систем управления и радиоэлектроникиРоссияTomsk State University of Control Systems and RadioelectronicsRussian Federation2023081220232135671Copyright © Харченко И.К., Боровской И.Г., Шельмина Е.А., 20232023Харченко И.К., Боровской И.Г., Шельмина Е.А.Kharchenko I.K., Borovskoy I.G., Shelmina E.А.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://intechngu.elpub.ru/jour/article/view/239

Целью исследования является разбор современных подходов к реализации систем помощи водителю, а также реализация архитектуры системы помощи водителю, направленная на распознавание дорожного знака на максимальном удалении от него при сложных погодных условиях, для заблаговременной выдачи обратной связи водителю. В работе рассмотрены основные используемые сигналы при реализации и эксплуатации системы помощи водителю: данные с CAN-шины автомобиля, информация с GPS-приемника, фрагменты видео с цифровой камеры. В представленной модульной архитектуре используются перечисленные источники данных для оценивания дорожной обстановки, а также нейросетевые методы для распознавания дорожных знаков. Представлена модульная архитектура системы помощи водителю, позволяющая оповещать водителя о дорожных знаках. Система оснащена контролем границы полосы для оповещения водителя о знаках, относящихся к прилегающей проезжей части при поворотах. Экспериментально доказано, что представленная в работе модульная архитектура системы помощи водителю не уступает по скорости и точности работы альтернативным системам, выступая в виде комплексного автономного решения.

Analysis of modern approaches to the implementation of driver assistance systems, as well as the implementation of the architecture of the driver assistance system, aimed at recognizing traffic signs at the maximum distance from it under difficult weather conditions, for early feedback to the driver. The paper considers the main signals used in the implementation and operation of the driver assistance system: data from the car's CAN bus, information from a GPS receiver, video fragments from a digital camera. The presented modular architecture uses the listed data sources for estimating the traffic situation, as well as neural network methods for recognizing traffic signs. The modular architecture of the driver assistance system is presented, which allows notifying the driver about traffic signs. The system is equipped with lane boundary control to alert the driver to signs related to the adjacent carriageway when turning. It has been experimentally proven that the modular architecture of the driver assistance system presented in the paper is not inferior in speed and accuracy to alternative systems, acting as a comprehensive autonomous solution.

системы помощи водителюпомощь в удержании полосыраспознавание дорожных знаковнейросетевые архитектурыadvanced driver assistance systemslane-keep assistancetraffic sign recognitionartificial neural network architecturesИсследование выполнено при финансовой поддержке гранта на реализацию программы стратегического академического лидерства «Приоритет-2030» в рамках научного проекта «Пр2030-МОН наука БЧ/СП2/Б/19б».The study was supported by a grant for the implementation of the program of strategic academic leadership "Priority-2030" within the framework of the research project "Pr2030-MON science BCh/SP2/B/19b".ReferencesGonzález-Saavedra J. Ph., Figueroa M., Céspedes S., Montejo-Sánchez S. Survey of Cooperative Advanced Driver Assistance Systems: From a Holistic and Systemic Vision // Sensors. 2022. Vol. 22 (8). P. 3040–3080. https://doi.org/10.3390/s22083040González-Saavedra J. Ph., Figueroa M., Céspedes S., Montejo-Sánchez S. Survey of Cooperative Advanced Driver Assistance Systems: From a Holistic and Systemic Vision // Sensors. 2022. Vol. 22 (8). P. 3040–3080. https://doi.org/10.3390/s22083040Xing Y., Lv C., Wang H., Wang H., Ai Y., Cao D., et.al. Driver Lane Change Intention Inference for Intelligent Vehicles: Framework, Survey, and Challenges // IEEE Transactions on Vehicular Technology. 2019. Vol. 68 (5). P./ 4377–4390. https://doi.org/10.1109/TVT.2019.2903299Xing Y., Lv C., Wang H., Wang H., Ai Y., Cao D., et.al. Driver Lane Change Intention Inference for Intelligent Vehicles: Framework, Survey, and Challenges // IEEE Transactions on Vehicular Technology. 2019. Vol. 68 (5). P./ 4377–4390. https://doi.org/10.1109/TVT.2019.2903299Wang Z., Wu Y., Niu Q. Multi-Sensor Fusion in Automated Driving: A Survey // IEEE Access. 2020. Vol. 8. P. 2847–2868. http://dx.doi.org/10.1109/ACCESS.2019.2962554Wang Z., Wu Y., Niu Q. Multi-Sensor Fusion in Automated Driving: A Survey // IEEE Access. 2020. Vol. 8. P. 2847–2868. http://dx.doi.org/10.1109/ACCESS.2019.2962554Yanovsky F. J. Millimeter Wave Technology in Wireless PAN, LAN, and MAN. Sebastopol, CA: Auerbach Publications CRC Press, 2008. 448 p.Yanovsky F. J. Millimeter Wave Technology in Wireless PAN, LAN, and MAN. Sebastopol, CA: Auerbach Publications CRC Press, 2008. 448 p.De-Las-Heras G., Sánchez-Soriano J., Puertas E. Advanced Driver Assistance Systems (ADAS) Based on Machine Learning Techniques for the Detection and Transcription of Variable Message Signs on Roads // Sensors. 2021. Vol. 21(17). P. 5866‑5880. https://doi.org/10.3390/s21175866De-Las-Heras G., Sánchez-Soriano J., Puertas E. Advanced Driver Assistance Systems (ADAS) Based on Machine Learning Techniques for the Detection and Transcription of Variable Message Signs on Roads // Sensors. 2021. Vol. 21(17). P. 5866‑5880. https://doi.org/10.3390/s21175866Lin T.-Y., Goyal P., Girshick R., He K., Dollár P. Focal Loss for Dense Object Detection // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020. Vol. 42(2). P. 318‑327. https://doi.org/10.1109/TPAMI.2018.2858826Lin T.-Y., Goyal P., Girshick R., He K., Dollár P. Focal Loss for Dense Object Detection // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020. Vol. 42(2). P. 318‑327. https://doi.org/10.1109/TPAMI.2018.2858826Robby G. A., Tandra A., Susanto I., Harefa J., Chowanda A. Implementation of Optical Character Recognition using Tesseract with the Javanese Script Target in Android Application // Procedia Computer Science. 2019. Vol. 157. P. 499‑505. https://doi.org/10.1016/j.procs.2019.09.006Robby G. A., Tandra A., Susanto I., Harefa J., Chowanda A. Implementation of Optical Character Recognition using Tesseract with the Javanese Script Target in Android Application // Procedia Computer Science. 2019. Vol. 157. P. 499‑505. https://doi.org/10.1016/j.procs.2019.09.006Lindemann B., Müller T., Vietz H., Jazdi N., Weyrich M. A survey on long short-term memory networks for time series prediction // Procedia CIRP. 2021. Vol. 99. P. 650‑655. https://doi.org/10.1016/j.procir.2021.03.088Lindemann B., Müller T., Vietz H., Jazdi N., Weyrich M. A survey on long short-term memory networks for time series prediction // Procedia CIRP. 2021. Vol. 99. P. 650‑655. https://doi.org/10.1016/j.procir.2021.03.088Temel D., Alshawi T., Chen M.-H., AlRegib G. CURE-TSD: Challenging unreal and real environments for traffic sign detection // IEEE Dataport. URL: https://ieee-dataport.org/openaccess/cure-tsd-challenging-unreal-and-real-environment-traffic-sign-detection (дата обращения 07.07.2023).Temel D., Alshawi T., Chen M.-H., AlRegib G. CURE-TSD: Challenging unreal and real environments for traffic sign detection // IEEE Dataport. URL: https://ieee-dataport.org/openaccess/cure-tsd-challenging-unreal-and-real-environment-traffic-sign-detection (дата обращения 07.07.2023).Емельянов С. О., Иванова А. А., Швец Е. А., Николаев Д. П. Методы аугментации обучающих выборок в задачах классификации изображений // Сенсорные системы. 2018. Т. 32, № 3. С. 236–245.Emelyanov, C. O., Ivanova, A. A., Shvets, E. A., Nikolaev, D. P. Data agumentation methods of training datasets for image classification task. Sensorniye sistemy = Sensor systems. 2018, vol. 32(3), pp. 236‑245. (In Russ.) Available from: https://elibrary.ru/item.asp?doi=10.1134/S0235009218030058 [Accessed 2nd July 2023]Ahmed S., Kamal U., Hasan M. K. DFR-TSD: A Deep Learning Based Framework for Robust Traffic Sign Detection Under Challenging Weather Conditions // IEEE Transactions on Intelligent Transportation Systems. 2021. Vol. 23(6). P. 5150–5162. https://doi.org/10.1109/TITS.2020.3048878Ahmed S., Kamal U., Hasan M. K. DFR-TSD: A Deep Learning Based Framework for Robust Traffic Sign Detection Under Challenging Weather Conditions // IEEE Transactions on Intelligent Transportation Systems. 2021. Vol. 23(6). P. 5150–5162. https://doi.org/10.1109/TITS.2020.3048878Харченко И. К., Боровской И. Г., Шельмина Е. А. Использование ансамбля сверточных нейронных сетей для распознавания дорожных знаков // Вестник Томского гос. ун-та. Управление, вычислительная техника и информатика. 2022. № 61. С. 88–96. 10.17223/19988605/61/9Kharchenko, I. K., Borovskoy, I. G., Shelmina, E. A. Usage of convolutional neural networkAbdennour N., Ouni T., Ben Amor N. Driver identification using only the CAN-Bus vehicle data through an RCN deep learning approach // Robotics and Autonomous Systems. 2021. Vol. 136. https://doi.org/10.1016/j.robot.2020.103707nsemble for traffic sign recognition. Vestnik Tomskogo gosudarstvennogo universiteta. Upravlenie, vychislitelnaja tehnika i informatika = Bulletin of Tomsk State University. Management, Computer Engineering and Informatics. Tomsk State University Journal of Control and Computer Science. 2022, vol. 61, pp. 88–96. (In Russ.) DOI 10.17223/19988605/61/9Zhang J., Wu Z., Li F., Xie C., Ren T., Chen J. et al. A Deep Learning Framework for Driving Behavior Identification on In-Vehicle CAN-BUS Sensor Data // Sensors. 2019. Vol. 19(6). P. 1356‑1373. https://doi.org/10.3390/s19061356Abdennour N., Ouni T., Ben Amor N. Driver identification using only the CAN-Bus vehicle data through an RCN deep learning approach // Robotics and Autonomous Systems. 2021. Vol. 136. https://doi.org/10.1016/j.robot.2020.103707Kubiš M., Beňo P. Realization of communication via the CAN bus // Transportation Research Procedia. 2019. Vol. 40. P. 332–337. https://doi.org/10.1016/j.trpro.2019.07.049Zhang J., Wu Z., Li F., Xie C., Ren T., Chen J. et al. A Deep Learning Framework for Driving Behavior Identification on In-Vehicle CAN-BUS Sensor Data // Sensors. 2019. Vol. 19(6). P. 1356‑1373. https://doi.org/10.3390/s19061356Zhou A., Li Z., Shen Y. Anomaly Detection of CAN Bus Messages Using a Deep Neural Network for Autonomous Vehicles // Applied Sciences. 2019. Vol. 9(15). P. 3174‑3186. https://doi.org/10.3390/app9153174Kubiš M., Beňo P. Realization of communication via the CAN bus // Transportation Research Procedia. 2019. Vol. 40. P. 332–337. https://doi.org/10.1016/j.trpro.2019.07.049Khanam F., Munmun F. A., Ritu N. A., Saha A. K., Mridha M. F. Text to Speech Synthesis: A Systematic Review, Deep Learning Based Architecture and Future Research Direction // Journal of Advances in Information Technology. 2022. Vol. 13(5). P. 398‑412. http://dx.doi.org/10.12720/jait.13.5.398-412Zhou A., Li Z., Shen Y. Anomaly Detection of CAN Bus Messages Using a Deep Neural Network for Autonomous Vehicles // Applied Sciences. 2019. Vol. 9(15). P. 3174‑3186. https://doi.org/10.3390/app9153174Simonyan K., Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition // 3rd International Conference on Learning Representations. 2015. P. 1–14 URL: https://arxiv.org/abs/1409.1556v6 (02.06.2023). https://doi.org/10.48550/arXiv.1409.1556Khanam F., Munmun F. A., Ritu N. A., Saha A. K., Mridha M. F. Text to Speech Synthesis: A Systematic Review, Deep Learning Based Architecture and Future Research Direction // Journal of Advances in Information Technology. 2022. Vol. 13(5). P. 398‑412. http://dx.doi.org/10.12720/jait.13.5.398-412Lucy C., Yu M., Melat K. W., Diarra B., Daniel M. G., Jessilyn D. Does deidentification of data from wearable devices give us a false sense of security? A systematic review // The Lancet Digital Health. 2023. Vol. 5(4). P. 239‑247. https://doi.org/10.1016/S2589-7500(22)00234-5Simonyan K., Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition // 3rd International Conference on Learning Representations. 2015. P. 1–14 URL: https://arxiv.org/abs/1409.1556v6 (02.06.2023). https://doi.org/10.48550/arXiv.1409.1556Smith M., Miller S. Biometric Identifcation, Law and Ethics. Canberra, ALucy C., Yu M., Melat K. W., Diarra B., Daniel M. G., Jessilyn D. Does deidentification of data from wearable devices give us a false sense of security? A systematic review // The Lancet Digital Health. 2023. Vol. 5(4). P. 239‑247. https://doi.org/10.1016/S2589-7500(22)00234-5Smith M., Miller S. Biometric Identifcation, Law and Ethics. Canberra, ASmith M., Miller S. Biometric Identifcation, Law and Ethics. Canberra, A

The authors declare that there are no conflicts of interest present.