An Introduction to BCI and Its Use in Video Games: A Review
DOI:
https://doi.org/10.47611/jsrhs.v12i1.3924Keywords:
electroencephalography, video games, brain-computer interface, machine learning, signal analysisAbstract
The input of a video game can vary from a keyboard, a mouse, a controller, and a plethora of other methods. The electroencephalogram (EEG) is a cap worn on the head that can detect electrical signals in the brain. This device is becoming seen as either an alternative to traditional controllers or a supplement. The EEG can be used with a computer to become a Brain Computer Interface (BCI), where a feedback loop is created between the game and direct signals from the brain. BCIs are increasingly being used for video games, whether for entertainment or serious purposes. In this paper, we review the components of a BCI and assess the general state of its use in video games. We describe the EEG, what inputs to measure, common preprocessing techniques, and different machine learning algorithms. We assess the game-making side, discussing the types of games made. We conclude the paper by listing current limitations in different disciplines and point towards possible areas that need further innovation for the technology to become widespread.
Downloads
References or Bibliography
Albawi, S., Mohammed, T. A., & Al-Zawi, S. (2017). Understanding of a convolutional neural network. 2017 International Conference on Engineering and Technology (ICET). doi:10.1109/icengtechnol.2017
Ali, A., & Puthusserypady, S. (2015). A 3D learning playground for potential attention training in ADHD: A brain computer interface approach. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference, 2015, 67–70. doi:10.1109/EMBC.2015.7318302
Allison, B. Z., Brunner, C., Altstätter, C., Wagner, I. C., Grissmann, S., & Neuper, C. (2012). A hybrid ERD/SSVEP BCI for continuous simultaneous two dimensional cursor control. Journal of Neuroscience Methods, 209(2), 299–307. doi:10.1016/j.jneumeth.2012.06.022
Aphex34 (2015). Typical cnn. [Diagram]. Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Typical_cnn.png.
Baek, H. J., Kim, H. S., Ahn, M., Cho, H., & Ahn, S. (2020). Ergonomic Issues in Brain-Computer Interface Technologies: Current Status, Challenges, and Future Direction. Computational Intelligence and Neuroscience, 2020, 1–2. doi:10.1155/2020/4876397
Berta, R., Bellotti, F., De Gloria, A., Pranantha, D., & Schatten, C. (2013). Electroencephalogram and physiological signal analysis for assessing flow in games. IEEE Transactions on Computational Intelligence and AI in Games, 5(2), 164-175. doi:10.1109/TCIAIG.2013.2260340
Bordoloi, S., Sharmah, U., & Hazarika, S. M. (2012). Motor imagery based BCI for a maze game. 2012 4th International Conference on Intelligent Human Computer Interaction (IHCI). doi:10.1109/ihci.2012.6481848
Bradberry, T. J., Gentili, R. J., & Contreras-Vidal, J. L. (2011). Fast attainment of computer cursor control with noninvasively acquired brain signals. Journal of Neural Engineering, 8(3), 036010. doi:10.1088/1741-2560/8/3/036010
Bugli, C., & Lambert, P. (2007). Comparison between Principal Component Analysis and Independent Component Analysis in Electroencephalograms Modelling. Biometrical Journal, 49(2), 312–327. doi:10.1002/bimj.200510285
Cabañero-Gómez, L., Hervas, R., Bravo, J., & Rodriguez-Benitez, L. (2018). Computational EEG analysis techniques when playing video games: a systematic review. Multidisciplinary Digital Publishing Institute Proceedings, 2(19), 483. doi:10.3390/proceedings2190483
Cattan, G. (2021). The use of brain–computer interfaces in games is not ready for the general public. Frontiers in computer science, 3, 628773. doi:10.3389/fcomp.2021.628773
Chanel, G., Rebetez, C., Bétrancourt, M., & Pun, T. (2011). Emotion assessment from physiological signals for adaptation of game difficulty. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 41(6), 1052-1063. doi:10.1109/TSMCA.2011.2116000
Coenen, F., Scheepers, F. E., Palmen, S., de Jonge, M. V., & Oranje, B. (2020). Serious Games as Potential Therapies: A Validation Study of a Neurofeedback Game. Clinical EEG and neuroscience, 51(2), 87–93. doi:10.1177/1550059419869471
Coin, A., Mulder, M., & Dubljević, V. (2020). Ethical aspects of BCI technology: what is the state of the art?. Philosophies, 5(4), 31. doi:10.3390/philosophies5040031
Comon, P. (1994). Independent component analysis, a new concept?. Signal processing, 36(3), 287-314. doi:10.1016/0165-1684(94)90029-9
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine learning, 20(3), 273-297. doi:10.1007/BF00994018
Coyle, D., Garcia, J., Satti, A. R., & McGinnity, T. M. (2011). EEG-based continuous control of a game using a 3 channel motor imagery BCI: BCI game. 2011 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB). doi:10.1109/ccmb.2011.5952128
Debener, S., Makeig, S., Delorme, A., & Engel, A. K. (2005). What is novel in the novelty oddball paradigm? Functional significance of the novelty P3 event-related potential as revealed by independent component analysis. Cognitive Brain Research, 22(3), 309-321. doi:10.1016/j.cogbrainres.2004.09.006
Dornhege, G., Blankertz, B., Krauledat, M., Losch, F., Curio, G., & Muller, K.-R. (2006). Combined Optimization of Spatial and Temporal Filters for Improving Brain-Computer Interfacing. IEEE Transactions on Biomedical Engineering, 53(11), 2274–2281. doi:10.1109/tbme.2006.883649
Fouad, I. A. (2021). A robust and reliable online P300-based BCI system using Emotiv EPOC+ headset. Journal of Medical Engineering & Technology, 45(2), 94-114. doi:10.1080/03091902.2020.1853840
Hu, L., & Zhang, Z. (Eds.). (2019). EEG Signal Processing and Feature Extraction. doi:10.1007/978-981-13-9113-2
Huang, D., Qian, K., Fei, D. Y., Jia, W., Chen, X., & Bai, O. (2012). Electroencephalography (EEG)-based brain–computer interface (BCI): A 2-D virtual wheelchair control based on event-related desynchronization/synchronization and state control. IEEE transactions on Neural Systems and Rehabilitation engineering, 20(3), 379-388. doi:10.1109/TNSRE.2012.2190299
Izenman, A. J. (2013). Linear Discriminant Analysis. Modern Multivariate Statistical Techniques, 237–280. doi:10.1007/978-0-387-78189-1_8
Kato, K., Takahashi, K., Mizuguchi, N., & Ushiba, J. (2018). Online detection of amplitude modulation of motor-related EEG desynchronization using a lock-in amplifier: Comparison with a fast Fourier transform, a continuous wavelet transform, and an autoregressive algorithm. Journal of Neuroscience Methods, 293, 289–298. doi:10.1016/j.jneumeth.2017.10.015
Kerous, B., Skola, F., & Liarokapis, F. (2017). EEG-based BCI and video games: a progress report. Virtual Reality, 22(2), 119–135. doi:10.1007/s10055-017-0328-x
Kuzovkin, I. V.. “Pattern recognition for non-invasive EEG-based BCI.” (2011).
Lalor, E. C., Kelly, S. P., Finucane, C., Burke, R., Smith, R., Reilly, R. B., & Mcdarby, G. (2005[ 8/25/22, 10:29 Not that this article is from 2005, mention or take into account when using]). Steady-state VEP-based brain-computer interface control in an immersive 3D gaming environment. EURASIP Journal on Advances in Signal Processing, 2005(19), 1-9. doi:10.1155/ASP.2005.3156
Lécuyer, A. (2016). Bcis and video games: State of the art with the openvibe2 project. Brain–Computer Interfaces 2: Technology and Applications, 85-99. doi:10.1002/9781119332428.ch5
Lee, M.-H., Kwon, O.-Y., Kim, Y.-J., Kim, H.-K., Lee, Y.-E., Williamson, J., … Lee, S.-W. (2019). EEG Dataset and OpenBMI Toolbox for Three BCI Paradigms: An Investigation into BCI Illiteracy. GigaScience. doi:10.1093/gigascience/giz002
Li, M., Li, F., Pan, J., Zhang, D., Zhao, S., Li, J., & Wang, F. (2021). The MindGomoku: An online P300 BCI game based on Bayesian deep learning. Sensors, 21(5), 1613. doi:10.3390/s21051613
Liao, L.-D., Chen, C.-Y., Wang, I-Jan., Chen, S.-F., Li, S.-Y., Chen, B.-W., Chang, J.-Y., & Lin, C.-T. (2012). Gaming control using a wearable and wireless EEG-based brain-computer interface device with novel dry foam-based sensors. Journal of NeuroEngineering and Rehabilitation, 9(1), 5. doi:10.1186/1743-0003-9-5
Liu, X., Shen, Y., Liu, J., Yang, J., Xiong, P., & Lin, F. (2020). Parallel Spatial–Temporal Self-Attention CNN-Based Motor Imagery Classification for BCI. Frontiers in Neuroscience, 14. doi:10.3389/fnins.2020.587520
Mai, Q. (2013). A review of discriminant analysis in high dimensions. Wiley Interdisciplinary Reviews: Computational Statistics, 5(3), 190–197. doi:10.1002/wics.1257
Martišius, I., & Damaševičius, R. (2016). A Prototype SSVEP Based Real Time BCI Gaming System. Computational Intelligence and Neuroscience, 2016, 1–15. doi:10.1155/2016/3861425
Mathewson, K. E., Harrison, T. J. L., & Kizuk, S. A. D. (2016). High and dry? Comparing active dry EEG electrodes to active and passive wet electrodes. Psychophysiology, 54(1), 74–82. doi:10.1111/psyp.12536
Mercado, J., Escobedo, L., & Tentori, M. (2021). A BCI video game using neurofeedback improves the attention of children with autism. Journal on Multimodal User Interfaces, 15(3), 273-281. doi:10.1007/s12193-020-00339-7
Neuper, C., Scherer, R., Wriessnegger, S., & Pfurtscheller, G. (2009). Motor imagery and action observation: Modulation of sensorimotor brain rhythms during mental control of a brain–computer interface. Clinical Neurophysiology, 120(2), 239–247. doi:10.1016/j.clinph.2008.11.015
“OpenBCI Home Page.” OpenBCI, openbci.com/.
O'Shea, K., & Nash, R. (2015). An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458. doi:10.48550/arXiv.1511.08458
Peterson, V., Galván, C., Hernández, H., & Spies, R. (2020). A feasibility study of a complete low-cost consumer-grade brain-computer interface system. Heliyon, 6(3), e03425. doi:10.1016/j.heliyon.2020.e03425
Pires, G., Torres, M., Casaleiro, N., Nunes, U., & Castelo-Branco, M. (2011, November 1). Playing Tetris with non-invasive BCI. IEEE Xplore. doi:10.1109/SeGAH.2011.6165454
Refaeilzadeh, P., Tang, L., & Liu, H. (2009). Cross-validation. Encyclopedia of database systems, 5, 532-538. doi:10.1007/978-1-4899-7993-3_565-2
Renard Y., Lotte F., Gibert G., Congedo M., Maby E., Delannoy V., Bertrand O., & Lécuyer A. (2010). Openvibe: An open-source software platform to design, test, and use brain--computer interfaces in real and virtual environments. Presence: Teleoperators and Virtual Environments, 19(1), 35–53. doi:10.1162/pres.19.1.35
Rosenfeld, J. V., & Wong, Y. T. (2017). Neurobionics and the brain–computer interface: current applications and future horizons. Medical Journal of Australia, 206(8), 363–368. doi:10.5694/mja16.01011
Sanei, S., & Chambers, J. A. (2021). EEG Signal Processing and Machine Learning, 2nd Edition. John Wiley & Sons. ISBN:1119386942.
Scherer, R., Lee, F., Schlogl, A., Leeb, R., Bischof, H., & Pfurtscheller, G. (2008). Toward self-paced brain–computer communication: navigation through virtual worlds. IEEE Transactions on Biomedical Engineering, 55(2), 675-682. doi:10.1109/TBME.2007.903709
Schlüter, H., & Bermeitinger, C. (2017). Emotional oddball: A review on variants, results, and mechanisms. Review of General Psychology, 21(3), 179-222. doi:10.1037/gpr0000120
Shad, E. H. T., Molinas, M., & Ytterdal, T. (2020). Impedance and Noise of Passive and Active Dry EEG Electrodes: A Review. IEEE Sensors Journal, 1–1. doi:10.1109/jsen.2020.3012394
Shanmuganathan, S. (2016). Artificial Neural Network Modelling: An Introduction. Studies in Computational Intelligence, 1–14. doi:10.1007/978-3-319-28495-8_1
Tangermann, M., Krauledat, M., Grzeska, K., Sagebaum, M., Vidaurre, C., Blankertz, B., & Müller, K.R. (2008). Playing Pinball with Non-Invasive BCI. In Proceedings of the 21st International Conference on Neural Information Processing Systems (pp. 1641–1648). Curran Associates Inc.
Thompson, M. C. (2018). Critiquing the Concept of BCI Illiteracy. Science and Engineering Ethics. doi:10.1007/s11948-018-0061-1
Torres, E. P., Torres, E. A., Hernández-Álvarez, M., & Yoo, S. G. (2020). EEG-Based BCI Emotion Recognition: A Survey. Sensors, 20(18), 5083. doi:10.3390/s20185083
Vaid, S., Singh, P., & Kaur, C. (2015). EEG signal analysis for BCI interface: A review. In 2015 fifth international conference on advanced computing & communication technologies, 143-147. IEEE. doi:10.1109/ACCT.2015.72
Vourvopoulos, A., & Bermúdez i Badia, S. (2016). Motor priming in virtual reality can augment motor-imagery training efficacy in restorative brain-computer interaction: a within-subject analysis. Journal of NeuroEngineering and Rehabilitation, 13(1). doi:10.1186/s12984-016-0173-2
Waldert, S., Pistohl, T., Braun, C., Ball, T., Aertsen, A., & Mehring, C. (2009). A review on directional information in neural signals for brain-machine interfaces. Journal of Physiology-Paris, 103(3-5), 244–254. doi:10.1016/j.jphysparis.2009.08.007
Wang, Q., Sourina, O., & Nguyen, M. K. (2011). Fractal dimension based neurofeedback in serious games. The Visual Computer, 27(4), 299-309. doi:10.1007/s00371-011-0551-5
Wang, Y., Wang, Y., & Jung, T. (2010). Visual stimulus design for high-rate SSVEP BCI. Electronics Letters, 46, 1057-1058. doi:10.1049/el.2010.0923
Wen, D., Fan, Y., Hsu, S.-H., Xu, J., Zhou, Y., Tao, J., … Li, F. (2020). Combining brain–computer interface and virtual reality for rehabilitation in neurological diseases: A narrative review. Annals of Physical and Rehabilitation Medicine. doi:10.1016/j.rehab.2020.03.015
Wolpaw, J. R. (2013). Brain–computer interfaces. In Handbook of Clinical Neurology (Vol. 110, pp. 67-74). Elsevier. doi:10.1016/B978-0-444-52901-5.00006-X
Xanthopoulos, P., Pardalos, P. M., & Trafalis, T. B. (2012). Linear Discriminant Analysis. Robust Data Mining, 27–33. doi:10.1007/978-1-4419-9878-1_4
Yong X., Ward, R. K., & Birch, G. E. (2008). Robust Common Spatial Patterns for EEG signal preprocessing. 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. doi:10.1109/iembs.2008.4649604
Zheng, Y., Li, R., Li, S., Zhang, Y., Yang, S., & Ning, H. (2021). A Review on Serious Games for ADHD. arXiv preprint arXiv:2105.02970. doi:10.48550/arXiv.2105.02970
トマトン124 (2010). Electrode locations of International 10-20 system for EEG (electroencephalography) recording. [Clipart]. Wikimedia Commons. https://commons.wikimedia.org/wiki/File:21_electrodes_of_International_10-20_system_for_EEG.svg
Published
How to Cite
Issue
Section
Copyright (c) 2023 Noah Thomasson
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright holder(s) granted JSR a perpetual, non-exclusive license to distriute & display this article.
