A Novel Approach to Optimize Memory Reconstruction Using Joint Multimodal Networks
DOI:
https://doi.org/10.47611/jsrhs.v12i3.4622Keywords:
Machine Learning, Generative Adversarial Networks, Conditional Variational Autoencoder-Generative Adversarial Network, EEG, Memory ReconstructionAbstract
Memories play a crucial role in the human life experience. Whether they are used to make decisions or understand current situations, memories influence the past, present, and future. However, some memories are unreliable; in the court of law, the false memories of victims can cause inaccurate verdicts. To reconstruct human memories directly from the brain, recent studies have used electroencephalographic (EEG) brain signals in machine learning frameworks like generative adversarial networks (GANs), which generates new data from old ones. However, traditional GANs tend to produce the same images, an issue called mode collapse. Therefore, a conditional variational autoencoder-generative adversarial network (CVAE-GAN), which had not been used before in memory reconstruction, was developed to address GAN failures by jointly-training (1) an encoder and (2) GAN. CVAE-GAN correlated the MindBigData dataset’s EEG brain signals with the corresponding ImageNet dataset images and produced new memories of participants, such as pandas, humans, and fish. CVAE-GAN’s Inception Score, which marked how diverse and distinct its generated images were, was 1.00 on average. The number of floating operations, or FLOPS, was 102 Gigaflops, which was less than the traditional GAN’s 198 Gigaflops. While limitations with time, computational memory, and mode collapse prevented the CVAE-GAN from recreating accurate memories, it still generated distinct image colors and general features. Future studies can build on existing network architectures and include more homogeneity in datasets. Ultimately, CVAE-GAN has the potential to advance new understandings of the brain and could elevate the memory reconstruction field.
Downloads
References or Bibliography
Fares, A., Zhong, S.-H., & Jiang, J. (n.d.). Brain-media: A dual conditioned and
lateralization supported GAN (DCLS-GAN) towards visualization of
image-evoked brain activities. ACM DL Digital Library. https://doi.org/
1145/3394171.3413858
Kavasidis, I., Palazzo, S., Spampinato, C., Giordano, D., & Shah, M. (n.d.).
Brain2Image. MM '17: ACM Multimedia Conference, 1809-1817.
https://doi.org/10.1145/3123266.3127907
Bao, J., Chen, D., Wen, F., Li, H., & Hua, G. (n.d.). CVAE-GAN: Fine-Grained
image generation through asymmetric training. arXiv:1703.10155.
Multi-task generative adversarial learning on geometrical shape reconstruction
from EEG brain signals. (2019). Iconip 2019. https://doi.org/10.48550/
arXiv.1907.13351
Shimizu, H., & Srinivasan, R. (2022). Improving classification and
reconstruction of imagined images from EEG signals. BioRxiv.
https://doi.org/10.1101/2022.06.01.494379
Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H. C.,
Sadato, N., & Kamitani, Y. (2008). Visual image reconstruction from human
brain activity using a combination of multiscale local image decoders.
Neuron, 60(5), 915-929. https://doi.org/10.1016/j.neuron.2008.11.004
Shen, G., Dwivedi, K., Majima, K., Horikawa, T., & Kamitani, Y. (2019).
End-to-End deep image reconstruction from human brain activity.
Frontiers in Computational Neuroscience, 13. https://doi.org/10.3389/
Fncom.2019.00021
Wakita, S., Orima, T., & Motoyoshi, I. (2021). Photorealistic reconstruction of
visual texture from EEG signals. Frontiers in Computational Neuroscience,
https://doi.org/10.3389/fncom.2021.754587
Ye, Z., Yao, L., Zhang, Y., & Gustin, S. (n.d.). See what you see:
Self-supervised cross-modal retrieval of visual stimuli from brain
activity. arXiv:2208.03666. https://doi.org/10.48550/arXiv.2208.03666
Zubarev, I., Vranou, G., & Parkkonen, L. (2022). MNEflow: Neural networks for
eeg/meg decoding and interpretation. SoftwareX, 17, 100951. https://doi.org/
1016/j.softx.2021.100951
Nemrodov, D., Niemeier, M., Patel, A., & Nestor, A. (2018). The neural dynamics
of facial identity processing: Insights from eeg-based pattern analysis and
image reconstruction. Eneuro, 5(1), ENEURO.0358-17.2018. https://doi.org/
1523/ENEURO.0358-17.2018
Rashkov, G., Bobe, A., Fastovets, D., & Komarova, M. (2019). Natural image
reconstruction from brain waves: A novel visual BCI system with native
feedback. bioRxiv. https://doi.org/10.1101/787101
Du, C., Du, C., & He, H. (n.d.). Sharing deep generative representation for
perceived image reconstruction from human brain activity. 2017
International Joint Conference on Neural Networks. https://doi.org/10.1109/
IJCNN.2017.7965968
Larsen, A. B. L., Sønderby, S. K., Larochelle, H., & Winther, O. (n.d.).
Autoencoding beyond pixels using a learned similarity metric.
arXiv:1512.09300. https://doi.org/10.48550/arXiv.1512.09300
Published
How to Cite
Issue
Section
Copyright (c) 2023 Jeremy Lu; Cathy Messenger
![Creative Commons License](http://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png)
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright holder(s) granted JSR a perpetual, non-exclusive license to distriute & display this article.