A Novel Approach to Optimize Memory Reconstruction Using Joint Multimodal Networks

Abstract

Memories play a crucial role in the human life experience. Whether they are used to make decisions or understand current situations, memories influence the past, present, and future. However, some memories are unreliable; in the court of law, the false memories of victims can cause inaccurate verdicts. To reconstruct human memories directly from the brain, recent studies have used electroencephalographic (EEG) brain signals in machine learning frameworks like generative adversarial networks (GANs), which generates new data from old ones. However, traditional GANs tend to produce the same images, an issue called mode collapse. Therefore, a conditional variational autoencoder-generative adversarial network (CVAE-GAN), which had not been used before in memory reconstruction, was developed to address GAN failures by jointly-training (1) an encoder and (2) GAN. CVAE-GAN correlated the MindBigData dataset’s EEG brain signals with the corresponding ImageNet dataset images and produced new memories of participants, such as pandas, humans, and fish. CVAE-GAN’s Inception Score, which marked how diverse and distinct its generated images were, was 1.00 on average. The number of floating operations, or FLOPS, was 102 Gigaflops, which was less than the traditional GAN’s 198 Gigaflops. While limitations with time, computational memory, and mode collapse prevented the CVAE-GAN from recreating accurate memories, it still generated distinct image colors and general features. Future studies can build on existing network architectures and include more homogeneity in datasets. Ultimately, CVAE-GAN has the potential to advance new understandings of the brain and could elevate the memory reconstruction field.