Skip to content

Latest commit

 

History

History
50 lines (37 loc) · 2.89 KB

README.md

File metadata and controls

50 lines (37 loc) · 2.89 KB
Raw

EEG-to-Text Decoding using LLMs and Real-Time Web App for inference

Our project leverages EEG (electroencephalography) technology and powerful large language models (LLMs) to decode brain signals into text and implement this in real-time. Our goal is to provide a communication tool for individuals with severe communication impairments.

Research Paper Implementation

We have implemented a research paper on Semantic-aware Contrastive Learning (C-SCL) for EEG-to-text decoding. Due to the large size of the model weights, they are available via a Google Drive link with all the codes for preprocessing and training.

Approach

Our approach involves using a preencoder trained with C-SCL to transform EEG signals into a representation that can be effectively processed by a pretrained BART model to generate natural language text.

Key Components

  • C-SCL Preencoder: The C-SCL technique is applied to train the preencoder, which helps in mapping EEG signals to a latent space that is semantically aware and suitable for language decoding.
  • Pretrained BART Model: We utilize a pretrained BART model, a transformer-based sequence-to-sequence model, to generate text from the latent representations produced by the preencoder.

Implementation Details

  • Preencoder Weights: The trained preencoder weights, obtained using different labeled training configurations, are available in the saved_models_cscl folder.
  • Complete Model Weights: The full model weights, including those for the BART model and the preencoder, can be found in the src/checkpoints directory. This directory contains the best checkpoints from the entire training process as well as the final model state.

Directory Structure

  • saved_models_cscl: Contains the weights of the preencoder trained with various configurations.
  • src/checkpoints: Includes the complete model checkpoints. You can find the best checkpoints after the entire training or the final model weights in this directory.

Getting Started with the web app

Prerequisites

  • Python (3.7 or higher recommended)
  • Node.js and npm (or yarn)
  • Muselsl (EEG streaming library): Install following their guide: https://github.com/alexandrebarachant/muse-lsl
  • A compatible EEG device that can stream data using Muselsl

Installation & Setup

  1. Download the Project: Clone or download this repository to your local machine.

  2. Install Dependencies:

    cd <project_directory>
npm install         #this installs main project dependencies
cd frontend
npm install         #this then installs frontend dependencies
cd ..

  3. Start your frontend :

    npm start

  4. Start your backend :

    cd backend
python neuro1.py