A Generative AI Extension for the OHIF Viewer

This Repository contains the OHIF Viewer with an Generative AI extension, that enables the user to input a text and ganerates a CT scan of the chest. The OHIF Viewer is a medical image viewer provided by the Open Health Imaging Foundation (OHIF). This extension requires an backend server to run the generative AI model (MedSyn) to convert text input into 3D CT scans.

---

Screenshot of Generative AI extension. Left: Findings and Impressions of original CT scan. Right: Enter prompt to generate CT and server Status, below already generated images.

Developing

Requirements

• Yarn 1.17.3+
• Node 18+
• Docker
• Yarn Workspaces should be enabled on your machine:
  • yarn config set workspaces-experimental true
• To make inference with MedSyn model GPU with 40GB RAM required

Getting Started

Run Application

1. Clone this repository
   • git clone https://github.com/TomWartm/Viewers.git
2. Navigate to the cloned project's directory
3. yarn install to restore dependencies and link projects
4. Start backend server yarn orthanc:up
5. Start the Application with Orthanc as backend yarn dev:orthanc (in a new terminal)

Run Backend

1. Clone the backend repository (on a machine with large GPU RAM)
   • git clone https://github.com/TomWartm/MedsynBackend
2. Navigate to the cloned project's directory
3. Install required python packages conda env create --file environment.yml
4. Actiave environment conda activate medsyn-3-8
5. Navigate to src folder
6. Run flask server python app.py

Add dummy Data

Add NIfTI files to the folder data/nifti (some are available on our google drive) and use the notebook in backend/nifti_to_orthan.ipynb to converti files into DICOM and upload to the Orthanc server.

Project

The OHIF Medical Image Viewing Platform is maintained as a monorepo. This means that this repository, instead of containing a single project, contains many projects. If you explore our project structure, you'll see the following:

.
├── extensions               #
│   ├── _example             # Skeleton of example extension
│   ├── default              # basic set of useful functionalities (datasources, panels, etc)
│   ├── cornerstone       # image rendering and tools w/ Cornerstone3D
│   ├── cornerstone-dicom-sr # DICOM Structured Report rendering and export
│   ├── cornerstone-dicom-sr # DICOM Structured Report rendering and export
│   ├── cornerstone-dicom-seg # DICOM Segmentation rendering and export
│   ├── cornerstone-dicom-rt # DICOM RTSTRUCT rendering
│   ├── cornerstone-microscopy # Whole Slide Microscopy rendering
│   ├── dicom-pdf # PDF rendering
│   ├── dicom-video # DICOM RESTful Services
│   ├── measurement-tracking # Longitudinal measurement tracking
|   ├── text-input-extension # generative ML model promting
│   ├── tmtv # Total Metabolic Tumor Volume (TMTV) calculation
|

│
├── modes                    #
│   ├── _example             # Skeleton of example mode
│   ├── generative-ai       # generative ML model promting
│   ├── basic-dev-mode       # Basic development mode
│   ├── longitudinal         # Longitudinal mode (measurement tracking)
│   ├── tmtv       # Total Metabolic Tumor Volume (TMTV) calculation mode
│   └── microscopy          # Whole Slide Microscopy mode
│
├── platform                 #
│   ├── core                 # Business Logic
│   ├── i18n                 # Internationalization Support
│   ├── ui                   # React component library
│   ├── docs                 # Documentation
│   └── viewer               # Connects platform and extension projects
│
├── ...                      # misc. shared configuration
├── lerna.json               # MonoRepo (Lerna) settings
├── package.json             # Shared devDependencies and commands
└── README.md                # This file

Howto

Manually load images

To manually load images into the Tool you can drag-and-drop with the Upload feature on the study overview page, opload directly to Orthanc server on its interface (http://localhost:8042/app/explorer.html) or programmatically with python (check backend/nifti_to_orthanc.ipynb)

Backend

Images are stored on a the Orthanc server you can open up the Interface running on http://localhost:8042/app/explorer.html.

License