This Repository contains the OHIF Viewer with an Generative AI extension, that enables the user to input a text and ganerates a CT scan of the chest. The OHIF Viewer is a medical image viewer provided by the Open Health Imaging Foundation (OHIF). This extension requires an backend server to run the generative AI model (MedSyn) to convert text input into 3D CT scans.
Screenshot of Generative AI extension. Left: Findings and Impressions of original CT scan. Right: Enter prompt to generate CT and server Status, below already generated images.
- Yarn 1.17.3+
- Node 18+
- Docker
- Yarn Workspaces should be enabled on your machine:
yarn config set workspaces-experimental true
- To make inference with MedSyn model GPU with 40GB RAM required
- Clone this repository
git clone https://github.com/TomWartm/Viewers.git
- Navigate to the cloned project's directory
yarn install
to restore dependencies and link projects- Start backend server
yarn orthanc:up
- Start the Application with Orthanc as backend
yarn dev:orthanc
(in a new terminal)
- Clone the backend repository (on a machine with large GPU RAM)
git clone https://github.com/TomWartm/MedsynBackend
- Navigate to the cloned project's directory
- Install required python packages
conda env create --file environment.yml
- Actiave environment
conda activate medsyn-3-8
- Navigate to src folder
- Run flask server
python app.py
Add NIfTI files to the folder data/nifti
(some are available on our google drive) and use the notebook in backend/nifti_to_orthan.ipynb
to converti files into DICOM and upload to the Orthanc server.
The OHIF Medical Image Viewing Platform is maintained as a
monorepo
. This means that this repository, instead of containing a
single project, contains many projects. If you explore our project structure,
you'll see the following:
.
├── extensions #
│ ├── _example # Skeleton of example extension
│ ├── default # basic set of useful functionalities (datasources, panels, etc)
│ ├── cornerstone # image rendering and tools w/ Cornerstone3D
│ ├── cornerstone-dicom-sr # DICOM Structured Report rendering and export
│ ├── cornerstone-dicom-sr # DICOM Structured Report rendering and export
│ ├── cornerstone-dicom-seg # DICOM Segmentation rendering and export
│ ├── cornerstone-dicom-rt # DICOM RTSTRUCT rendering
│ ├── cornerstone-microscopy # Whole Slide Microscopy rendering
│ ├── dicom-pdf # PDF rendering
│ ├── dicom-video # DICOM RESTful Services
│ ├── measurement-tracking # Longitudinal measurement tracking
| ├── text-input-extension # generative ML model promting
│ ├── tmtv # Total Metabolic Tumor Volume (TMTV) calculation
|
│
├── modes #
│ ├── _example # Skeleton of example mode
│ ├── generative-ai # generative ML model promting
│ ├── basic-dev-mode # Basic development mode
│ ├── longitudinal # Longitudinal mode (measurement tracking)
│ ├── tmtv # Total Metabolic Tumor Volume (TMTV) calculation mode
│ └── microscopy # Whole Slide Microscopy mode
│
├── platform #
│ ├── core # Business Logic
│ ├── i18n # Internationalization Support
│ ├── ui # React component library
│ ├── docs # Documentation
│ └── viewer # Connects platform and extension projects
│
├── ... # misc. shared configuration
├── lerna.json # MonoRepo (Lerna) settings
├── package.json # Shared devDependencies and commands
└── README.md # This file
To manually load images into the Tool you can drag-and-drop with the Upload feature on the study overview page, opload directly to Orthanc server on its interface (http://localhost:8042/app/explorer.html
) or programmatically with python (check backend/nifti_to_orthanc.ipynb
)
Images are stored on a the Orthanc server you can open up the Interface running on http://localhost:8042/app/explorer.html
.