+
+Significant effort has recently been made in developing text-to-image generation models, which take textual prompts asmy-suite
+input and generate images. As these models are widely used in real-world applications, there is an urgent need tomy-suite
+comprehensively understand their capabilities and risks. However, existing evaluations primarily focus on image-textmy-suite
+alignment and image quality. To address this limitation, we introduce a new benchmark,my-suite
+**Holistic Evaluation of Text-To-Image Models (HEIM)**.
+
+We identify 12 different aspects that are important in real-world model deployment, including:
+
+- image-text alignment
+- image quality
+- aesthetics
+- originality
+- reasoning
+- knowledge
+- bias
+- toxicity
+- fairness
+- robustness
+- multilinguality
+- efficiency
+
+By curating scenarios encompassing these aspects, we evaluate state-of-the-art text-to-image models using this benchmark.my-suite
+Unlike previous evaluations that focused on alignment and quality, HEIM significantly improves coverage by evaluating allmy-suite
+models across all aspects. Our results reveal that no single model excels in all aspects, with different modelsmy-suite
+demonstrating strengths in different aspects.
+
+## References
+
+- [Leaderboard](https://crfm.stanford.edu/helm/heim/latest/)
+- [Paper](https://arxiv.org/abs/2311.04287)
+
+## Installation
+
+First, follow the [installation instructions](installation.md) to install the base HELM Python page.
+
+To install the additional dependencies to run HEIM, run:
+
+```sh
+pip install "crfm-helm[heim]"
```
-echo 'entries: [{description: "mscoco:model=huggingface/stable-diffusion-v1-4", priority: 1}]' > run_entries.conf
+
+Some models (e.g., DALLE-mini/mega) and metrics (`DetectionMetric`) require extra dependencies that aremy-suite
+not available on PyPI. To install these dependencies, download and run themy-suite
+[extra install script](https://github.com/stanford-crfm/helm/blob/main/install-heim-extras.sh):
+
+```sh
+bash install-heim-extras.sh
```
-2. Run the benchmark with certain number of instances (e.g., 10 instances):
-`helm-run --conf-paths run_entries.conf --suite heim_v1 --max-eval-instances 10`
-Examples of run specs configuration files can be found [here](https://github.com/stanford-crfm/helm/tree/main/src/helm/benchmark/presentation).
-We used [this configuration file](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_entries_heim.conf)
-to produce results of the paper.
+## Getting Started
+
+The following is an example of evaluating [Stable Diffusion v1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4) on the [MS-COCO scenario](https://github.com/stanford-crfm/heim/blob/main/src/helm/benchmark/scenarios/image_generation/mscoco_scenario.py) using 10 instances.
+
+```sh
+helm-run --run-entries mscoco:model=huggingface/stable-diffusion-v1-4 --suite my-heim-suite --max-eval-instances 10
+```
+
+## Reproducing the Leaderboard
+
+To reproduce the [entire HEIM leaderboard](https://crfm.stanford.edu/helm/heim/latest/), refer to the instructions for HEIM on the [Reproducing Leaderboards](reproducing_leaderboards.md) documentation.
diff --git a/docs/huggingface_models.md b/docs/huggingface_models.md
index 4511cce2fa..f781e4ae2a 100644
--- a/docs/huggingface_models.md
+++ b/docs/huggingface_models.md
@@ -1,18 +1,25 @@
# Hugging Face Model Hub Integration
-HELM can be used to evaluate `AutoModelForCausalLM` models (e.g. [`BioMedLM`](https://huggingface.co/stanford-crfm/BioMedLM)) on [Hugging Face Model Hub](https://huggingface.co/models).
+HELM can be used to evaluate `AutoModelForCausalLM` models (e.g. [`BioMedLM`](https://huggingface.co/stanford-crfm/BioMedLM)) on [Hugging Face Model Hub](https://huggingface.co/models) or local disk. Note that only `AutoModelForCausalLM` models are supported; other classes such as `AutoModelForSeq2SeqLM` may be supported in the future.
+
+## Using `model_deployments.yaml`
+
+You can add Hugging Face models using the method discussed in [Adding New Models](adding_new_models.md). This can be used for both models on Hugging Face Hub and local disk. Please refer to that page for instructions for how to do so.
+
+## Using command-line flags
+
+In some cases, you can use command-line flags with `helm-run` to evaluating Hugging Face models. This provides a more convenient way to use Hugging Face models that does not require configuration files.
To use `AutoModelForCausalLM` models from Hugging Face Model Hub, add the Hugging Face model IDs to the `--enable-huggingface-models` flags to `helm-run`. This will make the corresponding Hugging Face models available to use in your run spec descriptions. In the run spec description, use the Hugging Face model ID as the model name.
To use a revision of a model other than the default main revision, append a `@` followed by the revision name to the model ID passed to the `--enable-huggingface-models` flag.
-Current restrictions:
+Current restrictions with command-line flags:
-- Only `AutoModelForCausalLM` is supported; other classes such as `AutoModelForSeq2SeqLM` may be supported in the future.
- Models without a namespace are not supported (e.g. `bert-base-uncased`).
-- Models at local file paths are not supported.
+- The model must have `model_max_length` set in the tokenizer configuration.
-Examples:
+Example model on Hugging Face Hub:
```bash
# Run boolq on stanford-crfm/BioMedLM at the default main revision
@@ -30,16 +37,13 @@ helm-run \
--max-eval-instances 10
```
-To use Optimum Intel, add `--openvino` flag to `helm-run`. Optimum Intel provides a simple interface to optimize Transformer models and convert them to OpenVINO™ Intermediate Representation format to accelerate end-to-end pipelines on Intel® architectures using OpenVINO™ runtime. It runs the model on the CPU.
-
-Examples:
+Example model on local disk:
```bash
-# Run boolq on stanford-crfm/BioMedLM optimized by Optimum Intel OpenNIVO
+# Run boolq on stanford-crfm/BioMedLM at the default main revision
helm-run \
- --run-entries boolq:model=stanford-crfm/BioMedLM \
- --enable-huggingface-models stanford-crfm/BioMedLM \
+ --run-entries boolq:model=your-org/your-model \
+ --enable-local-huggingface-models path/to/your-org/your-model \
--suite v1 \
- --max-eval-instances 10 \
- --openvino
+ --max-eval-instances 10
```
diff --git a/docs/index.md b/docs/index.md
index 0071f90eed..5075f628af 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -18,6 +18,11 @@ To add new models and scenarios, refer to the Developer Guide's chapters:
- [Developer Setup](developer_setup.md)
- [Code Structure](code.md)
+## Papers
-We also support evaluating text-to-image models as introduced in **Holistic Evaluation of Text-to-Image Models (HEIM)**
-([paper](https://arxiv.org/abs/2311.04287), [website](https://crfm.stanford.edu/heim/latest)).
+This repository contains code used to produce results for the following papers:
+
+- **Holistic Evaluation of Vision-Language Models (VHELM)** - [paper](https://arxiv.org/abs/2410.07112), [leaderboard](https://crfm.stanford.edu/helm/vhelm/latest/), [documentation](https://crfm-helm.readthedocs.io/en/latest/vhelm/)
+- **Holistic Evaluation of Text-To-Image Models (HEIM)** - [paper](https://arxiv.org/abs/2311.04287), [leaderboard](https://crfm.stanford.edu/helm/heim/latest/), [documentation](https://crfm-helm.readthedocs.io/en/latest/heim/)
+
+The HELM Python package can be used to reproduce the published model evaluation results from these papers. To get started, refer to the documentation links above for the corresponding paper, or the [main Reproducing Leaderboards documentation](https://crfm-helm.readthedocs.io/en/latest/reproducing_leaderboards/).
\ No newline at end of file
diff --git a/docs/installation.md b/docs/installation.md
index 21e1d3358c..3af5d55f6d 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -35,18 +35,9 @@ Within this virtual environment, run:
pip install crfm-helm
```
-### For HEIM (text-to-image evaluation)
+## Install Multimodal Support
-To install the additional dependencies to run HEIM, run:
+Additional steps are required for multimodal evaluations:
-```
-pip install "crfm-helm[heim]"
-```
-
-Some models (e.g., DALLE-mini/mega) and metrics (`DetectionMetric`) require extra dependencies that are
-not available on PyPI. To install these dependencies, download and run the
-[extra install script](https://github.com/stanford-crfm/helm/blob/main/install-heim-extras.sh):
-
-```
-bash install-heim-extras.sh
-```
+- **HEIM (Text-to-image Model Evaluation)** - to install the additional dependencies to run HEIM (text-to-image evaluation), refer to the [HEIM documentation](heim.md).
+- **VHELM (Vision-Language Models)** - To install the additional dependencies to run VHELM (Vision-Language Models), refer to the [VHELM documentation](vhelm.md).
diff --git a/docs/mkdocs_macros.py b/docs/mkdocs_macros.py
index 3bd3d05c8e..12a669a191 100644
--- a/docs/mkdocs_macros.py
+++ b/docs/mkdocs_macros.py
@@ -1,6 +1,6 @@
from typing import Dict, List, Type
-from helm.benchmark.model_metadata_registry import ALL_MODELS_METADATA, TEXT_MODEL_TAG, CODE_MODEL_TAG, ModelMetadata
+from helm.benchmark.model_metadata_registry import ALL_MODELS_METADATA, TEXT_MODEL_TAG, CODE_MODEL_TAG, DEPRECATED_MODEL_TAG, ModelMetadata
from helm.benchmark.run_expander import RUN_EXPANDERS, RunExpander
@@ -11,6 +11,8 @@ def models_by_organization_with_tag(tag: str) -> Dict[str, List[ModelMetadata]]:
result: Dict[str, List[ModelMetadata]] = {}
for model_metadata in ALL_MODELS_METADATA:
+ if DEPRECATED_MODEL_TAG in model_metadata.tags:
+ continue
if tag not in model_metadata.tags:
continue
if model_metadata.creator_organization == "simple":
diff --git a/docs/models.md b/docs/models.md
index ce4ed6a06f..f87d62d667 100644
--- a/docs/models.md
+++ b/docs/models.md
@@ -18,6 +18,47 @@
{% endfor %}
{% endfor %}
-## HEIM (text-to-image evaluation)
+## Vision-Language Models
-For a list of text-to-image models, please visit the [models page](https://crfm.stanford.edu/heim/latest/?models) of the HEIM results website.
+{% for organization, models in models_by_organization_with_tag("VISION_LANGUAGE_MODEL_TAG").items() %}
+
+### {{ organization }}
+
+{% for model in models %}
+
+#### {{ model.display_name }} — `{{ model.name }}`
+
+{{ model.description }}
+
+{% endfor %}
+{% endfor %}
+
+## Text-to-image Models
+
+{% for organization, models in models_by_organization_with_tag("TEXT_TO_IMAGE_MODEL_TAG").items() %}
+
+### {{ organization }}
+
+{% for model in models %}
+
+#### {{ model.display_name }} — `{{ model.name }}`
+
+{{ model.description }}
+
+{% endfor %}
+{% endfor %}
+
+## Audio-Language Models
+
+{% for organization, models in models_by_organization_with_tag("AUDIO_LANGUAGE_MODEL_TAG").items() %}
+
+### {{ organization }}
+
+{% for model in models %}
+
+#### {{ model.display_name }} — `{{ model.name }}`
+
+{{ model.description }}
+
+{% endfor %}
+{% endfor %}
\ No newline at end of file
diff --git a/docs/proxy-server.md b/docs/proxy_server.md
similarity index 88%
rename from docs/proxy-server.md
rename to docs/proxy_server.md
index 818b98b72f..42dadf9b84 100644
--- a/docs/proxy-server.md
+++ b/docs/proxy_server.md
@@ -1,4 +1,6 @@
-# Proxy Access to Language Models
+# Proxy Server
+
+**Warning** — The document is stale. The information below may be outdated and incorrect. Please proceed with caution!
We provide a single unified entry point into accessing large language models
(e.g., GPT-3, Jurassic). This provides both a web interface and a REST API.
diff --git a/docs/quick_start.md b/docs/quick_start.md
index 3a1fb2f561..c9785256f1 100644
--- a/docs/quick_start.md
+++ b/docs/quick_start.md
@@ -3,24 +3,14 @@
Run the following:
```
-# Create a run specs configuration
-echo 'entries: [{description: "mmlu:subject=philosophy,model=openai/gpt2", priority: 1}]' > run_entries.conf
-
# Run benchmark
-helm-run --conf-paths run_entries.conf --suite v1 --max-eval-instances 10
+helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10
# Summarize benchmark results
-helm-summarize --suite v1
+helm-summarize --suite my-suite
# Start a web server to display benchmark results
-helm-server
+helm-server --suite my-suite
```
Then go to http://localhost:8000/ in your browser.
-
-
-## Next steps
-
-Click [here](get_helm_rank.md) to find out how to run the full benchmark and get your model's leaderboard rank.
-
-For the quick start page for HEIM, visit [here](heim.md).
\ No newline at end of file
diff --git a/docs/reproducing_leaderboards.md b/docs/reproducing_leaderboards.md
index 77b517911b..72b0336edf 100644
--- a/docs/reproducing_leaderboards.md
+++ b/docs/reproducing_leaderboards.md
@@ -94,6 +94,16 @@ export NUM_EVAL_INSTANCES=1000
export PRIORITY=2
```
+### VHELM >=v2.0.0
+
+```bash
+export RUN_ENTRIES_CONF_PATH=run_entries_vhelm.conf
+export SCHEMA_PATH=schema_vhelm.yaml
+export NUM_TRAIN_TRIALS=1
+export NUM_EVAL_INSTANCES=1000
+export PRIORITY=2
+```
+
### AIR-Bench
```bash
diff --git a/docs/run_entries_configuration_files.md b/docs/run_entries_configuration_files.md
index 09cb069ea6..22d92a917e 100644
--- a/docs/run_entries_configuration_files.md
+++ b/docs/run_entries_configuration_files.md
@@ -28,7 +28,7 @@ You would then use this file with `helm-run` with this command:
helm-run --conf-file tutorial_run_entries.conf --suite tutorial --max-eval-instances 10
```
-## Model run expander wildcards
+## Model Run Expander Wildcards
It is very common to use run entries configuration file with a **model run expander wildcards** e.g. `model=text`. For instance,
diff --git a/docs/tutorial.md b/docs/tutorial.md
index c000825d9a..77620560e7 100644
--- a/docs/tutorial.md
+++ b/docs/tutorial.md
@@ -8,34 +8,26 @@ We will run two runs using the `mmlu` scenario on the `openai/gpt2` model. The `
`helm-run` is a command line tool for running benchmarks.
-To run this benchmark using the HELM command-line tools, we need to specify **run spec descriptions** that describes the desired runs. For this example, the run spec descriptions are `mmlu:subject=anatomy,model=openai/gpt2` (for anatomy) and `mmlu:subject=philosophy,model=openai/gpt2` (for philosophy).
+To run this benchmark using the HELM command-line tools, we need to specify **run entries** that describes the desired runs. For this example, the run entries are `mmlu:subject=anatomy,model=openai/gpt2` (for anatomy) and `mmlu:subject=philosophy,model=openai/gpt2` (for philosophy).
-Next, we need to create a **run spec configuration file** containing these run spec descriptions. A run spec configuration file is a text file containing `RunEntries` serialized to JSON, where each entry in `RunEntries` contains a run spec description. The `description` field of each entry should be a **run spec description**. Create a text file named `run_entries.conf` with the following contents:
+We will now use `helm-run` to execute the runs. Run this command:
-```
-entries: [
- {description: "mmlu:subject=anatomy,model=openai/gpt2", priority: 1},
- {description: "mmlu:subject=philosophy,model=openai/gpt2", priority: 1},
-]
-```
-
-We will now use `helm-run` to execute the runs that have been specified in this run spec configuration file. Run this command:
-
-```
-helm-run --conf-paths run_entries.conf --suite v1 --max-eval-instances 10
+```sh
+helm-run --run-entries mmlu:subject=anatomy,model=openai/gpt2 mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10
```
-The meaning of the additional arguments are as follows:
+The meaning of the arguments are as follows:
+- `--run-entries` specifies the run entries from the desired runs.
- `--suite` specifies a subdirectory under the output directory in which all the output will be placed.
-- `--max-eval-instances` limits evaluation to only the first *N* inputs (i.e. instances) from the benchmark.
+- `--max-eval-instances` limits evaluation to only *N* instances (i.e. items) from the benchmark, using a randomly shuffled order of instances.
`helm-run` creates an environment directory environment and an output directory by default.
- The environment directory is `prod_env/` by default and can be set using `--local-path`. Credentials for making API calls should be added to a `credentials.conf` file in this directory.
- The output directory is `benchmark_output/` by default and can be set using `--output-path`.
-After running this command, navigate to the `benchmark_output/runs/v1/` directory. This should contain a two sub-directories named `mmlu:subject=anatomy,model=openai_gpt2` and `mmlu:subject=philosophy,model=openai_gpt2`. Note that the names of these sub-directories is based on the run spec descriptions we used earlier, but with `/` replaced with `_`.
+After running this command, navigate to the `benchmark_output/runs/my-suite/` directory. This should contain a two sub-directories named `mmlu:subject=anatomy,model=openai_gpt2` and `mmlu:subject=philosophy,model=openai_gpt2`. Note that the names of these sub-directories is based on the run entries we used earlier, but with `/` replaced with `_`.
Each output sub-directory will contain several JSON files that were generated during the corresponding run:
@@ -45,60 +37,35 @@ Each output sub-directory will contain several JSON files that were generated du
- `per_instance_stats.json` contains a serialized list of `PerInstanceStats`, which contains the statistics produced for the metrics for each instance (i.e. input).
- `stats.json` contains a serialized list of `PerInstanceStats`, which contains the statistics produced for the metrics, aggregated across all instances (i.e. inputs).
-`helm-run` provides additional arguments that can be used to filter out `--models-to-run`, `--groups-to-run` and `--priority`. It can be convenient to create a large `run_entries.conf` file containing every run spec description of interest, and then use these flags to filter down the RunSpecs to actually run. As an example, the main `run_specs.conf` file used for the HELM benchmarking paper can be found [here](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_specs.conf).
-
-**Using model or model_deployment:** Some models have several deployments (for exmaple `eleutherai/gpt-j-6b` is deployed under `huggingface/gpt-j-6b`, `gooseai/gpt-j-6b` and `together/gpt-j-6b`). Since the results can differ depending on the deployment, we provide a way to specify the deployment instead of the model. Instead of using `model=eleutherai/gpt-g-6b`, use `model_deployment=huggingface/gpt-j-6b`. If you do not, a deployment will be arbitrarily chosen. This can still be used for models that have a single deployment and is a good practice to follow to avoid any ambiguity.
-
## Using `helm-summarize`
The `helm-summarize` reads the output files of `helm-run` and computes aggregate statistics across runs. Run the following:
-```
-helm-summarize --suite v1
+```sh
+helm-summarize --suite my-suite
```
-This reads the pre-existing files in `benchmark_output/runs/v1/` that were written by `helm-run` previously, and writes the following new files back to `benchmark_output/runs/v1/`:
+This reads the pre-existing files in `benchmark_output/runs/my-suite/` that were written by `helm-run` previously, and writes the following new files back to `benchmark_output/runs/my-suite/`:
- `summary.json` contains a serialized `ExecutiveSummary` with a date and suite name.
-- `run_specs.json` contains the run spec descriptions for all the runs.
+- `run_specs.json` contains the run entries for all the runs.
- `runs.json` contains serialized list of `Run`, which contains the run path, run spec and adapter spec and statistics for each run.
- `groups.json` contains a serialized list of `Table`, each containing information about groups in a group category.
- `groups_metadata.json` contains a list of all the groups along with a human-readable description and a taxonomy.
-Additionally, for each group and group-relavent metric, it will output a pair of files: `benchmark_output/runs/v1/groups/latex/
+
{/* Image section */}
{/* Leaderboard section */}
-
-
-
diff --git a/helm-frontend/src/components/InstanceData.tsx b/helm-frontend/src/components/InstanceData.tsx
index 36d763eb52..3d51352761 100644
--- a/helm-frontend/src/components/InstanceData.tsx
+++ b/helm-frontend/src/components/InstanceData.tsx
@@ -20,14 +20,8 @@ export default function InstanceData({
predictions,
metricFieldMap,
}: Props) {
- const renderInstanceId = (instance: Instance): string => {
- return instance.perturbation === undefined
- ? `Instance id: ${instance.id} [split: ${instance.split}]`
- : `Instance id: ${instance.id} [split: ${instance.split}][perturbation: ${instance.perturbation.name}]`;
- };
return (
-
-
-
-
-
+
+
+
+
+
-
([]);
+ const [displayPredictionsMap, setDisplayPredictionsMap] = useState<
+ undefined | Record>
+ >();
+ const [displayRequestsMap, setDisplayRequestsMap] = useState<
+ undefined | Record>
+ >();
+ const [currentInstancesPage, setCurrentInstancesPage] = useState(1);
+
+ useEffect(() => {
+ const controller = new AbortController();
+ async function fetchData() {
+ const signal = controller.signal;
+
+ const [instancesResp, displayPredictions, displayRequests] =
+ await Promise.all([
+ getInstances(runName, signal, suite),
+ getDisplayPredictionsByName(runName, signal, suite),
+ getDisplayRequestsByName(runName, signal, suite),
+ ]);
+ setInstances(instancesResp);
+
+ const tempDisplayRequestsMap: {
+ [key: string]: { [key: string]: DisplayRequest[] };
+ } = {};
+ displayRequests.forEach((displayRequest) => {
+ const instanceId = displayRequest.instance_id;
+ const perturbationName = displayRequest.perturbation?.name || "";
+ if (tempDisplayRequestsMap[instanceId] === undefined) {
+ tempDisplayRequestsMap[instanceId] = {};
+ }
+ if (
+ tempDisplayRequestsMap[instanceId][perturbationName] === undefined
+ ) {
+ tempDisplayRequestsMap[instanceId][perturbationName] = [];
+ }
+ tempDisplayRequestsMap[instanceId][perturbationName].push(
+ displayRequest,
+ );
+ });
+ setDisplayRequestsMap(tempDisplayRequestsMap);
+
+ const tempDisplayPredictionsMap: {
+ [key: string]: { [key: string]: DisplayPrediction[] };
+ } = {};
+ displayPredictions.forEach((displayPrediction) => {
+ const instanceId = displayPrediction.instance_id;
+ const perturbationName = displayPrediction.perturbation?.name || "";
+ if (tempDisplayPredictionsMap[instanceId] === undefined) {
+ tempDisplayPredictionsMap[instanceId] = {};
+ }
+ if (
+ tempDisplayPredictionsMap[instanceId][perturbationName] === undefined
+ ) {
+ tempDisplayPredictionsMap[instanceId][perturbationName] = [];
+ }
+ tempDisplayPredictionsMap[instanceId][perturbationName].push(
+ displayPrediction,
+ );
+ });
+ setDisplayPredictionsMap(tempDisplayPredictionsMap);
+ }
+
+ void fetchData();
+
+ return () => controller.abort();
+ }, [runName, suite]);
+
+ const pagedInstances = instances.slice(
+ (currentInstancesPage - 1) * INSTANCES_PAGE_SIZE,
+ (currentInstancesPage - 1) * INSTANCES_PAGE_SIZE + INSTANCES_PAGE_SIZE,
+ );
+ const totalInstancesPages = Math.ceil(instances.length / INSTANCES_PAGE_SIZE);
+
+ // Handle scrolling to anchored instance
+ useEffect(() => {
+ const anchoredInstance = searchParams.get("instance");
+ if (
+ anchoredInstance &&
+ !window.helmHasScrolledToInstance &&
+ pagedInstances.length > 0
+ ) {
+ // Find the index of the anchored instance
+ const instanceIndex = pagedInstances.findIndex(
+ (i) => i.id === anchoredInstance,
+ );
+ if (instanceIndex === -1) return;
+
+ // Wait for the DOM to be updated with the correct page
+ requestAnimationFrame(() => {
+ const element = document.getElementById(`instance-${anchoredInstance}`);
+ if (element) {
+ element.scrollIntoView({ behavior: "smooth" });
+ }
+ });
+
+ window.helmHasScrolledToInstance = true;
+ }
+ }, [searchParams, currentInstancesPage, setSearchParams, pagedInstances]);
+
+ const renderInstanceId = (instance: Instance): string => {
+ return instance.perturbation === undefined
+ ? `Instance id: ${instance.id} [split: ${instance.split}]`
+ : `Instance id: ${instance.id} [split: ${instance.split}][perturbation: ${instance.perturbation.name}]`;
+ };
+
+ if (displayPredictionsMap === undefined || displayRequestsMap === undefined) {
+ return {
+ const nextInstancePage = Math.min(
+ currentInstancesPage + 1,
+ totalInstancesPages,
+ );
+ setCurrentInstancesPage(nextInstancePage);
+ searchParams.set("instancesPage", String(nextInstancePage));
+ setSearchParams(searchParams);
+ }}
+ onPrevPage={() => {
+ const prevInstancePage = Math.max(currentInstancesPage - 1, 1);
+ setCurrentInstancesPage(prevInstancePage);
+ searchParams.set("instancesPage", String(prevInstancePage));
+ setSearchParams(searchParams);
+ }}
+ currentPage={currentInstancesPage}
+ totalPages={totalInstancesPages}
+ />
+
+ );
+}
diff --git a/helm-frontend/src/components/Landing/Image2StructLanding.tsx b/helm-frontend/src/components/Landing/Image2StructLanding.tsx
index d4315d6f67..5686b26fc2 100644
--- a/helm-frontend/src/components/Landing/Image2StructLanding.tsx
+++ b/helm-frontend/src/components/Landing/Image2StructLanding.tsx
@@ -10,7 +10,10 @@ export default function Image2StructLanding() {
Extracting Structured Information from Images
{renderInstanceId(instance)}
+Input
{instance.input.multimedia_content !== undefined ? (
+ {pagedInstances.map((instance, idx) => (
+
+
+
+ ))}
+
+
+ {renderInstanceId(instance)}
+ +
-
+
Paper
We introduce 3 tasks:
- - LaTex: equations, tables, plots and algorithms form ArXiV papers + LaTex: equations, tables, plots and algorithms from ArXiV papers.
- - Webpages: pages from GitHub written in HTML, CSS and Javascript, + Webpages: pages from GitHub written in HTML, CSS and Javascript. ... -
- - Music sheets: crops of measures from music sheets from IMSLP - +
- Music sheets: crops of measures from IMSLP music sheets.
diff --git a/helm-frontend/src/components/Landing/SafetyLanding.tsx b/helm-frontend/src/components/Landing/SafetyLanding.tsx
index 5e2d43be45..69714378ef 100644
--- a/helm-frontend/src/components/Landing/SafetyLanding.tsx
+++ b/helm-frontend/src/components/Landing/SafetyLanding.tsx
@@ -1,4 +1,5 @@
import MiniLeaderboard from "@/components/MiniLeaderboard";
+import helmSafety from "@/assets/helm-safety.png";
export default function SafetyLanding() {
return (
@@ -7,6 +8,12 @@ export default function SafetyLanding() {
+ 
preview
+ ) : indexOfCurrentVersion === 0 ? (
+ latest
+ ) : (
+ stale
+ );
return (
Language models demonstrate powerful capabilities and pose significant risks. Given their widespread deployment, standardized @@ -20,12 +27,12 @@ export default function SafetyLanding() { closed models.
- {/*
Blog Post
- */}
+
Full Leaderboard
diff --git a/helm-frontend/src/components/MediaObjectDisplay.tsx b/helm-frontend/src/components/MediaObjectDisplay.tsx
index 469fcd1bd7..a92b131886 100644
--- a/helm-frontend/src/components/MediaObjectDisplay.tsx
+++ b/helm-frontend/src/components/MediaObjectDisplay.tsx
@@ -21,6 +21,20 @@ export default function MediaObjectDisplay({ mediaObject }: Props) {
);
+ } else if (mediaObject.content_type.includes("audio")) {
+ if (mediaObject.location === undefined) {
+ return null;
+ }
+ const url = getBenchmarkEndpoint(
+ mediaObject.location
+ .replace("benchmark_output/", "")
+ .replace("prod_env/", "../"),
+ );
+ return (
+
+
+
+ );
} else {
if (
mediaObject.text &&
diff --git a/helm-frontend/src/components/NavDropdown.tsx b/helm-frontend/src/components/NavDropdown.tsx
index d1b8ae07aa..486edd593e 100644
--- a/helm-frontend/src/components/NavDropdown.tsx
+++ b/helm-frontend/src/components/NavDropdown.tsx
@@ -9,6 +9,22 @@ function NavDropdown() {
ProjectMetadata | undefined
>();
+ useEffect(() => {
+ if (
+ currProjectMetadata &&
+ currProjectMetadata.title &&
+ currProjectMetadata.title !== "All Leaderboards"
+ ) {
+ const titlePrefix =
+ currProjectMetadata.title === "Lite" ||
+ currProjectMetadata.title === "Classic"
+ ? "HELM " + currProjectMetadata.title
+ : currProjectMetadata.title;
+ document.title =
+ titlePrefix + " - Holistic Evaluation of Language Models (HELM)";
+ }
+ }, [currProjectMetadata]);
+
useEffect(() => {
fetch(
"https://raw.githubusercontent.com/stanford-crfm/helm/main/helm-frontend/project_metadata.json",
diff --git a/helm-frontend/src/components/ReleaseDropdown.tsx b/helm-frontend/src/components/ReleaseDropdown.tsx
index 7e648b56ea..6258de2ee3 100644
--- a/helm-frontend/src/components/ReleaseDropdown.tsx
+++ b/helm-frontend/src/components/ReleaseDropdown.tsx
@@ -1,3 +1,4 @@
+import { Badge } from "@tremor/react";
import { useEffect, useState } from "react";
import getReleaseSummary from "@/services/getReleaseSummary";
import ReleaseSummary from "@/types/ReleaseSummary";
@@ -40,13 +41,6 @@ function ReleaseDropdown() {
});
}, []);
- function getReleases(): string[] {
- return currProjectMetadata !== undefined &&
- currProjectMetadata.releases !== undefined
- ? currProjectMetadata.releases
- : ["v1.0.0"];
- }
-
useEffect(() => {
const controller = new AbortController();
async function fetchData() {
@@ -58,20 +52,34 @@ function ReleaseDropdown() {
return () => controller.abort();
}, []);
- const releases = getReleases();
+ const releases =
+ currProjectMetadata !== undefined &&
+ currProjectMetadata.releases !== undefined
+ ? currProjectMetadata.releases
+ : ["v1.0.0"];
- if (!summary.release && !summary.suite) {
+ const currentVersion = summary.release || summary.suite || null;
+
+ if (!currentVersion) {
return null;
}
- const releaseInfo = `Release ${summary.release || summary.suite} (${
- summary.date
- })`;
+ const releaseInfo = `Release ${currentVersion} (${summary.date})`;
if (releases.length <= 1) {
return {releaseInfo}
;
}
+ const indexOfCurrentVersion = releases.indexOf(currentVersion);
+
+ const badge =
+ indexOfCurrentVersion < 0 ? (
+
- {releaseInfo}
+ {releaseInfo} {badge}
-
{releases.map((release) => (
diff --git a/helm-frontend/src/components/RunMetrics.tsx b/helm-frontend/src/components/RunMetrics.tsx
new file mode 100644
index 0000000000..5428571493
--- /dev/null
+++ b/helm-frontend/src/components/RunMetrics.tsx
@@ -0,0 +1,131 @@
+import { useEffect, useState } from "react";
+import { useSearchParams } from "react-router-dom";
+import getStatsByName from "@/services/getStatsByName";
+import type Stat from "@/types/Stat";
+import type MetricFieldMap from "@/types/MetricFieldMap";
+import Loading from "@/components/Loading";
+import Pagination from "@/components/Pagination";
+import StatNameDisplay from "@/components/StatNameDisplay";
+
+const METRICS_PAGE_SIZE = 50;
+const TABLE_COLUMN_NAMES = [
+ "name",
+ "mean",
+ "min",
+ "max",
+ "sum",
+ "sum_squared",
+ "variance",
+ "stddev",
+];
+
+interface Props {
+ runName: string;
+ suite: string;
+ metricFieldMap: MetricFieldMap;
+}
+
+export default function Run({ runName, suite, metricFieldMap }: Props) {
+ const [searchParams, setSearchParams] = useSearchParams();
+ const [stats, setStats] = useState
+ {/* Search bar */}
+ {
+ const nextMetricsPage = Math.min(
+ currentMetricsPage + 1,
+ totalMetricsPages,
+ );
+ setCurrentMetricsPage(nextMetricsPage);
+ searchParams.set("metricsPage", String(nextMetricsPage));
+ setSearchParams(searchParams);
+ }}
+ onPrevPage={() => {
+ const prevMetricsPage = Math.max(currentMetricsPage - 1, 1);
+ setCurrentMetricsPage(prevMetricsPage);
+ searchParams.set("metricsPage", String(prevMetricsPage));
+ setSearchParams(searchParams);
+ }}
+ currentPage={currentMetricsPage}
+ totalPages={totalMetricsPages}
+ />
+
+ );
+}
diff --git a/helm-frontend/src/components/VHELMLanding.tsx b/helm-frontend/src/components/VHELMLanding.tsx
index 6369df7854..4621d9961d 100644
--- a/helm-frontend/src/components/VHELMLanding.tsx
+++ b/helm-frontend/src/components/VHELMLanding.tsx
@@ -17,10 +17,13 @@ export default function VHELMLanding() {
Paper
+
+ Leaderboard
+
(0);
const [runSpec, setRunSpec] = useState
+ setSearchTerm(e.target.value)}
+ />
+
+
+
+
+
+
+ | {key} | + ))} +|
|---|---|
| {value} | ; + } else { + return ( +
+ |
+ );
+ }
+ })}
+
@@ -278,117 +179,17 @@ export default function Run() {
{activeTab === 0 ? (
- <>
-
- {pagedInstances.map((instance, idx) => (
-
-
- {/* Search bar */}
- {
- const nextMetricsPage = Math.min(
- currentMetricsPage + 1,
- totalMetricsPages,
- );
- setCurrentMetricsPage(nextMetricsPage);
- searchParams.set("metricsPage", String(nextMetricsPage));
- setSearchParams(searchParams);
- }}
- onPrevPage={() => {
- const prevMetricsPage = Math.max(currentMetricsPage - 1, 1);
- setCurrentMetricsPage(prevMetricsPage);
- searchParams.set("metricsPage", String(prevMetricsPage));
- setSearchParams(searchParams);
- }}
- currentPage={currentMetricsPage}
- totalPages={totalMetricsPages}
- />
-
+
- setSearchTerm(e.target.value)}
- />
-
-
-
-
-
-
- | {key} | - ))} -|
|---|---|
|
- |
- );
- }
- return {value} | ; - })} -