Qiaolong Cai*, Zhaowei Wang*, Shizhe Diao, James Kwok, Yangqiu Song
* Equal contribution
- [2024/12/22] Initial code released.
- [2024/08/25] Paper released on arXiv.
This is the code repository for the paper: CodeGraph: Enhancing Graph Reasoning of LLMs with Code.
We introduce CodeGraph, a method for reasoning about basic graph problems by encoding graph problem solutions as code. This method solves new graph problems by learning from exemplars, generating programs, and executing them via a program interpreter.
In the upper example, the limitation of LLMs in handling arithmetic within the graph task is highlighted, while our method addresses this issue through code and external execution. In the lower-left example, LLM performance is shown to be sensitive to prompt templates that convert graph structures into natural language. The lower-right example demonstrates that the LLM’s performance is significantly influenced by the graph structure.
Follow the steps below to set up and start using CodeGraph:
Set up the required environment and APIs by following the instructions in the Installation Guide.
Generate graphs and prepare the dataset needed for evaluation by referring to the Dataset Preparation Guide.
Learn how to create prompts for CodeGraph and other prompt-based baseline methods by following the Prompt Preparation Guide.
Reproduce the evaluation experiments presented in the paper. Detailed steps are available in the Evaluation Guide.
Our work is inspired by these excellent open-sourced repos: PaL: Program-Aided Language Model, Program-of-Thoughts.
Our code builds upon Talk like a Graph. Many thanks to the them!
If you find this project helpful, please consider citing the following paper:
@misc{cai2024codegraphenhancinggraphreasoning,
title={CodeGraph: Enhancing Graph Reasoning of LLMs with Code},
author={Qiaolong Cai and Zhaowei Wang and Shizhe Diao and James Kwok and Yangqiu Song},
year={2024},
eprint={2408.13863},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2408.13863},
}This project is licensed under the Apache License 2.0. It incorporates code from Talk like a Graph, also licensed under the Apache License 2.0.
For questions or comments about the implementation, please contact: [email protected]