third_party/LlamaIndex/llamaindex_arxiv_agentic_rag.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f2626ec2-88a2-4a8c-8dbc-723617a4d6a8",
   "metadata": {},
   "source": [
    "# Building an LLM Agent to Find Relevant Research Papers from Arxiv\n",
    "\n",
    "This notebook was created by Andrei Chernov ([Github](https://github.com/ChernovAndrey), [Linkedin](https://www.linkedin.com/in/andrei-chernov-58b157236/))\n",
    "In this tutorial, we will create an LLM agent based on the **MistralAI** language model. The agent's primary purpose will be to find and summarize research papers from **Arxiv** that are relevant to the user's query. To build the agent, we will use the **LlamaIndex** framework.\n",
    "\n",
    "## Tools Used by the Agent\n",
    "\n",
    "The agent will utilize the following three tools:\n",
    "\n",
    "1. **RAG Query Engine**\n",
    "   This tool will store and retrieve recent papers from Arxiv, serving as a knowledge base for efficient and quick access to relevant information.\n",
    "\n",
    "2. **Paper Fetch Tool**\n",
    "   If the user specifies a topic that is not covered in the RAG Query Engine, this tool will fetch recent papers on the specified topic directly from Arxiv.\n",
    "\n",
    "3. **PDF Download Tool**\n",
    "   This tool allows the agent to download a research paper's PDF file locally using a link provided by Arxiv."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a9ca8ed-e873-4d3c-a5b7-a58ecc1e7a35",
   "metadata": {},
   "source": [
    "### First, let's install necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "588a271f-4ec1-4943-a94c-a9a402136b9a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: arxiv==2.1.3 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (2.1.3)\n",
      "Requirement already satisfied: llama_index==0.12.3 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (0.12.3)\n",
      "Requirement already satisfied: llama-index-llms-mistralai==0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (0.3.0)\n",
      "Requirement already satisfied: llama-index-embeddings-mistralai==0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (0.3.0)\n",
      "Requirement already satisfied: feedparser~=6.0.10 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from arxiv==2.1.3) (6.0.11)\n",
      "Requirement already satisfied: requests~=2.32.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from arxiv==2.1.3) (2.32.3)\n",
      "Requirement already satisfied: llama-index-agent-openai<0.5.0,>=0.4.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.4.0)\n",
      "Requirement already satisfied: llama-index-cli<0.5.0,>=0.4.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.4.0)\n",
      "Requirement already satisfied: llama-index-core<0.13.0,>=0.12.3 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.12.3)\n",
      "Requirement already satisfied: llama-index-embeddings-openai<0.4.0,>=0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.3.1)\n",
      "Requirement already satisfied: llama-index-indices-managed-llama-cloud>=0.4.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.6.3)\n",
      "Requirement already satisfied: llama-index-legacy<0.10.0,>=0.9.48 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.9.48.post4)\n",
      "Requirement already satisfied: llama-index-llms-openai<0.4.0,>=0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.3.2)\n",
      "Requirement already satisfied: llama-index-multi-modal-llms-openai<0.4.0,>=0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.3.0)\n",
      "Requirement already satisfied: llama-index-program-openai<0.4.0,>=0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.3.1)\n",
      "Requirement already satisfied: llama-index-question-gen-openai<0.4.0,>=0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.3.0)\n",
      "Requirement already satisfied: llama-index-readers-file<0.5.0,>=0.4.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.4.1)\n",
      "Requirement already satisfied: llama-index-readers-llama-parse>=0.4.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (0.4.0)\n",
      "Requirement already satisfied: nltk>3.8.1 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama_index==0.12.3) (3.9.1)\n",
      "Requirement already satisfied: mistralai>=1.0.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-llms-mistralai==0.3.0) (1.2.5)\n",
      "Requirement already satisfied: sgmllib3k in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from feedparser~=6.0.10->arxiv==2.1.3) (1.0.0)\n",
      "Requirement already satisfied: openai>=1.14.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-agent-openai<0.5.0,>=0.4.0->llama_index==0.12.3) (1.57.0)\n",
      "Requirement already satisfied: PyYAML>=6.0.1 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (6.0.2)\n",
      "Requirement already satisfied: SQLAlchemy>=1.4.49 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (2.0.36)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (3.11.10)\n",
      "Requirement already satisfied: dataclasses-json in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (0.6.7)\n",
      "Requirement already satisfied: deprecated>=1.2.9.3 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.2.15)\n",
      "Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.0.8)\n",
      "Requirement already satisfied: filetype<2.0.0,>=1.2.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.2.0)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (2024.10.0)\n",
      "Requirement already satisfied: httpx in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (0.27.2)\n",
      "Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.6.0)\n",
      "Requirement already satisfied: networkx>=3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (3.4.2)\n",
      "Requirement already satisfied: numpy in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (2.1.3)\n",
      "Requirement already satisfied: pillow>=9.0.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (11.0.0)\n",
      "Requirement already satisfied: pydantic<2.10.0,>=2.7.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (2.9.2)\n",
      "Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.2.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (8.5.0)\n",
      "Requirement already satisfied: tiktoken>=0.3.3 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (0.8.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (4.67.1)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (4.12.2)\n",
      "Requirement already satisfied: typing-inspect>=0.8.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (0.9.0)\n",
      "Requirement already satisfied: wrapt in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.17.0)\n",
      "Requirement already satisfied: llama-cloud>=0.1.5 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-indices-managed-llama-cloud>=0.4.0->llama_index==0.12.3) (0.1.6)\n",
      "Requirement already satisfied: pandas in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-legacy<0.10.0,>=0.9.48->llama_index==0.12.3) (2.2.3)\n",
      "Requirement already satisfied: beautifulsoup4<5.0.0,>=4.12.3 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-readers-file<0.5.0,>=0.4.0->llama_index==0.12.3) (4.12.3)\n",
      "Requirement already satisfied: pypdf<6.0.0,>=5.1.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-readers-file<0.5.0,>=0.4.0->llama_index==0.12.3) (5.1.0)\n",
      "Requirement already satisfied: striprtf<0.0.27,>=0.0.26 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-readers-file<0.5.0,>=0.4.0->llama_index==0.12.3) (0.0.26)\n",
      "Requirement already satisfied: llama-parse>=0.5.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from llama-index-readers-llama-parse>=0.4.0->llama_index==0.12.3) (0.5.17)\n",
      "Requirement already satisfied: eval-type-backport<0.3.0,>=0.2.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from mistralai>=1.0.0->llama-index-llms-mistralai==0.3.0) (0.2.0)\n",
      "Requirement already satisfied: jsonpath-python<2.0.0,>=1.0.6 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from mistralai>=1.0.0->llama-index-llms-mistralai==0.3.0) (1.0.6)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.8.2 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from mistralai>=1.0.0->llama-index-llms-mistralai==0.3.0) (2.9.0.post0)\n",
      "Requirement already satisfied: click in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from nltk>3.8.1->llama_index==0.12.3) (8.1.7)\n",
      "Requirement already satisfied: joblib in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from nltk>3.8.1->llama_index==0.12.3) (1.4.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from nltk>3.8.1->llama_index==0.12.3) (2024.11.6)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from requests~=2.32.0->arxiv==2.1.3) (3.4.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from requests~=2.32.0->arxiv==2.1.3) (3.10)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from requests~=2.32.0->arxiv==2.1.3) (2.2.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from requests~=2.32.0->arxiv==2.1.3) (2024.8.30)\n",
      "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (2.4.4)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (24.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.5.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (6.1.0)\n",
      "Requirement already satisfied: propcache>=0.2.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (0.2.1)\n",
      "Requirement already satisfied: yarl<2.0,>=1.17.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.18.3)\n",
      "Requirement already satisfied: soupsieve>1.2 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from beautifulsoup4<5.0.0,>=4.12.3->llama-index-readers-file<0.5.0,>=0.4.0->llama_index==0.12.3) (2.6)\n",
      "Requirement already satisfied: anyio in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from httpx->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (4.7.0)\n",
      "Requirement already satisfied: httpcore==1.* in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from httpx->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.0.7)\n",
      "Requirement already satisfied: sniffio in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from httpx->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.3.1)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from httpcore==1.*->httpx->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (0.14.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from openai>=1.14.0->llama-index-agent-openai<0.5.0,>=0.4.0->llama_index==0.12.3) (1.9.0)\n",
      "Requirement already satisfied: jiter<1,>=0.4.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from openai>=1.14.0->llama-index-agent-openai<0.5.0,>=0.4.0->llama_index==0.12.3) (0.8.0)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from pydantic<2.10.0,>=2.7.0->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.23.4 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from pydantic<2.10.0,>=2.7.0->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (2.23.4)\n",
      "Requirement already satisfied: six>=1.5 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from python-dateutil<3.0.0,>=2.8.2->mistralai>=1.0.0->llama-index-llms-mistralai==0.3.0) (1.17.0)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from SQLAlchemy>=1.4.49->SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (3.1.1)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (1.0.0)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from dataclasses-json->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (3.23.1)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from pandas->llama-index-legacy<0.10.0,>=0.9.48->llama_index==0.12.3) (2024.2)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from pandas->llama-index-legacy<0.10.0,>=0.9.48->llama_index==0.12.3) (2024.2)\n",
      "Requirement already satisfied: packaging>=17.0 in /Users/andreichernov/opt/anaconda3/envs/mistral-llama/lib/python3.12/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.13.0,>=0.12.3->llama_index==0.12.3) (24.2)\n"
     ]
    }
   ],
   "source": [
    "!pip install arxiv==2.1.3 llama_index==0.12.3 llama-index-llms-mistralai==0.3.0 llama-index-embeddings-mistralai==0.3.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d789906a-1b70-434b-9584-74abbf3cef92",
   "metadata": {},
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "import requests\n",
    "import sys\n",
    "import arxiv\n",
    "from llama_index.llms.mistralai import MistralAI\n",
    "from llama_index.embeddings.mistralai import MistralAIEmbedding\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Document, StorageContext, load_index_from_storage, PromptTemplate, Settings\n",
    "from llama_index.core.tools import FunctionTool, QueryEngineTool\n",
    "from llama_index.core.agent import ReActAgent\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ef29af-ee10-4a73-8c7d-a8ca58f15265",
   "metadata": {},
   "source": [
    "### Additionally, You Need to Provide Your API Key to Access Mistral Models\n",
    "\n",
    "You can obtain an API key [here](https://console.mistral.ai/api-keys/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed41d8b5-2926-4b81-a751-834ef51a68fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Type your API Key ········\n"
     ]
    }
   ],
   "source": [
    "api_key= getpass(\"Type your API Key\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "216e1bb9-afd1-450f-a7cc-b1022a1e2a14",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = MistralAI(api_key=api_key, model='mistral-large-latest')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f54a66d4-2bce-4a2b-ba41-aa93e70ccc75",
   "metadata": {},
   "source": [
    "### To Build a RAG Query Engine, We Will Need an Embedding Model\n",
    "\n",
    "For this tutorial, we will use the MistralAI embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "88e2982b-3608-4c69-b3d3-d33df51d0353",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"mistral-embed\"\n",
    "embed_model = MistralAIEmbedding(model_name=model_name, api_key=api_key)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "116f358b-1709-4439-bd8e-36a3ecd13b4e",
   "metadata": {},
   "source": [
    "### Now, We Will Download Recent Papers About Large Language Models from ArXiv\n",
    "\n",
    "To keep this tutorial accessible with the free Mistral API version, we will download only the last 10 papers. Downloading more would exceed the limit later while building the RAG query engine. However, if you have a Mistral subscription, you can download additional papers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a80c1d8b-afc5-4ebd-be13-e9457d8b0d0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fetch_arxiv_papers(title :str, papers_count: int):\n",
    "    search_query = f'all:\"{title}\"'\n",
    "    search = arxiv.Search(\n",
    "        query=search_query,\n",
    "        max_results=papers_count,\n",
    "        sort_by=arxiv.SortCriterion.SubmittedDate,\n",
    "        sort_order=arxiv.SortOrder.Descending\n",
    "    )\n",
    "\n",
    "    papers = []\n",
    "    # Use the Client for searching\n",
    "    client = arxiv.Client()\n",
    "    \n",
    "    # Execute the search\n",
    "    search = client.results(search)\n",
    "\n",
    "    for result in search:\n",
    "        paper_info = {\n",
    "                'title': result.title,\n",
    "                'authors': [author.name for author in result.authors],\n",
    "                'summary': result.summary,\n",
    "                'published': result.published,\n",
    "                'journal_ref': result.journal_ref,\n",
    "                'doi': result.doi,\n",
    "                'primary_category': result.primary_category,\n",
    "                'categories': result.categories,\n",
    "                'pdf_url': result.pdf_url,\n",
    "                'arxiv_url': result.entry_id\n",
    "            }\n",
    "        papers.append(paper_info)\n",
    "\n",
    "    return papers\n",
    "    \n",
    "papers = fetch_arxiv_papers(\"Language Models\", 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a1cd5cd8-45b8-45e5-96d4-f91cdab08490",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Generative Semantic Communication: Architectures, Technologies, and Applications'],\n",
       " ['Fast Prompt Alignment for Text-to-Image Generation'],\n",
       " ['Multimodal Latent Language Modeling with Next-Token Diffusion'],\n",
       " ['Synthetic Vision: Training Vision-Language Models to Understand Physics'],\n",
       " ['Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models'],\n",
       " ['Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning'],\n",
       " ['Competition and Diversity in Generative AI'],\n",
       " ['AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models'],\n",
       " ['Preference Discerning with LLM-Enhanced Generative Retrieval'],\n",
       " ['Empirical Measurements of AI Training Power Demand on a GPU-Accelerated Node']]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[[p['title']] for p in papers]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efd68006-29f8-49be-a730-f1c7e4d047dd",
   "metadata": {},
   "source": [
    "### To Build a RAG Agent, We First Need to Index All Documents\n",
    "\n",
    "This process creates a vector representation for each chunk of a document using the embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b312fbf7-d7e9-4189-8698-d6bba4f92f29",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_documents_from_papers(papers):\n",
    "    documents = []\n",
    "    for paper in papers:\n",
    "        content = f\"Title: {paper['title']}\\n\" \\\n",
    "                  f\"Authors: {', '.join(paper['authors'])}\\n\" \\\n",
    "                  f\"Summary: {paper['summary']}\\n\" \\\n",
    "                  f\"Published: {paper['published']}\\n\" \\\n",
    "                  f\"Journal Reference: {paper['journal_ref']}\\n\" \\\n",
    "                  f\"DOI: {paper['doi']}\\n\" \\\n",
    "                  f\"Primary Category: {paper['primary_category']}\\n\" \\\n",
    "                  f\"Categories: {', '.join(paper['categories'])}\\n\" \\\n",
    "                  f\"PDF URL: {paper['pdf_url']}\\n\" \\\n",
    "                  f\"arXiv URL: {paper['arxiv_url']}\\n\"\n",
    "        documents.append(Document(text=content))\n",
    "    return documents\n",
    "\n",
    "\n",
    "\n",
    "#Create documents for LlamaIndex\n",
    "documents = create_documents_from_papers(papers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2895d49f-856f-4152-a6c8-9894be0f2617",
   "metadata": {},
   "outputs": [],
   "source": [
    "Settings.chunk_size = 1024\n",
    "Settings.chunk_overlap = 50\n",
    "\n",
    "index = VectorStoreIndex.from_documents(documents, embed_model=embed_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc3e904f-a574-4eba-80a4-71385ddeddca",
   "metadata": {},
   "source": [
    "### Now, We Will Store the Index\n",
    "\n",
    "Indexing a large number of texts can be time-consuming and costly since it requires making API calls to the embedding model. In real-world applications, it is better to store the index in a vector database to avoid reindexing. However, for simplicity, we will store the index locally in a directory in this tutorial, without using a vector database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dbf5f441-81c5-4fc0-ab18-514dafd2791e",
   "metadata": {},
   "outputs": [],
   "source": [
    "index.storage_context.persist('index/')\n",
    "# rebuild storage context\n",
    "storage_context = StorageContext.from_defaults(persist_dir='index/')\n",
    "\n",
    "#load index\n",
    "index = load_index_from_storage(storage_context, embed_model=embed_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9c20ce1-f839-426b-8da1-0f62479fb220",
   "metadata": {},
   "source": [
    "### We Are Ready to Build a RAG Query Engine for Our Agent\n",
    "\n",
    "It is a good practice to provide a meaningful name and a clear description for each tool. This helps the agent select the most appropriate tool when needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5454d715-4c46-4901-98bc-1c2c6930f80b",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = index.as_query_engine(llm=llm, similarity_top_k=5)\n",
    "\n",
    "rag_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine,\n",
    "    name=\"research_paper_query_engine_tool\",\n",
    "    description=\"A RAG engine with recent research papers.\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "347ef41c-d45d-49eb-b538-9913118b8646",
   "metadata": {},
   "source": [
    "### Let's Take a Look at the Prompts the RAG Tool Uses to Answer a Query Based on Context\n",
    "\n",
    "Note that there are two prompts. By default, LlamaIndex uses a refine prompt before returning an answer. You can find more information about the response modes [here](https://docs.llamaindex.ai/en/v0.10.34/module_guides/deploying/query_engine/response_modes/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e86379cf-e30d-4588-a257-558d498694a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Prompt Key**: response_synthesizer:text_qa_template**Text:** "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context information is below.\n",
      "---------------------\n",
      "{context_str}\n",
      "---------------------\n",
      "Given the context information and not prior knowledge, answer the query.\n",
      "Query: {query_str}\n",
      "Answer: \n"
     ]
    },
    {
     "data": {
      "text/markdown": [],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Prompt Key**: response_synthesizer:refine_template**Text:** "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original query is as follows: {query_str}\n",
      "We have provided an existing answer: {existing_answer}\n",
      "We have the opportunity to refine the existing answer (only if needed) with some more context below.\n",
      "------------\n",
      "{context_msg}\n",
      "------------\n",
      "Given the new context, refine the original answer to better answer the query. If the context isn't useful, return the original answer.\n",
      "Refined Answer: \n"
     ]
    },
    {
     "data": {
      "text/markdown": [],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from llama_index.core import PromptTemplate\n",
    "from IPython.display import Markdown, display\n",
    "# define prompt viewing function\n",
    "def display_prompt_dict(prompts_dict):\n",
    "    for k, p in prompts_dict.items():\n",
    "        text_md = f\"**Prompt Key**: {k}\" f\"**Text:** \"\n",
    "        display(Markdown(text_md))\n",
    "        print(p.get_template())\n",
    "        display(Markdown(\"\"))\n",
    "        \n",
    "prompts_dict = query_engine.get_prompts()\n",
    "display_prompt_dict(prompts_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23cd08e3-d7b0-4473-b55e-5c77b4b686b3",
   "metadata": {},
   "source": [
    "### Building two other tools is straightforward because they are simply Python functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0ecd8c50-4634-477a-9da5-8ac9b9cc048c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_pdf(pdf_url, output_file):\n",
    "    \"\"\"\n",
    "    Downloads a PDF file from the given URL and saves it to the specified file.\n",
    "\n",
    "    Args:\n",
    "        pdf_url (str): The URL of the PDF file to download.\n",
    "        output_file (str): The path and name of the file to save the PDF to.\n",
    "\n",
    "    Returns:\n",
    "        str: A message indicating success or the nature of an error.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        # Send a GET request to the PDF URL\n",
    "        response = requests.get(pdf_url)\n",
    "        response.raise_for_status()  # Raise an error for HTTP issues\n",
    "\n",
    "        # Write the content of the PDF to the output file\n",
    "        with open(output_file, \"wb\") as file:\n",
    "            file.write(response.content)\n",
    "\n",
    "        return f\"PDF downloaded successfully and saved as '{output_file}'.\"\n",
    "\n",
    "    except requests.exceptions.RequestException as e:\n",
    "        return f\"An error occurred: {e}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "14780249-0a3d-40bf-9d2b-c7d47380b4c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "download_pdf_tool = FunctionTool.from_defaults(\n",
    "    download_pdf,\n",
    "    name='download_pdf_file_tool',\n",
    "    description='python function, which downloads a pdf file by link'\n",
    ")\n",
    "fetch_arxiv_tool = FunctionTool.from_defaults(\n",
    "    fetch_arxiv_papers,\n",
    "    name='fetch_from_arxiv',\n",
    "    description='download the {max_results} recent papers regarding the topic {title} from arxiv' \n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ece38fe3-5c80-4115-b5ce-18c64bdbe530",
   "metadata": {},
   "outputs": [],
   "source": [
    "# building an ReAct Agent with the three tools.\n",
    "agent = ReActAgent.from_tools([download_pdf_tool, rag_tool, fetch_arxiv_tool], llm=llm, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "753bc269-1403-49fe-998a-72e5d11702f7",
   "metadata": {},
   "source": [
    "### Let's Chat with Our Agent\n",
    "\n",
    "We built a ReAct agent, which operates in two main stages:\n",
    "\n",
    "1. **Reasoning**: Upon receiving a query, the agent evaluates whether it has enough information to answer directly or if it needs to use a tool.\n",
    "2. **Acting**: If the agent decides to use a tool, it executes the tool and then returns to the Reasoning stage to determine whether it can now answer the query or if further tool usage is necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fed70500-6f23-471c-b919-e8138c71ec98",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a prompt template to chat with an agent\n",
    "q_template = (\n",
    "    \"I am interested in {topic}. \\n\"\n",
    "    \"Find papers in your knowledge database related to this topic; use the following template to query research_paper_query_engine_tool tool: 'Provide title, summary, authors and link to download for papers related to {topic}'. If there are not, could you fetch the recent one from arXiv? \\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "97f89bfe-a791-4a63-b196-e29c599b2471",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Running step 7ff11a0d-341a-4fd3-ba60-9769da4350a2. Step input: I am interested in Audio-Language Models. \n",
      "Find papers in your knowledge database related to this topic; use the following template to query research_paper_query_engine_tool tool: 'Provide title, summary, authors and link to download for papers related to Audio-Language Models'. If there are not, could you fetch the recent one from arXiv? \n",
      "\n",
      "\u001B[1;3;38;5;200mThought: The current language of the user is: English. I need to use a tool to help me answer the question.\n",
      "Action: research_paper_query_engine_tool\n",
      "Action Input: {'input': 'Provide title, summary, authors and link to download for papers related to Audio-Language Models'}\n",
      "\u001B[0m\u001B[1;3;34mObservation: The title of the paper related to Audio-Language Models is \"AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models.\" The authors are Mintong Kang, Chejian Xu, and Bo Li.\n",
      "\n",
      "Here is a summary of the paper:\n",
      "Recent advancements in large audio-language models (LALMs) have enabled speech-based user interactions, significantly enhancing user experience and accelerating the deployment of LALMs in real-world applications. However, ensuring the safety of LALMs is crucial to prevent risky outputs that may raise societal concerns or violate AI regulations. Despite the importance of this issue, research on jailbreaking LALMs remains limited due to their recent emergence and the additional technical challenges they present compared to attacks on DNN-based audio models. Specifically, the audio encoders in LALMs, which involve discretization operations, often lead to gradient shattering, hindering the effectiveness of attacks relying on gradient-based optimizations. The behavioral variability of LALMs further complicates the identification of effective (adversarial) optimization targets. Moreover, enforcing stealthiness constraints on adversarial audio waveforms introduces a reduced, non-convex feasible solution space, further intensifying the challenges of the optimization process. To overcome these challenges, we develop AdvWave, the first jailbreak framework against LALMs. We propose a dual-phase optimization method that addresses gradient shattering, enabling effective end-to-end gradient-based optimization. Additionally, we develop an adaptive adversarial target search algorithm that dynamically adjusts the adversarial optimization target based on the response patterns of LALMs for specific queries. To ensure that adversarial audio remains perceptually natural to human listeners, we design a classifier-guided optimization approach that generates adversarial noise resembling common urban sounds. Extensive evaluations on multiple advanced LALMs demonstrate that AdvWave outperforms baseline methods, achieving a 40% higher average jailbreak attack success rate.\n",
      "\n",
      "You can download the paper [here](http://arxiv.org/pdf/2412.08608v1).\n",
      "\u001B[0m> Running step 484715a2-640a-4139-922a-9f8a3dc06189. Step input: None\n",
      "\u001B[1;3;38;5;200mThought: I can answer without using any more tools. I'll use the user's language to answer\n",
      "Answer: The title of the paper related to Audio-Language Models is \"AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models.\" The authors are Mintong Kang, Chejian Xu, and Bo Li.\n",
      "\n",
      "Here is a summary of the paper:\n",
      "Recent advancements in large audio-language models (LALMs) have enabled speech-based user interactions, significantly enhancing user experience and accelerating the deployment of LALMs in real-world applications. However, ensuring the safety of LALMs is crucial to prevent risky outputs that may raise societal concerns or violate AI regulations. Despite the importance of this issue, research on jailbreaking LALMs remains limited due to their recent emergence and the additional technical challenges they present compared to attacks on DNN-based audio models. Specifically, the audio encoders in LALMs, which involve discretization operations, often lead to gradient shattering, hindering the effectiveness of attacks relying on gradient-based optimizations. The behavioral variability of LALMs further complicates the identification of effective (adversarial) optimization targets. Moreover, enforcing stealthiness constraints on adversarial audio waveforms introduces a reduced, non-convex feasible solution space, further intensifying the challenges of the optimization process. To overcome these challenges, we develop AdvWave, the first jailbreak framework against LALMs. We propose a dual-phase optimization method that addresses gradient shattering, enabling effective end-to-end gradient-based optimization. Additionally, we develop an adaptive adversarial target search algorithm that dynamically adjusts the adversarial optimization target based on the response patterns of LALMs for specific queries. To ensure that adversarial audio remains perceptually natural to human listeners, we design a classifier-guided optimization approach that generates adversarial noise resembling common urban sounds. Extensive evaluations on multiple advanced LALMs demonstrate that AdvWave outperforms baseline methods, achieving a 40% higher average jailbreak attack success rate.\n",
      "\n",
      "You can download the paper [here](http://arxiv.org/pdf/2412.08608v1).\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "answer = agent.chat(q_template.format(topic=\"Audio-Language Models\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b807dbac-21ec-4dcf-8453-07e815f5fa7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The title of the paper related to Audio-Language Models is \"AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models.\" The authors are Mintong Kang, Chejian Xu, and Bo Li.\n",
       "\n",
       "Here is a summary of the paper:\n",
       "Recent advancements in large audio-language models (LALMs) have enabled speech-based user interactions, significantly enhancing user experience and accelerating the deployment of LALMs in real-world applications. However, ensuring the safety of LALMs is crucial to prevent risky outputs that may raise societal concerns or violate AI regulations. Despite the importance of this issue, research on jailbreaking LALMs remains limited due to their recent emergence and the additional technical challenges they present compared to attacks on DNN-based audio models. Specifically, the audio encoders in LALMs, which involve discretization operations, often lead to gradient shattering, hindering the effectiveness of attacks relying on gradient-based optimizations. The behavioral variability of LALMs further complicates the identification of effective (adversarial) optimization targets. Moreover, enforcing stealthiness constraints on adversarial audio waveforms introduces a reduced, non-convex feasible solution space, further intensifying the challenges of the optimization process. To overcome these challenges, we develop AdvWave, the first jailbreak framework against LALMs. We propose a dual-phase optimization method that addresses gradient shattering, enabling effective end-to-end gradient-based optimization. Additionally, we develop an adaptive adversarial target search algorithm that dynamically adjusts the adversarial optimization target based on the response patterns of LALMs for specific queries. To ensure that adversarial audio remains perceptually natural to human listeners, we design a classifier-guided optimization approach that generates adversarial noise resembling common urban sounds. Extensive evaluations on multiple advanced LALMs demonstrate that AdvWave outperforms baseline methods, achieving a 40% higher average jailbreak attack success rate.\n",
       "\n",
       "You can download the paper [here](http://arxiv.org/pdf/2412.08608v1)."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Markdown(answer.response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ec372d9-b760-40f9-b2af-1c4a45b026b0",
   "metadata": {},
   "source": [
    "### The agent chose to use the RAG tool, found the relevant papers, and summarized them for us.  \n",
    "### Since the agent retains the chat history, we can request to download the papers without mentioning them explicitly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "dba3d1ba-3d87-4c58-90fb-fc6f4c0bef07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Running step e9f367d6-3bb8-4cd1-b71b-4ef96f189fa8. Step input: Download the papers, which you mentioned above\n",
      "\u001B[1;3;38;5;200mThought: I need to use a tool to help me answer the question.\n",
      "Action: download_pdf_file_tool\n",
      "Action Input: {'pdf_url': 'http://arxiv.org/pdf/2412.08608v1', 'output_file': 'AdvWave_Stealthy_Adversarial_Jailbreak_Attack_against_Large_Audio-Language_Models.pdf'}\n",
      "\u001B[0m\u001B[1;3;34mObservation: PDF downloaded successfully and saved as 'AdvWave_Stealthy_Adversarial_Jailbreak_Attack_against_Large_Audio-Language_Models.pdf'.\n",
      "\u001B[0m> Running step 3e3382c2-00f4-4f0d-a401-d23cb766a964. Step input: None\n",
      "\u001B[1;3;38;5;200mThought: I can answer without using any more tools. I'll use the user's language to answer\n",
      "Answer: The paper \"AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models\" has been downloaded successfully and saved as 'AdvWave_Stealthy_Adversarial_Jailbreak_Attack_against_Large_Audio-Language_Models.pdf'.\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "answer = agent.chat(\"Download the papers, which you mentioned above\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a1f81867-aec9-43a3-a136-e6f315bb236e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The paper \"AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models\" has been downloaded successfully and saved as 'AdvWave_Stealthy_Adversarial_Jailbreak_Attack_against_Large_Audio-Language_Models.pdf'."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Markdown(answer.response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80fbd7de-aff6-47af-93fe-403a2f04bd7b",
   "metadata": {},
   "source": [
    "### Let's see what happens if we ask about a topic that is not available in the RAG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d63a1ea3-5dc0-4849-b4d7-4bb602df10d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Running step 2501b676-8090-4ed0-a6e2-a141e6ddcfec. Step input: I am interested in Gaussian process. \n",
      "Find papers in your knowledge database related to this topic; use the following template to query research_paper_query_engine_tool tool: 'Provide title, summary, authors and link to download for papers related to Gaussian process'. If there are not, could you fetch the recent one from arXiv? \n",
      "\n",
      "\u001B[1;3;38;5;200mThought: I need to use a tool to help me answer the question.\n",
      "Action: research_paper_query_engine_tool\n",
      "Action Input: {'input': 'Provide title, summary, authors and link to download for papers related to Gaussian process'}\n",
      "\u001B[0m\u001B[1;3;34mObservation: I'm sorry, but there are no papers related to Gaussian process in the given context information.\n",
      "\u001B[0m> Running step adefe2fc-b885-4483-be4a-27f00faa1446. Step input: None\n",
      "\u001B[1;3;38;5;200mThought: I need to use a tool to help me answer the question.\n",
      "Action: fetch_from_arxiv\n",
      "Action Input: {'title': 'Gaussian process', 'papers_count': 1}\n",
      "\u001B[0m\u001B[1;3;34mObservation: [{'title': 'Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty', 'authors': ['Jake Thomas', 'Jeremie Houssineau'], 'summary': 'A popular strategy for active learning is to specifically target a reduction\\nin epistemic uncertainty, since aleatoric uncertainty is often considered as\\nbeing intrinsic to the system of interest and therefore not reducible. Yet,\\ndistinguishing these two types of uncertainty remains challenging and there is\\nno single strategy that consistently outperforms the others. We propose to use\\na particular combination of probability and possibility theories, with the aim\\nof using the latter to specifically represent epistemic uncertainty, and we\\nshow how this combination leads to new active learning strategies that have\\ndesirable properties. In order to demonstrate the efficiency of these\\nstrategies in non-trivial settings, we introduce the notion of a possibilistic\\nGaussian process (GP) and consider GP-based multiclass and binary\\nclassification problems, for which the proposed methods display a strong\\nperformance for both simulated and real datasets.', 'published': datetime.datetime(2024, 12, 11, 9, 19, 20, tzinfo=datetime.timezone.utc), 'journal_ref': None, 'doi': None, 'primary_category': 'stat.ME', 'categories': ['stat.ME', 'cs.LG'], 'pdf_url': 'http://arxiv.org/pdf/2412.08225v1', 'arxiv_url': 'http://arxiv.org/abs/2412.08225v1'}]\n",
      "\u001B[0m> Running step a8393a8d-2d5c-41c1-8ba9-1911993089d1. Step input: None\n",
      "\u001B[1;3;38;5;200mThought: I can answer without using any more tools. I'll use the user's language to answer\n",
      "Answer: The title of the paper related to Gaussian process is \"Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty.\" The authors are Jake Thomas and Jeremie Houssineau.\n",
      "\n",
      "Here is a summary of the paper:\n",
      "A popular strategy for active learning is to specifically target a reduction in epistemic uncertainty, since aleatoric uncertainty is often considered as being intrinsic to the system of interest and therefore not reducible. Yet, distinguishing these two types of uncertainty remains challenging and there is no single strategy that consistently outperforms the others. We propose to use a particular combination of probability and possibility theories, with the aim of using the latter to specifically represent epistemic uncertainty, and we show how this combination leads to new active learning strategies that have desirable properties. In order to demonstrate the efficiency of these strategies in non-trivial settings, we introduce the notion of a possibilistic Gaussian process (GP) and consider GP-based multiclass and binary classification problems, for which the proposed methods display a strong performance for both simulated and real datasets.\n",
      "\n",
      "You can download the paper [here](http://arxiv.org/pdf/2412.08225v1).\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "answer = agent.chat(q_template.format(topic=\"Gaussian process\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "c404a5f6-933e-4e2e-bc82-ce5f9c3ca167",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The title of the paper related to Gaussian process is \"Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty.\" The authors are Jake Thomas and Jeremie Houssineau.\n",
       "\n",
       "Here is a summary of the paper:\n",
       "A popular strategy for active learning is to specifically target a reduction in epistemic uncertainty, since aleatoric uncertainty is often considered as being intrinsic to the system of interest and therefore not reducible. Yet, distinguishing these two types of uncertainty remains challenging and there is no single strategy that consistently outperforms the others. We propose to use a particular combination of probability and possibility theories, with the aim of using the latter to specifically represent epistemic uncertainty, and we show how this combination leads to new active learning strategies that have desirable properties. In order to demonstrate the efficiency of these strategies in non-trivial settings, we introduce the notion of a possibilistic Gaussian process (GP) and consider GP-based multiclass and binary classification problems, for which the proposed methods display a strong performance for both simulated and real datasets.\n",
       "\n",
       "You can download the paper [here](http://arxiv.org/pdf/2412.08225v1)."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Markdown(answer.response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98070a51-5bed-4ca1-84da-34745af3a29d",
   "metadata": {},
   "source": [
    "### As You Can See, the Agent Did Not Find the Papers in Storage and Fetched Them from ArXiv."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}