diff --git a/README.md b/README.md index 1d3e69e..56ba991 100644 --- a/README.md +++ b/README.md @@ -3,13 +3,15 @@ [![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://splendidbug.github.io/DocsScraper.jl/dev/) [![Build Status](https://github.com/splendidbug/DocsScraper.jl/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/splendidbug/DocsScraper.jl/actions/workflows/CI.yml?query=branch%3Amain) [![Aqua](https://raw.githubusercontent.com/JuliaTesting/Aqua.jl/master/badge.svg)](https://github.com/JuliaTesting/Aqua.jl) -DocsScraper is a package designed to create a vector database from input URLs. It scrapes and parses the URLs and with the help of PromptingTools.jl, creates a vector store that can be used in a RAG applications. It integrates with AIHelpMe.jl and PromptingTools.jl to offer highly efficient and relevant query retrieval, ensuring that the responses generated by the system are specific to the content in the created database. +DocsScraper is a package designed to create "knowledge packs" from online documentation sites for the Julia language. + +It scrapes and parses the URLs and with the help of PromptingTools.jl, creates an index of chunks and their embeddings that can be used in RAG applications. It integrates with AIHelpMe.jl and PromptingTools.jl to offer highly efficient and relevant query retrieval, ensuring that the responses generated by the system are specific to the content in the created database. ## Features - **URL Scraping and Parsing**: Automatically scrapes and parses input URLs to extract relevant information, paying particular attention to code snippets and code blocks. Gives an option to customize the chunk sizes - **URL Crawling**: Optionally crawls the input URLs to look for multiple pages in the same domain. -- **Vector Database Creation**: Leverages PromptingTools.jl to create embeddings with customizable embedding model, size and type (Bool and Float32). +- **Knowledge Index Creation**: Leverages PromptingTools.jl to create embeddings with customizable embedding model, size and type (Bool and Float32). ## Installation @@ -28,7 +30,7 @@ Pkg.add("DocsScraper") - OpenAI API keys with available credits. See [How to Obtain API Keys](#how-to-obtain-api-keys). -## Usage +## Building the Index ```julia index = make_knowledge_packs(; single_urls=["https://docs.sciml.ai/Overview/stable/"], index_name="sciml", embedding_size=1024) ``` @@ -50,11 +52,17 @@ a sciml__v20240817__textembedding3large-1024-Bool__v1.0.hdf5 ``` `make_knowledge_packs` is the entry point to the package. This function takes in the URLs to parse and returns the index. This index can be passed to AIHelpMe.jl to answer queries on the built knowledge packs. -**Using the created index:** +## Using the Index for Questions + ```julia using AIHelpMe -sciml_index = AIHelpMe.load_index!(index) -aihelp(sciml_index, "what is Sciml") + +# Either use the index explicitly +aihelp(index, "what is Sciml") + +# or set it as the "default" index, then it will be automatically used for every question +AIHelpMe.load_index!(index) +aihelp("what is Sciml") ``` ``` [ Info: Updated RAG pipeline to `:bronze` (Configuration key: "textembedding3large-1024-Bool"). @@ -69,3 +77,9 @@ SciML, or Scientific Machine Learning, is an ecosystem developed in the Julia pr scientific computing and machine learning. It provides a range of tools with unified APIs, enabling features like differentiability, sensitivity analysis, high performance, and parallel implementations. The SciML organization supports these tools and promotes their coherent use for various scientific applications. ``` + +Tip: Use `pprint` for nicer outputs with sources +```julia +using AIHelpMe: pprint, last_result +print(last_result) +```