Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
svilupp authored Aug 18, 2024
1 parent d187d54 commit 0d6c954
Showing 1 changed file with 20 additions and 6 deletions.
26 changes: 20 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,15 @@
[![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://splendidbug.github.io/DocsScraper.jl/dev/) [![Build Status](https://github.com/splendidbug/DocsScraper.jl/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/splendidbug/DocsScraper.jl/actions/workflows/CI.yml?query=branch%3Amain) [![Aqua](https://raw.githubusercontent.com/JuliaTesting/Aqua.jl/master/badge.svg)](https://github.com/JuliaTesting/Aqua.jl)


DocsScraper is a package designed to create a vector database from input URLs. It scrapes and parses the URLs and with the help of PromptingTools.jl, creates a vector store that can be used in a RAG applications. It integrates with AIHelpMe.jl and PromptingTools.jl to offer highly efficient and relevant query retrieval, ensuring that the responses generated by the system are specific to the content in the created database.
DocsScraper is a package designed to create "knowledge packs" from online documentation sites for the Julia language.

It scrapes and parses the URLs and with the help of PromptingTools.jl, creates an index of chunks and their embeddings that can be used in RAG applications. It integrates with AIHelpMe.jl and PromptingTools.jl to offer highly efficient and relevant query retrieval, ensuring that the responses generated by the system are specific to the content in the created database.

## Features

- **URL Scraping and Parsing**: Automatically scrapes and parses input URLs to extract relevant information, paying particular attention to code snippets and code blocks. Gives an option to customize the chunk sizes
- **URL Crawling**: Optionally crawls the input URLs to look for multiple pages in the same domain.
- **Vector Database Creation**: Leverages PromptingTools.jl to create embeddings with customizable embedding model, size and type (Bool and Float32).
- **Knowledge Index Creation**: Leverages PromptingTools.jl to create embeddings with customizable embedding model, size and type (Bool and Float32).

## Installation

Expand All @@ -28,7 +30,7 @@ Pkg.add("DocsScraper")
- OpenAI API keys with available credits. See [How to Obtain API Keys](#how-to-obtain-api-keys).


## Usage
## Building the Index
```julia
index = make_knowledge_packs(; single_urls=["https://docs.sciml.ai/Overview/stable/"], index_name="sciml", embedding_size=1024)
```
Expand All @@ -50,11 +52,17 @@ a sciml__v20240817__textembedding3large-1024-Bool__v1.0.hdf5
```
`make_knowledge_packs` is the entry point to the package. This function takes in the URLs to parse and returns the index. This index can be passed to AIHelpMe.jl to answer queries on the built knowledge packs.

**Using the created index:**
## Using the Index for Questions

```julia
using AIHelpMe
sciml_index = AIHelpMe.load_index!(index)
aihelp(sciml_index, "what is Sciml")

# Either use the index explicitly
aihelp(index, "what is Sciml")

# or set it as the "default" index, then it will be automatically used for every question
AIHelpMe.load_index!(index)
aihelp("what is Sciml")
```
```
[ Info: Updated RAG pipeline to `:bronze` (Configuration key: "textembedding3large-1024-Bool").
Expand All @@ -69,3 +77,9 @@ SciML, or Scientific Machine Learning, is an ecosystem developed in the Julia pr
scientific computing and machine learning. It provides a range of tools with unified APIs, enabling features like differentiability, sensitivity analysis, high performance, and parallel
implementations. The SciML organization supports these tools and promotes their coherent use for various scientific applications.
```

Tip: Use `pprint` for nicer outputs with sources
```julia
using AIHelpMe: pprint, last_result
print(last_result)
```

0 comments on commit 0d6c954

Please sign in to comment.