Add url filtering to Cheerio scraper. Also fix multiple issues of link limit enforcement. #1417
Conversation
Added more console logging for web crawl to better indicating progress
…control of web crawling
…ng dupes and not valid docs
jitsmaster
changed the title
jitsmaster
changed the title
| @@ -1,67 +1,66 @@ | |||
| import axios from 'axios' | |||
There was a problem hiding this comment.
are you using smtg like formatter that will rearrange the imports by alphabetical order?
There was a problem hiding this comment.
It's a VS code extension. If it's causing trouble, I can manually change them back.
There was a problem hiding this comment.
ah okay, if you can remove the changes that are not part of the actual web scraping changes, that'd be great!
|
many thanks for the PR! Can you highlight some of the cases where the current solution doesnt work/or has limitation? that in this PR we manage to solve that? |
|
Sorry for my late response. Thank you very much for spending your valuable time reviewing this PR. The purpose of this PR is to allow scraping large set of web pages without node out of memory. All the filtering mechanisms and re-enforcement of max count is for that purpose. Since we can never know how much memory it will actually take to store all the scraped content, there will be quite a bit of trials and errors to figure out the max size that is acceptable. My original intent is to use a piece by piece model to store any content scraped right away, but LangChain doesn't support it, and seems to have no intention to adopt this model. I do plan for a second stage changes based on the current model. That is to store the content on disk instead of memory, and before text split. That will be another PR. Cheers! |
|
this would be really helpful, I just read this PR after posting here: #1566 (comment) |
No description provided.