Some method of properly passing Cloudflare protections?

[Kadigan]

Hey.

So Cloudflare is everywhere now (enough to bring Discord down when they furk up), and it seems WRP doesn't know how to work with it. I can't seem to be able to pass Cloudflare's "verify you are human", and it seems to be used more and more aggressively.

Any suggestions?

[TheTechRobo]

Maybe cloudflare is detecting the headless Chrome?

[Kadigan]

The particular website (fanfiction.net) is actually known for being very aggressive about it (or maybe Cloudflare is particularly aggressive about that website? who knows). All I know is that I see 30-50 "checking the security of the connection" pages on an average day (and some days - at every page load), using a regular browser.

I don't think hiding the headless Chrome better will fix it. There has to be a way to pass this check.

[TheTechRobo]

Headless Chrome is often used for data scraping. Websites like Cloudflare block it more aggressively.

What exactly happens when you go to the page? Is there an "I'm not a robot" checkbox?

[tenox7]

yeah crap, we need to come up with some solution for this

[Kadigan]

@TheTechRobo Yes, there is. It does the whole song & dance, and then redirects. Assuming the check works, the "browser" would need to be able to navigate, and/or send screen updates or something.

[tenox7]

do you have any handy examples of pages that fail this?

[Kadigan]

Anything on fanfiction.net should qualify. They pursue verification so aggressively, that I have to go through the verification almost every other chapter on my desktop. It happens most often when switching chapters using the prev/next buttons.

That was actually the website I was hoping to use it with, as previously mentioned.

[tenox7]

🤷 I have released 4.6.2; it has some basic anti crawler detection bits, also allows setting user agent with a flag; this seems to help the most;

I have tried ./wrp -ua="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" and it seems better