Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duckduckgo wait_for_results selector doesn't work in case of no results #54

Open
PinkSheep opened this issue Nov 29, 2019 · 1 comment

Comments

@PinkSheep
Copy link

Hello
It seems like the selector you are currently using to scrape the Duckduckgo Result Page is not properly working in the case that the search doesn't produce any results:

[i] duckduckgo scrapes keyword ... on page 1
Problem with scraping ... in search engine duckduckgo: TimeoutError: waiting for selector ".result__body" failed: timeout 10000ms exceeded

I think the issue here is the selector you are using:

async wait_for_results() {
        await this.page.waitForSelector('.result__body', { timeout: this.STANDARD_TIMEOUT });
    }

If the search produces no results, the Duckduckgo Results Page doesn't have a div with the class .result__body.
As a fix I suggest using one of the following selectors instead (I have not tested it though, but they are present in both cases):

#links
.serp__results
@sasokyleader
Copy link

have you got a solution for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants