Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed GitDorker with Internon changes #42

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Internon
Copy link

I have changed the GitDorker for my own purpose in order to work as I prefer.
Those are the changes:

  • The threads depend on the number of tokens in the -tf parameter file, each thread uses only 1 token.
  • The sleep 60 seconds to don't reach the API rate limit has been changed in order to take the number of requests that are left from the response header "X-RateLimit-Remaining" (Each token has its own remaining number)
  • The sleep 60 seconds has been changed to sleep until the "X-RateLimit-Reset" header says, so we sleep only the necessary time.
  • There seems to be a "secondary-rate-limit" on GitHub API. As I haven't seen how to bypass it, I have added two things, random sleep between requests on the same thread (From 0.5 to 1.5) and a while to only stop the GitDorker when all requests have been correctly processed. (At least the requests where the problem is an API error)
  • In addition to the while to process all the Dorks, I have added a try-except in order to get the Control+C on the script and end the API requests but continue with the output process. (If the GitDorker process is too slow, you can stop the API requests and make a manually process with the URLs that are left)
  • Between loops in order to process all the dorks, I have added some output to see the time spent and the unique errors to see some debug information.

With all those changes, I can process all the Dorks without a problem and if I use more tokens, the process is faster.
Make any changes to the code as you want if you like this pull request and if you have any questions don't hesitate to contact me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant