Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Ignore Rate Limit for Whitelisted Clients #27

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,17 @@ Example: 100 requests per hour
except TooManyRequests:
return '429 Too Many Requests'

Example: you can also pass an optional list of ignored_clients to bypass Rate Limit

.. code-block:: python

from redis_rate_limit import RateLimit, TooManyRequests
try:
with RateLimit(resource='users_list', client='192.168.0.10', max_requests=100, ignored_clients=['192.168.0.10'], expire=3600):
return '200 OK'
except TooManyRequests:
return '429 Too Many Requests'

Example: you can also setup a factory to use it later

.. code-block:: python
Expand Down
17 changes: 13 additions & 4 deletions redis_rate_limit/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ class RateLimit(object):
This class offers an abstraction of a Rate Limit algorithm implemented on
top of Redis >= 2.6.0.
"""
def __init__(self, resource, client, max_requests, expire=None, redis_pool=REDIS_POOL):
def __init__(self, resource, client, max_requests, ignored_clients=None, expire=None, redis_pool=REDIS_POOL):
"""
Class initialization method checks if the Rate Limit algorithm is
actually supported by the installed Redis version and sets some
Expand All @@ -54,10 +54,14 @@ def __init__(self, resource, client, max_requests, expire=None, redis_pool=REDIS
:param resource: resource identifier string (i.e. ‘user_pictures’)
:param client: client identifier string (i.e. ‘192.168.0.10’)
:param max_requests: integer (i.e. ‘10’)
:param ignored_clients: list of ip addresses (i.e. ['127.0.0.1'])
:param expire: seconds to wait before resetting counters (i.e. ‘60’)
:param redis_pool: instance of redis.ConnectionPool.
Default: ConnectionPool(host='127.0.0.1', port=6379, db=0)
"""
self.client = client
self.ignored_clients = ignored_clients

self._redis = Redis(connection_pool=redis_pool)
if not self._is_rate_limit_supported():
raise RedisVersionNotSupported()
Expand Down Expand Up @@ -88,7 +92,6 @@ def get_usage(self):
"""
Returns actual resource usage by client. Note that it could be greater
than the maximum number of requests set.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've removed a bunch of blank lines from docstrings. Could you revert it, please?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes reverted.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blank line is still removed, could you revert it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you pushed those updates?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry forgot to push changes. Updated.

:return: integer: current usage
"""
return int(self._redis.get(self._rate_limit_key) or 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the check should be done here.

Suggested change
return int(self._redis.get(self._rate_limit_key) or 0)
if self.ignored_clients and self.client in self.ignored_clients:
return 0
return int(self._redis.get(self._rate_limit_key) or 0)

There are two reasons for it:

  • adding a client to the list has always an instant bypass effect
  • removing a client from the list automatically enforces the limits again

The reason is to avoid edge cases like this:

  • client reaches max requests limit
  • client is added to the ignored clients list
  • client will still need to wait for expire time before being allowed to make requests again

Or this other one:

  • client is added to ignored list by mistake and overflows max number of requests
  • client is removed from the list
  • there's no back-off

What do you think, @italorossi?

Expand Down Expand Up @@ -129,6 +132,9 @@ def increment_usage(self, increment_by=1):

:return: integer: current usage
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the first line here could be:

if self.client in self.bypass_clients:
    return 0

The same could be done to other methods since they don't even need to query Redis if the client is supposed to be bypassed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

if self.ignored_clients and self.client in self.ignored_clients:
return 0

if increment_by > self._max_requests:
raise ValueError('increment_by {increment_by} overflows '
'max_requests of {max_requests}'
Expand Down Expand Up @@ -172,17 +178,19 @@ def _reset(self):


class RateLimiter(object):
def __init__(self, resource, max_requests, expire=None, redis_pool=REDIS_POOL):
def __init__(self, resource, max_requests, ignored_clients=None, expire=None, redis_pool=REDIS_POOL):
"""
Rate limit factory. Checks if RateLimit is supported when limit is called.
:param resource: resource identifier string (i.e. ‘user_pictures’)
:param max_requests: integer (i.e. ‘10’)
:param ignored_clients: list of ip addresses (i.e. ['127.0.0.1'])
:param expire: seconds to wait before resetting counters (i.e. ‘60’)
:param redis_pool: instance of redis.ConnectionPool.
Default: ConnectionPool(host='127.0.0.1', port=6379, db=0)
"""
self.resource = resource
self.max_requests = max_requests
self.ignored_clients = ignored_clients
self.expire = expire
self.redis_pool = redis_pool

Expand All @@ -194,6 +202,7 @@ def limit(self, client):
resource=self.resource,
client=client,
max_requests=self.max_requests,
ignored_clients=self.ignored_clients,
expire=self.expire,
redis_pool=self.redis_pool,
)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, revert this change leaving a blank line at the end of the file.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

13 changes: 13 additions & 0 deletions tests/rate_limit_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,19 @@ def test_limit_10_max_request(self):
self.assertEqual(self.rate_limit.get_usage(), 11)
self.assertEqual(self.rate_limit.has_been_reached(), True)

def test_ignored_clients(self):
"""
Should not increment counter if client is part of ignored_clients list.
"""
self.rate_limit = RateLimit(resource='test', client='localhost', ignored_clients=['localhost'],
max_requests=10, expire=2)
self.assertEqual(self.rate_limit.get_usage(), 0)
self.assertEqual(self.rate_limit.has_been_reached(), False)

self._make_10_requests()
self.assertEqual(self.rate_limit.get_usage(), 0)
self.assertEqual(self.rate_limit.has_been_reached(), False)

def test_expire(self):
"""
Should not raise TooManyRequests Exception when trying to increment for
Expand Down