Skip to content

Commit 5394d45

Browse files
cslzchenfelliott
authored andcommitted
Implement rate-limiting in WB with redis
* Add rate-limiting support to WaterButler via redis. This implementation uses the fixed window algorithm. * METHOD: Users are distinguished first by their credentials and then by their IP address. The rate limiter recognizes different types of auth and rate-limits each type separately. The four recognized auth types are: OSF cookie, OAuth bearer token, basic auth with base64-encoded username/password, and un-authed. OSF cookies, OAuth access tokens, and base64-encoded usernames/passwords are used as redis keys during rate-limiting. WB obfuscates them for the same reason that only password hashes are stored in a database. SHA-256 is used in this case. A prefix is also added to the digest to identify which type it is. The "No Auth" case is hashed as well (unnecessarily) so that the keys all have the same look and length. Auth by OSF cookie currently bypasses the rate limiter to avoid throttling web users. * CONFIGURATION: Rate limiting settings are found in `waterbutler.server.settings`. By default, WB allows 3600 requests per auth per hour. Rate-limiting is turned OFF by default; set `ENABLE_RATE_LIMITING` to `True` turn it on. The relevant envvars are: SERVER_CONFIG_ENABLE_RATE_LIMITING: Boolean. Defaults to `False`. SERVER_CONFIG_REDIS_HOST: The host redis is listening on. Default is '192.168.168.167'. SERVER_CONFIG_REDIS_PORT: The port redis is listening on. Default is '6379'. SERVER_CONFIG_REDIS_PASSWORD: The password for the configured redis instance. Default is `None`. SERVER_CONFIG_RATE_LIMITING_FIXED_WINDOW_SIZE: Number of seconds until the redis key expires. Default is 3600s. SERVER_CONFIG_RATE_LIMITING_FIXED_WINDOW_LIMIT: Number of reqests permitted while the redis key is active. Default is 3600. * BEHAVIOR: Return the Retry-After header in the 429 response if the limit is hit. This header states when it will be acceptable to send another request. Other informative headers are included to provide context, though currently only after the rate limiting has been enforced. If rate-limiting is enabled and WB is unable to reach redis, a 503 Service Unavailable error will be thrown. Since redis is not expected to be available during ci, rate limiting is turned off. * Grab-bag of related updates: * Bump redis dep to 3.3.8. No consequential changes. * Don't throw errors in the error handling. Provide a fallback for the `resource` attribute if rate-limiting kicks in before that has been initialized. * Update some docstrings to clarify return values and process. * Refactor test rate-limiting auth testing to only extract data as needed. * Add docs to settings; use `config.get_bool()` on booleans. rebase: add password support to conn
1 parent 47ad15e commit 5394d45

File tree

9 files changed

+227
-3
lines changed

9 files changed

+227
-3
lines changed

dev-requirements.txt

-1
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,3 @@ pytest==5.2.1
1414
pytest-asyncio==0.10.0
1515
pytest-cov==2.8.1
1616
python-coveralls==2.9.3
17-
redis==3.3.8

requirements.txt

+1
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ pyjwt==1.4.0
1818
python-dateutil==2.5.3
1919
pytz==2017.2
2020
sentry-sdk==0.14.4
21+
redis==3.3.8
2122
setuptools==37.0.0
2223
stevedore==1.2.0
2324
tornado==6.0.3

tests/conftest.py

+4
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@
66
'CELERY_RESULT_BACKEND': 'redis://'
77
}
88

9+
10+
from waterbutler.server import settings as server_settings
11+
server_settings.ENABLE_RATE_LIMITING = False
12+
913
import aiohttpretty
1014

1115

tests/core/test_exceptions.py

+2
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ class TestExceptionSerialization:
3838
exceptions.UninitializedRepositoryError,
3939
exceptions.UnexportableFileTypeError,
4040
exceptions.InvalidProviderConfigError,
41+
exceptions.WaterButlerRedisError,
42+
exceptions.TooManyRequests,
4143
])
4244
def test_tolerate_dumb_signature(self, exception_class):
4345
"""In order for WaterButlerError-inheriting exception classes to survive

waterbutler/core/exceptions.py

+44
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33

44
from aiohttp.client_exceptions import ContentTypeError
55

6+
from waterbutler.server import settings
7+
68

79
DEFAULT_ERROR_MSG = 'An error occurred while making a {response.method} request to {response.url}'
810

@@ -55,6 +57,48 @@ def __str__(self):
5557
return '{}, {}'.format(self.code, self.message)
5658

5759

60+
class TooManyRequests(WaterButlerError):
61+
"""Indicates the user has sent too many requests in a given amount of time and is being
62+
rate-limited. Thrown as HTTP 429, ``Too Many Requests``. Exception response includes headers to
63+
inform user when to try again. Headers are:
64+
65+
* ``Retry-After``: Epoch time after which the rate-limit is reset
66+
* ``X-Waterbutler-RateLimiting-Window``: The number of seconds after the first request when\
67+
the limit resets
68+
* ``X-Waterbutler-RateLimiting-Limit``: Total number of requests that may be sent within the\
69+
window
70+
* ``X-Waterbutler-RateLimiting-Remaining``: How many more requests can be sent during the window
71+
* ``X-Waterbutler-RateLimiting-Reset``: Seconds until the rate-limit is reset
72+
"""
73+
def __init__(self, data):
74+
if type(data) != dict:
75+
message = ('Too many requests issued, but error lacks necessary data to build proper '
76+
'response. Got:({})'.format(data))
77+
else:
78+
message = {
79+
'error': 'API rate-limiting active due to too many requests',
80+
'headers': {
81+
'Retry-After': data['retry_after'],
82+
'X-Waterbutler-RateLimiting-Window': settings.RATE_LIMITING_FIXED_WINDOW_SIZE,
83+
'X-Waterbutler-RateLimiting-Limit': settings.RATE_LIMITING_FIXED_WINDOW_LIMIT,
84+
'X-Waterbutler-RateLimiting-Remaining': data['remaining'],
85+
'X-Waterbutler-RateLimiting-Reset': data['reset'],
86+
},
87+
}
88+
super().__init__(message, code=HTTPStatus.TOO_MANY_REQUESTS, is_user_error=True)
89+
90+
91+
class WaterButlerRedisError(WaterButlerError):
92+
"""Indicates the Redis server has returned an error. Thrown as HTTP 503, ``Service Unavailable``
93+
"""
94+
def __init__(self, redis_command):
95+
96+
message = {
97+
'error': 'The Redis server failed when processing command {}'.format(redis_command),
98+
}
99+
super().__init__(message, code=HTTPStatus.SERVICE_UNAVAILABLE, is_user_error=False)
100+
101+
58102
class InvalidParameters(WaterButlerError):
59103
"""Errors regarding incorrect data being sent to a method should raise either this
60104
Exception or a subclass thereof. Defaults status code to 400, Bad Request.

waterbutler/server/api/v1/core.py

+21-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import logging
2+
13
import tornado.web
24
import tornado.gen
35
import tornado.iostream
@@ -8,6 +10,8 @@
810
from waterbutler.server import utils
911
from waterbutler.core import exceptions
1012

13+
logger = logging.getLogger(__name__)
14+
1115

1216
class BaseHandler(utils.CORsMixin, utils.UtilMixin, tornado.web.RequestHandler):
1317

@@ -25,7 +29,23 @@ def write_error(self, status_code, exc_info):
2529
scope.level = 'info'
2630

2731
self.set_status(int(exc.code))
28-
finish_args = [exc.data] if exc.data else [{'code': exc.code, 'message': exc.message}]
32+
33+
# If the exception has a `data` property then we need to handle that with care.
34+
# The expectation is that we need to return a structured response. For now, assume
35+
# that involves setting the response headers to the value of the `headers`
36+
# attribute of the `data`, while also serializing the entire `data` data structure.
37+
if exc.data:
38+
self.set_header('Content-Type', 'application/json')
39+
headers = exc.data.get('headers', None)
40+
if headers:
41+
for key, value in headers.items():
42+
self.set_header(key, value)
43+
finish_args = [exc.data]
44+
self.write(exc.data)
45+
else:
46+
finish_args = [{'code': exc.code, 'message': exc.message}]
47+
self.write(exc.message)
48+
2949
elif issubclass(etype, tasks.WaitTimeOutError):
3050
self.set_status(202)
3151
scope.level = 'info'

waterbutler/server/api/v1/provider/__init__.py

+13-1
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,13 @@
1414
from waterbutler.core import remote_logging
1515
from waterbutler.server.auth import AuthHandler
1616
from waterbutler.core.log_payload import LogPayload
17+
from waterbutler.core.exceptions import TooManyRequests
1718
from waterbutler.core.streams import RequestStreamReader
19+
from waterbutler.server.settings import ENABLE_RATE_LIMITING
1820
from waterbutler.server.api.v1.provider.create import CreateMixin
1921
from waterbutler.server.api.v1.provider.metadata import MetadataMixin
2022
from waterbutler.server.api.v1.provider.movecopy import MoveCopyMixin
23+
from waterbutler.server.api.v1.provider.ratelimiting import RateLimitingMixin
2124

2225
logger = logging.getLogger(__name__)
2326
auth_handler = AuthHandler(settings.AUTH_HANDLERS)
@@ -32,13 +35,22 @@ def list_or_value(value):
3235
return [item.decode('utf-8') for item in value]
3336

3437

38+
# TODO: the order should be reverted though it doesn't have any functional effect for this class.
3539
@tornado.web.stream_request_body
36-
class ProviderHandler(core.BaseHandler, CreateMixin, MetadataMixin, MoveCopyMixin):
40+
class ProviderHandler(core.BaseHandler, CreateMixin, MetadataMixin, MoveCopyMixin, RateLimitingMixin):
3741
PRE_VALIDATORS = {'put': 'prevalidate_put', 'post': 'prevalidate_post'}
3842
POST_VALIDATORS = {'put': 'postvalidate_put'}
3943
PATTERN = r'/resources/(?P<resource>(?:\w|\d)+)/providers/(?P<provider>(?:\w|\d)+)(?P<path>/.*/?)'
4044

4145
async def prepare(self, *args, **kwargs):
46+
47+
if ENABLE_RATE_LIMITING:
48+
logger.debug('>>> checking for rate-limiting')
49+
limit_hit, data = self.rate_limit()
50+
if limit_hit:
51+
raise TooManyRequests(data=data)
52+
logger.debug('>>> rate limiting check passed ...')
53+
4254
method = self.request.method.lower()
4355

4456
# TODO Find a nicer way to handle this
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
import hashlib
2+
import logging
3+
from datetime import datetime, timedelta
4+
5+
from redis import Redis
6+
from redis.exceptions import RedisError
7+
8+
from waterbutler.server import settings
9+
from waterbutler.core.exceptions import WaterButlerRedisError
10+
11+
logger = logging.getLogger(__name__)
12+
13+
14+
class RateLimitingMixin:
15+
""" Rate-limiting WB API with Redis using the "Fixed Window" algorithm.
16+
"""
17+
18+
def __init__(self):
19+
20+
self.WINDOW_SIZE = settings.RATE_LIMITING_FIXED_WINDOW_SIZE
21+
self.WINDOW_LIMIT = settings.RATE_LIMITING_FIXED_WINDOW_LIMIT
22+
self.redis_conn = Redis(host=settings.REDIS_HOST, port=settings.REDIS_PORT,
23+
password=settings.REDIS_PASSWORD)
24+
25+
def rate_limit(self):
26+
""" Check with the WB Redis server on whether to rate-limit a request. Returns a tuple.
27+
First value is `True` if the limit is reached, `False` otherwise. Second value is the
28+
rate-limiting metadata (nbr of requests remaining, time to reset, etc.) if the request was
29+
rate-limited.
30+
"""
31+
32+
limit_check, redis_key = self.get_auth_naive()
33+
logger.debug('>>> RATE LIMITING >>> check={} key={}'.format(limit_check, redis_key))
34+
if not limit_check:
35+
return False, None
36+
37+
try:
38+
counter = self.redis_conn.incr(redis_key)
39+
except RedisError:
40+
raise WaterButlerRedisError('INCR {}'.format(redis_key))
41+
42+
if counter > self.WINDOW_LIMIT:
43+
# The key exists and the limit has been reached.
44+
try:
45+
retry_after = self.redis_conn.ttl(redis_key)
46+
except RedisError:
47+
raise WaterButlerRedisError('TTL {}'.format(redis_key))
48+
logger.debug('>>> RATE LIMITING >>> FAIL >>> key={} '
49+
'counter={} url={}'.format(redis_key, counter, self.request.full_url()))
50+
data = {
51+
'retry_after': int(retry_after),
52+
'remaining': 0,
53+
'reset': str(datetime.now() + timedelta(seconds=int(retry_after))),
54+
}
55+
return True, data
56+
elif counter == 1:
57+
# The key does not exist and `.incr()` returns 1 by default.
58+
try:
59+
self.redis_conn.expire(redis_key, self.WINDOW_SIZE)
60+
except RedisError:
61+
raise WaterButlerRedisError('EXPIRE {} {}'.format(redis_key, self.WINDOW_SIZE))
62+
logger.debug('>>> RATE LIMITING >>> NEW >>> key={} '
63+
'counter={} url={}'.format(redis_key, counter, self.request.full_url()))
64+
else:
65+
# The key exists and the limit has not been reached.
66+
logger.debug('>>> RATE LIMITING >>> PASS >>> key={} '
67+
'counter={} url={}'.format(redis_key, counter, self.request.full_url()))
68+
69+
return False, None
70+
71+
def get_auth_naive(self):
72+
""" Get the obfuscated authentication / authorization credentials from the request. Return
73+
a tuple ``(limit_check, auth_key)`` that tells the rate-limiter 1) whether to rate-limit,
74+
and 2) if so, limit by what key.
75+
76+
Refer to ``tornado.httputil.HTTPServerRequest`` for more info on tornado's request object:
77+
https://www.tornadoweb.org/en/stable/httputil.html#tornado.httputil.HTTPServerRequest
78+
79+
This is a NAIVE implementation in which WaterButler rate-limiter only checks the existence
80+
of auth creds in the requests without further verifying them with the OSF. Invalid creds
81+
will fail the next OSF auth part anyway even if it passes the rate-limiter.
82+
83+
There are four types of auth: 1) OAuth access token, 2) basic auth w/ base64-encoded
84+
username/password, 3) OSF cookie, and 4) no auth. The naive implementation checks each
85+
method in this order. Only cookie-based auth is permitted to bypass the rate-limiter.
86+
This order does not care about the validity of the auth mechanism. An invalid Basic auth
87+
header + an OSF cookie will be rate-limited according to the Basic auth header.
88+
89+
TODO: check with OSF API auth to see how it deals with multiple auth options.
90+
"""
91+
92+
auth_hdrs = self.request.headers.get('Authorization', None)
93+
94+
# CASE 1: Requests with a bearer token (PAT or OAuth)
95+
if auth_hdrs and auth_hdrs.startswith('Bearer '): # Bearer token
96+
bearer_token = auth_hdrs.split(' ')[1] if auth_hdrs.startswith('Bearer ') else None
97+
logger.debug('>>> RATE LIMITING >>> AUTH:TOKEN >>> {}'.format(bearer_token))
98+
return True, 'TOKEN__{}'.format(self._obfuscate_creds(bearer_token))
99+
100+
# CASE 2: Requests with basic auth using username and password
101+
if auth_hdrs and auth_hdrs.startswith('Basic '): # Basic auth
102+
basic_creds = auth_hdrs.split(' ')[1] if auth_hdrs.startswith('Basic ') else None
103+
logger.debug('>>> RATE LIMITING >>> AUTH:BASIC >>> {}'.format(basic_creds))
104+
return True, 'BASIC__{}'.format(self._obfuscate_creds(basic_creds))
105+
106+
# CASE 3: Requests with OSF cookies
107+
# SECURITY WARNING: Must check cookie last since it can only be allowed when used alone!
108+
cookies = self.request.cookies or None
109+
if cookies and cookies.get('osf'):
110+
osf_cookie = cookies.get('osf').value
111+
logger.debug('>>> RATE LIMITING >>> AUTH:COOKIE >>> {}'.format(osf_cookie))
112+
return False, 'COOKIE_{}'.format(self._obfuscate_creds(osf_cookie))
113+
114+
# TODO: Work with DevOps to make sure that the `remote_ip` is the real IP instead of our
115+
# load balancers. In addition, check relevatn HTTP headers as well.
116+
# CASE 4: Requests without any expected auth (case 1, 2 or 3 above).
117+
remote_ip = self.request.remote_ip or 'NOI.PNO.IPN.OIP'
118+
logger.debug('>>> RATE LIMITING >>> AUTH:NONE >>> {}'.format(remote_ip))
119+
return True, 'NOAUTH_{}'.format(self._obfuscate_creds(remote_ip))
120+
121+
@staticmethod
122+
def _obfuscate_creds(creds):
123+
"""Obfuscate authentication/authorization credentials: cookie, access token and password.
124+
125+
It is not recommended to store the plain OSF cookie or the OAuth bearer token as key and it
126+
is evil to store the base64-encoded username and password as key since it is reversible.
127+
"""
128+
129+
return hashlib.sha256(creds.encode('utf-8')).hexdigest().upper()

waterbutler/server/settings.py

+13
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,16 @@
3030
if not settings.DEBUG:
3131
assert HMAC_SECRET, 'HMAC_SECRET must be specified when not in debug mode'
3232
HMAC_SECRET = (HMAC_SECRET or 'changeme').encode('utf-8')
33+
34+
35+
# Configs for WB API Rate-limiting with Redis
36+
ENABLE_RATE_LIMITING = config.get_bool('ENABLE_RATE_LIMITING', False)
37+
REDIS_HOST = config.get('REDIS_HOST', '192.168.168.167')
38+
REDIS_PORT = config.get('REDIS_PORT', '6379')
39+
REDIS_PASSWORD = config.get('REDIS_PASSWORD', None)
40+
41+
# Number of seconds until the redis key expires
42+
RATE_LIMITING_FIXED_WINDOW_SIZE = int(config.get('RATE_LIMITING_FIXED_WINDOW_SIZE', 3600))
43+
44+
# number of reqests permitted while the redis key is active
45+
RATE_LIMITING_FIXED_WINDOW_LIMIT = int(config.get('RATE_LIMITING_FIXED_WINDOW_LIMIT', 3600))

0 commit comments

Comments
 (0)