Client
def log(msg, count=1):
local_redis.sadd('tracked_keys', msg)
local_redis.incrby(msg, count)
log('ask', 5)
log('respond')
log('homepage_view')Fisher
General Overview
- Fisher manages our realtime event stats across hundreds of machines
- Each machine has a local redis instance that
app stats are written to - Stats are pulled every minute
by a central collector
Collector
def fetch(host):
r = Redis(host)
keys = r.smembers('tracked_keys')
pipe = r.pipeline()
for key in keys:
pipe.getset(key, 0)
counts = pipe.execute()
return zip(keys, [int(c) for c in counts])Client Redis Log
SADD tracked_keys "ask"
:1
INCRBY ask 5
:5
SADD tracked_keys "respond"
:1
INCRBY respond 1
:1
SADD tracked_keys "homepage_view"
:1
INCRBY homepage_view 1
:1Collector Redis Log
SMEMBERS tracked_keys
*3
$9
"respond"
$5
"ask"
$15
"homepage_view"
GETSET respond 0
$1
1
GETSET ask 0
$1
5
GETSET homepage_view 0
$1
1
Gevent parallelization
def parallel_fetch(hosts):
pool = gevent.pool.Pool(100)
coros = []
with gevent.Timeout(10, False):
for host in hosts:
coros.append(pool.spawn(fetch, host))
pool.join()
stats = defaultdict(lambda: 0)
for job in coros:
if job.ready() and job.value:
for msg, count in job.value:
stats[msg] += count
return statsWrite the stats
- ☞ A plaintext file for Zenoss
- fetched via HTTP - ☞ A custom timeseries database
- a thin wrapper around Redis - We also have a thin javascript layer on that...
Rolling out a new feature
Contact
- @sku_
- formspring.me/jdmaturen
- jdmaturen on #redis
- idlerice.com