David's development/operations Blog

on porting Knockknock to PyNaCl

Posted in Uncategorized by david415 on September 30, 2013

Firstly, Moxie’s excellent port-knocker, Knockknock, doesn’t really need improvement… this was just an exercise for me to learn exactly how Knockknock works and to practice using PyNaCl… and as it turned out, Scapy, as well. Here’s my working fork of Knockknock.

What have I done? I ported Knockknock to PyNaCl (uses libsodium which is a fork of DJB’s http://nacl.cr.yp.to/) for the message authentication and encryption — this immediately caused two implementation dependencies:

  1. increased ciphertext payload by 6 bytes (from 12 to 18 bytes)
  2. requires the client and server to keep track of a nonce counter

I didn’t absolutely need to use Scapy for the Knockknock client… but I chose to because it makes it easy to troubleshoot and explore handcrafted packet possibilities. Moxie originally used hping3 to encoded the cipher text in the IP header ID field and three TCP header fields: ACK, SYN and WINDOW for 12 bytes total ciphertext payload. At first I thought this might be the maximum amount of data we can hide in the TCP header… thinking that the header could only be 20 bytes… However the TCP header size can be extended if optional fields are used. This is how I get the Knockknock client to send 6 more bytes in TCP’s MSS optional field :

    (idField, seqField, ackField, winField, opt1, opt2, opt3, opt4, opt5, opt6) = unpack('!HIIHcccccc', packetData)


    tcp = TCP(dport   = int(knockPort), 
              flags   = 'S',
              seq     = seqField,
              ack     = ackField,
              window  = winField,
              options = [('MSS', pack('cccccc', opt1, opt2, opt3, opt4, opt5, opt6))] )

    ip = IP(dst=host, id=idField)



Note that the NaCl nonce size is 24 bytes. I create the ciphertext like this after incrementing the old client nonce value :

    port         = pack('H', int(port))

    counter      = profile.loadCounter()
    counter      = counter + 1

    nonce        = pack('LLL', 0,0,counter)
    ciphertext   = profile.encrypt(port, nonce)

After sending the packet, the Knockknock client must store the counter for future uses:


I modified the Profile class… The encrypt method pretty simply wraps secret box encrypt:

    def encrypt(self, plaintext, nonce):
        return self.box.encrypt(plaintext, nonce)

But the decrypt method must load and increment the counter and pack it into 24 bytes (the size of a secret box nonce) just as the client does. This extra inconvenience for the client and server to keep a counter allows us to send the ciphertext without the 24 byte nonce.

  def decrypt(self, ciphertext):
        """Load counter, increment, use it as a nonce decrypt data and store counter."""

        counter = self.loadCounter()
        counter += 1
        nonce = pack('LLL', 0, 0, counter)

            plaintext = self.box.decrypt(ciphertext, nonce)
        except CryptoError:
            # failed to decrypt
            return None

        port = unpack('H', plaintext)
        port = port[0]


        return port

I really like Moxie’s clever idea for the Knockknock daemon to receive the client’s ciphertext from the iptables packet log. While I was playing around with Knockknock and watching my iptables packet log I noticed many of the packets in the log had the OPT field set like this:

Sep 29 23:32:19 localhost kernel: REJECT IN=eth0 OUT= MAC=... SRC=x.x.x.x DST=y.y.y.y LEN=48 TOS=0x00 PREC=0x00 TTL=51 ID=63287 PROTO=TCP SPT=20 DPT=667 SEQ=2569293503 ACK=13308 99571 WINDOW=59458 RES=0x00 SYN URGP=0 OPT (02082C16C5E086D5)

For our purposes it doesn’t really matter what these bit fields are as long as we can see our 6 bytes show up in this iptables log field. So I wrote a Scapy program like this to test it out:

#!/usr/bin/env python

import binascii
from scapy.all import TCP, IP

ip  = IP(dst=myhost)
tcp = TCP(dport   = 6200, 
              flags   = 'S',
              seq     = 32456,
              ack     = 32456,
              window  = 32456,
              options = [('MSS',binascii.unhexlify("DEADBEEFCAFE"))])

I then observe that “DEADBEEFCAFE” showed up in my iptables packet log in the OPT field; 4 characters (2 hex bytes) from the start:

Sep 30 00:51:16 localhost kernel: REJECT IN=eth0 OUT= MAC=... SRC=y.y.y.y DST=x.x.x.x LEN=48 TOS=0x00 PREC=0x00 TTL=51 ID=1 PROTO=TCP SPT=20 DPT=6200 SEQ=32456 ACK=32456 WINDOW=32456 RES=0x00 SYN URGP=0 OPT (0208DEADBEEFCAFE)

After getting all that to work… I also played around with using NetFilter Sockets to grab the entire packet instead of just the header from the iptables LOG. This API: scapy-nflog-capture
not only has a Scapy specific interface but also provides a generator interface. I made a branch of Knockknock that uses the Netfilter socket… But this is a more complex interaction… and I think Moxie’s trick of reading packet headers from the iptables LOG is more elegant. However I might use the Netfilter Socket in a future project when I need to read in the entire packet. So I wrote this in Twisted: NFLOG_reader which is essentially a packet sniffer which receives packets based on iptables rules.

It seems my hack of using NaCl is not so useful since the client and server must keep track of the nonces and if they fall out of sync (from packet loss…) then the server will not be able to authenticate and decrypt the ciphertext from the client. I could fix this situation by sending the 24 byte nonce along with the ciphertext… which is possible to do now that I’ve figure out how to use the extended TCP header to carry more data. In this scenario the server and client still must keep track of the nonces they’ve seen… (I’m using the nonce as a counter to make it simple) Of course the server must also reject any reuse of an old nonce to prevent replay attacks. Maybe I’ll get around to making those code changes later.

Tornado/Motor coroutines

Posted in Uncategorized by david415 on September 22, 2013

In this next code example we use the asynchronous callback interface in a recursive coding pattern to enforce a maximum concurrency when processing tasks asynchronously :

    def send_task(self, task):
        self.current_concurrency += 1
        response = yield self.send(task)

    def work_loop(self, cursor):
        self.current_concurrency = 0
        for i in range(self.max_concurrency):
            result = yield cursor.fetch_next
            if result != None:
                job = cursor.next_object()
                task = self.prepare(job)
                future = self.send_task(task)
                self.io_loop.add_future(future, functools.partial(self.send_task_callback, cursor))

    def send_task_callback(self, cursor, future):
        self.current_concurrency -= 1
        if self.current_concurrency < self.max_concurrency:
            cursor.each(callback=functools.partial(self.each_job_callback, cursor))

    def each_job_callback(self, cursor, result, error):
        if error:
            raise error
        elif :
            job = result
            task = self.prepare(job)
            future  = self.send_task(task)
            self.io_loop.add_future(future, functools.partial(self.send_task_callback, cursor))

Instead we can use an iterative approach with coroutines to greatly shorten and simplify work_loop:

    def work_loop(self, cursor):
        workers = []
        for i in range(self.max_concurrency):
            yield workers

    def task_worker(self, cursor):
        while (yield cursor.fetch_next):
            job = cursor.next_object()
            task = self.prepare(job)
            result = yield self.send(task)

There are other advantages to the work_loop coroutine algorithm above; it does not mutate object state! The callback solution mutates self.current_concurrency. Why would you care about that? Firstly, because it is less complicated. Secondly, because if the algorithm doesn’t have to keep state in self.current_concurrency then a single instance could be used to process more than one database cursor. OK. I got it! Asynchronous coroutines can be used to produce some very elegant code.

porting a Tornado app from PyMongo to Motor

Posted in Uncategorized by david415 on September 7, 2013

I recently learned some things when I ported a 10,000 line Tornado app from PyMongo to Motor.

First of all, it is a bad idea to use PyMongo with Tornado because slow queries can block the entire Tornado thread. Tornado is a single threaded asynchronous framework; if a PyMongo database query takes a long time then it will block Tornado’s event loop and make your application crawl. Your seemingly fast PyMongo queries can unexpectedly become slower due to a change in data, schema or system load. Therefore it is much safer to use Motor than PyMongo in your Tornado application.

The goal here is to use an iterative approach to porting wherein each code change is relatively small AND the program should still work properly. We should be able to verify that a small component of the app works correctly.

So what are our options? At first there appear to be 2 options:

  1. Create a blocking PyMongo version and asynchronous Motor version for
    these functions that are used by the rest of the app
  2. Pick a low level PyMongo wrapper function and port it to Motor; it will be a coroutine and therefore will force all callers in it’s call tree to be coroutines.

If we choose option 1 then the downside is that we have two versions of the same code base to maintain. If the application is evolving quickly then this would not be a good approach. But upon closer examination perhaps this approach doesn’t really differ from option 2… Can we make a small incremental code change with this? Nope… changing a lower level function which wraps a PyMongo query cannot be changed in isolation because once it is changed to an asynchronous coroutine that uses Motor then all the callers must also be coroutines and use a yield statement to retrieve the results.

This means that you cannot control how big a code change would be necessary. But we can at least limit the complexity of the code change a bit… I will demonstrate below…

Now, I’d like to point out why I chose to use the Motor asynchronous coroutine syntax rather than the async callbacks directly. When using coroutines, exceptions are raised upon an error condition. If you need to catch them, you can, and if not then they’ll be surfaced to the developer.

Behold :

@gen.coroutine def save(db_async, data):
        result = yield motor.Op(db_async.profile.insert, data.serialize())

compared to error checking with callbacks :

def save(db_async, data):
    db_async.profile.insert(data.serialize(), callback=save_callback)

def save_callback(result, error):
    if error is not None:
        raise error

Way easier to use coroutines and let the exceptions be raised.

In my first naive attempt to port a Tornado app from PyMongo to Motor, it was not a good idea to try and port an entire GET or POST handler method to be completely asynchronous in one code change. In our app the GET and POST handler methods tend to use many different model functions that in turn make database calls. For me, this meant that what at first seemed like a small simple code change resulted in a large code change as I followed the call tree. One reason for this is that I am making non-backwards compatible code changes wherein ALL the calling sections of code need to be coroutines and gather the results of the async functions via a “yield” statement.

So here we have a find_by_email function:

def find_by_email(db_async, email, extend):
    '''Get a user by email'''
    query = {
        'email': email.lower(),
    user = yield motor.Op(db_async.accounts.find_one, query)
    raise gen.Return(user)

We must call it like this:

user = yield users.find_by_email(db_async=db_async, email=to_email, extend=False)

Using the yield statement to gather the results of find_by_email asynchronously means that the function calling find_by_email must also be a coroutine.

My first task in porting a Tornado web app from PyMongo to Motor is to make sure that our Tornado base RequestHandler class has access to a Motor db handle (I called it db_async) as well as a PyMongo handle (it was named “db”). But even if you are porting a Tornado app that is not a web app… you can do the equivalent and create a Motor database handle that lives side by side with the preexisting PyMongo db handle. Additionally on startup you must make sure to call open_sync() on your Motor handle.

The next step is to make all the GET and POST handler methods use the Tornado decorator gen.coroutine. Once this is working then you can begin the task of porting one or more low level model functions from PyMongo to Motor.

In our models section there are these mid-level functions that often call multiple lower level functions that wrap PyMongo queries. Therefore if I change a lower level function to use Motor… then… the calling mid-level function must either have ALL of it’s lower level functions be converted to Motor OR it must be passed db (PyMongo handle) AND db_async (Motor handle). Of course if I add an extra arg then all the calling parties must pass that additional argument. Anyhow that is what I’ve been doing in order to make small iterative code changes.

Consider this:

class MyContentHandler(MyBaseHandler):

    def get( self ):
        db       = self.db
        session  = self.current_session

        group_id = session.get('group_ids')[0]
        content   = mymodel.find_by_group_id(db=db, group_id=group_id)

        response = {
            'data': {
                'code': code,
                'status': 'success',
                'message': content,

        self.write( response )

in the “mymodels”:

def find_by_group_id(db, group_id):
    sessions = []
    query = { ... }
    results = db.notifications.find(query).sort('created_at', -1)
    for session in results:
        sessions = _add_actions(db=db, sessions=sessions)
    return sessions

def _add_actions(db, sessions):
    sessions_with_actions = []
    for session in sessions:
        query = {
            'content_id': session.get('_id'),
        shares = db.shares.find(query)
        session['actions'] = shares
    return sessions_with_actions

In preparation to port this to Motor I will add the coroutine decorator to the get handler method and make the Motor database handle available in self.db_async.

If I change find_by_group_id to accept only a Motor db handle instead of a PyMongo db handle then I have to also port _add_actions to Motor as well. However we can modify find_by_group_id to be a coroutine that takes a PyMongo and a Motor db handle like this:

def find_by_group_id(db, db_async, group_id):
    sessions = []
    query = { ... }
    results = yield motor.Op(db_async.notifications.find, query)
    for session in results:
        sessions = _add_actions(db=db, sessions=sessions)
    raise gen.Return(sessions)

And we call it like this:

content   = yield mymodel.find_by_group_id(db=db, db_async=db_async, group_id=group_id)

In this way we add a little tiny bit of complexity, an extra argument, and it helps us keep our iterative code changes small. When we are ready to port _add_actions to Motor then we get rid of the extra argument in find_by_group_id; like this:

def find_by_group_id(db_async, group_id):
    sessions = []
    query = { ... }
    sessions = yield motor.Op(db_async.notifications.find(query).sort('created_at', -1).limit(limit).to_list)
    sessions = yield _add_actions(db_async=db_async, sessions=sessions)
    raise gen.Return(sessions)

def _add_actions(db_async, sessions):
    sessions_with_actions = []
    for session in sessions:
        query = {
            'content_id': session.get('_id'),
        shares = yield motor.Op(db_async.shares.find, query)
        session['actions'] = shares
    raise gen.Return(sessions_with_actions)

And lastly, whenever possible, yield a list of futures that way they run concurrently :

future_user          = profiles.find_by_id(db_async=db_async, id=user_id)
future_group         = motor.Op(db_async.group.find_one, {'id':group_id})
future_session       = motor.Op(db_async.group.find_one, {'id':session_id})
user, group, session = yield [future_user, future_group, future_session]

In general I don’t like to write code that is temporary. If I were to write 2 versions of each database function, a blocking version and an asynchronous version, then I’d have a lot more temporary code to work with. It also would be possible to write both versions using Motor. Sometimes its better to just suck it up and port the application without writing a lot of temporary code. I prefer to make forward progress. Onward!

np-complete sysadmin

Posted in Uncategorized by david415 on January 29, 2013

Lately I’ve been asked to solve np-complete programming problems during interviews. This is interesting! It lead me to find Yizhan Sun’s excellent dissertation paper called “Complexity of System Configuration Management“. This paper blew my mind even though I cannot understand the formal mathematical proofs; the concepts are easy enough to grok… This inspired me to sign up for a Coursera class that covers mathematical proofs.

The paper illustrates many insights into the theory behind systems administration from a computational theory perspective. Read it!

using mincore to detect page cache efficiency of Mysql’s MyIsam

Posted in Uncategorized by david415 on November 21, 2010

We use MyIsam tables on our archive mysql database servers in an append only manner. This means that we’ve got data files that are continuously growing in size… In each mysql database archive shard there is a “hot” replica that is responsible for caching (in the Linux file system page cache) the last 4 hours worth of data. This latest 4 hours worth of data at the end of the file we call the “hot-window”. I’ve written a couple tools to help us detect if the hot-window is in page cache. I’ve added report_filechanges and pages_by_time to my python-ftools project on github.

Usage: report_filechanges [options]

-h, --help show this help message and exit
historical log directory
director to monitor

report_filechanges is called from cron every five minutes; it produces a log file entry for each file in the Mysql MyIsam data directory. Each log entry contains the filename, the time (in Unix epoch seconds) and the file size. We later use these historical file sizes as offsets in the file. This gives us the ability to determine the boundary of the hot region.

Usage: pages_by_time [options]

-h, --help show this help message and exit
--evict Evict historical region of files
--summary Summarize page cache usage of file regions
historical log directory
eviction time

pages_by_time uses these logs files to summarize page cache usage (via the mincore system call) or to evict pages from file system page cache (via the fadvise system call).

Cassandra data storage performance statistics with inotify and fincore

Posted in Systems Engineering / Unix Systems Operations by david415 on September 3, 2010

We were interested in knowing which files were accessed (reads and seeks)
the most among the Cassandra data files (index, filter and data files)…
I wrote this simple Python program, inotify-access, to print these file read access statistics for a given time duration and directory. Find it in my github repository:

This program makes use of the Linux kernel’s inotify system call.
If you run Debian/Ugabuga (I mean Ubuntu) then you’ll need to install the pynotify library: apt-get install python-pyinotify

To determine page cache usage of these files you can use fincore. I have forked the linux-ftools’s fincore to make the output more easily parsable at my github repository here:

Additionally I’ve written cassandra_pagecache_usage, a Python program that uses the mincore system call to report page cache usage for Cassandra data sets. Previously this program used to parse the output of linux-ftools’s fincore. However I have since switched to using the
the Python C extension fincore_ratio which returns a 2 tuples (cached pages, total pages). python-ftools is a linux-ftools port to Python C extensions; find it in my github repository here :

I’ve written a Python version of fadvise for the commandline…
fadvise example usage:

Perhaps your cassandra node has been rebooted. You could “warm up” certain Column Families like this :

./fadvise -m willneed /mnt/var/cassandra/data/BunnyFufu/ForestActivity*

Find cassandra_pagecache_usage at my github repository here:

Usage: cassandra_pagecache_usage [options] <cassandra-data-directory>
  -h, --help            show this help message and exit
  -c, --columnfamily-summarize
                        Summarize cached Cassandra data on a per Column Family
  --exclude-filter      Exclude statistics for Cassandra Filter files.
  --exclude-index       Exclude statistics for Cassandra Index files.
  --exclude-data        Exclude statistics for Cassandra Data files.

example output:

my-cassandra-node:~/bin# PYTHONPATH=~/lib ./cassandra_pagecache_usage -c /mnt/var/cassandra/data/BunnyFufu/
Column Family    Bytes in FS page-cache    
ForestActivity   3712839680                
Indexes          2902822912                
AnimalIndex      2369015808                
AnimalCounts     1470619648                
Items            786214912                 
Activity         264978432                 
Animals          133816320                 
Hops             127442944                 

highly available mysql database topology

Posted in Systems Engineering / Unix Systems Operations by david415 on January 16, 2010

This Mysql database model I am about to describe could be applied to a distributed database built on top of Mysql.
In that sense I’m describing a single shard in a distributed database.
The shard consists of one master replicating to two slaves.

When I talk about database replicas (a master and slaves) crashing, I’m referring to any catastrophic failure that causes us to have to restore the entire dataset. Mysql replication has a number of race conditions that can result in data corruption when the Mysql node is rebooted, seg faulted, kernel paniced or otherwise crashed. Fiery explosions and semi automatic gun fire obviously also cause hardware failure and data loss. In the case of hardware failures we use one of the spare servers…

In this model we are prepared to handle any of these catastrophic failures to a single node in the shard.
Either the master or one of the two slaves crash. If a slave crashes we use the other slave as our data source to recover from.

If the master database crashes then we promote a slave to be master.
The old master becomes a slave; we recover it’s data from the other slave.
This procedure assumes all the slaves are at the same replication binary log position.
I imagine it would be nice to use Google’s Global Transaction ID patch (read about Global Transactions IDs here); That way if the slaves were behind in replication, the most up to date slave could be promoted
to be the master and the other slave could easily resume replication with the new master.

Without Google’s Global Transaction ID patch to Mysql we’d have to be more clever to be able to restore the slave from the newly promoted master since the new master will have a different set of transaction IDs than the old master. This shouldn’t be too hard with log_slave_updates enabled on all replicas and some simple subtraction of binary log positions.

Normally client queries would only be directed to the master database. If we were to scale reads to the slaves (this is a waste of cache!) then we may not have enough io capacity available to both restore the crashed node and service database queries within the SLA (service level agreement)… Therefore if we are going to scale reads to all three nodes we must first do load testing to determine the max queries per second we can safely send each node during normal operation and during recovering…

Without the benefit of load testing we’d have to send the read queries to only the master so that we can guarantee the io capacity of one replica for operational capacity for recovering from outages, rolling upgrades etc.

running linux with no swap

Posted in Systems Engineering / Unix Systems Operations by david415 on November 21, 2009

I do not ever want my laptop or any of the servers I maintain to swap.
For all my applications it is never a good idea to swap. RAM is cheap.
It is true that I don’t want the OOM killer to kill the wrong process… But I don’t worry about
that because I know how to use the oom_adj to prevent Mysql or sshd from getting killed.

According to a knowledgeable Linux kernel developer I spoke with if you are going to disable swap (as in swapoff -a) it is beneficial to recompile with the kernel config option SWAP=n. This avoids a performance problem; it’s been reported that kswapd will otherwise use lots of CPU. While your configuring your new kernel you may also want to disable tmpfs

# swapon -a
swapon: /dev/mapper/longhaul-swap_1: swapon failed: Function not implemented

Dear Linux,
Please do not swap ever again!
kthx bye

Mysql Innodb Hotcopy in Python

Posted in Systems Engineering / Unix Systems Operations by david415 on February 1, 2009

Lately for Spinn3r I’ve been writing programs in Python to automate database operations. This hotcopy script utilizes LVM. Previously we didn’t use LVM and so this script used to do an Innodb Freeze. The simple idea here is to restore a mysql shard replica’s data from another replica…

Look at the code here:
mysql cluster tools @ bitbucket.org

The cool feature this hotcopy script has is an undo/restore mechanism that
kicks in when an exception is caught. Basically I’ve implemented a nullary function queue.
With each state change, I push the reverse (or undo/restore operation) onto the queue. When an exception is caught the rollback() method repeatedly pops and executes the last nullary off the queue until there are none left.

What I’m calling a nullary function is merely a function call wrapped in a closure (although maybe that’s not correct because they say Python doesn’t have true closures) in the form of a lambda with zero arguments.

Python’s excellent exception handling makes this rollback() feature really useful because there are lots of moving parts at work that could break and throw an exception. The hotcopy process causes several state changes on the source replica such as : set single user mode, stop replication (if the source is a slave), take LVM snapshot, mount snapshot etc…

I don’t like LVM snapshots laying around so we can COW forever! Nor would I want a database server to remain in “single user mode”…

Three ways to extend this project:

  • Patch MySQL for faster InnoDB crash recovery.
  • A distributed/highly available persistent storage mechanism for the restore queue to allow a rollback even after the server running the hotcopy program, crashes.
  • A mechanism to invoke this program/API to fully automate crash recovery. A centralized design involving a voting protocol…
  • I’m sure many like Spinn3r have similar infrastructure goals. The above InnoDB modification is one of several Desirable Innodb Features that Spinn3r will probably throw down cash for…

    Check it out at bitbucket.org, my Mysql Innodb Hotcopy program.
    This is not even a release candidate… but if anyone wants to look at the code… feel free.

    Please leave your thoughts and comments.

    Ganglia rules… sort of.

    Posted in Systems Engineering / Unix Systems Operations by david415 on February 1, 2009

    At first Ganglia seems like an excellent tool.
    It seems to have an excellently efficient and reliable/highly available design.
    But actually Ganglia is very brittle and suffers numerous bugs and design flaws.

    Ganglia seems to be mostly self configuring… except for a few modules out there that need configuration parameters. If I need to use these modules I’ll modify them to be self configuring like I did with multidisk.py… Anyone who maintains a fair sized cluster knows with a little code it pays to be lazy.

    Ganglia seems to be way better than Cacti and the rest. Initially I was disappointed with the standard set of modules which didn’t allow me to monitor the throughput on two different ethernet interfaces. Perhaps the ganglia-developers don’t run multi-homed servers. Or maybe they just don’t care about how much throughput they use. I wrote a python module to graph usage for an internal and external interface because this will help us project how much we’d be paying a different data-center facility to host our cluster.

    Gilad Raphaelli wrote a really cool embedded python module for monitoring mysql metrics… It monitors 100 metrics including various Innodb buffers. Perfect! It also comes with one custom report for queries :

    mysql query report

    innodb transactionsl
    mysql threads

    I haven’t had time to look at it yet but Silvan Mühlemann wrote
    a custom report php script for mysql

    Admittedly I don’t know how best to use gmond’s python module interface even though I’ve written several modules already… I think its supposed to make it easy to write metric monitors with some embedded python code. But I like using gmetric more. It seems to be less code to produce the equivalent monitoring of metrics than the embedded python interface.

    Also I’ve noticed people trying to use the python interface in interesting ways… e.g. spawning metric collector threads that populate a cache etc. I was hoping to be able to write ganglia metric modules without having to think about threads, cache and race conditions. Couldn’t a module harness take care of these details?

    I’m working on gmetric-daemon which is a simple python forking daemon with a modular interface that calls gmetric (via system() or popen())… It’s not very memory efficient.
    Perhaps using gmetric in this way is a silly approach… But right now I think I like it.

    Here’s the gitHub repository for the work in progress I call gmetric-daemon

    I’m going to try and make available (e.g. opensource via apache software license)
    lots more code that I write extending Ganglia.

    I might even write modules that send a “passive” Nagios alerts about metrics exceeding thresholds.