I do not ever want my laptop or any of the servers I maintain to swap.
For all my applications it is never a good idea to swap. RAM is cheap.
It is true that I don’t want the OOM killer to kill the wrong process… But I don’t worry about
that because I know how to use the oom_adj to prevent Mysql or sshd from getting killed.

According to a knowledgeable Linux kernel developer I spoke with if you are going to disable swap (as in swapoff -a) it is beneficial to recompile with the kernel config option SWAP=n. This avoids a performance problem; it’s been reported that kswapd will otherwise use lots of CPU. While your configuring your new kernel you should also disable tmpfs…

Dear Linux,
Please do not swap ever again!
kthx bye

Lately for Spinn3r I’ve been writing programs in Python to automate database operations. This hotcopy script utilizes LVM. Previously we didn’t use LVM and so this script used to do an Innodb Freeze. The simple idea here is to restore a mysql shard replica’s data from another replica…

Look at the code here:
mysql cluster tools @ bitbucket.org

The cool feature this hotcopy script has is an undo/restore mechanism that
kicks in when an exception is caught. Basically I’ve implemented a nullary function queue.
With each state change, I push the reverse (or undo/restore operation) onto the queue. When an exception is caught the rollback() method repeatedly pops and executes the last nullary off the queue until there are none left.

What I’m calling a nullary function is merely a function call wrapped in a closure (although maybe that’s not correct because they say Python doesn’t have true closures) in the form of a lambda with zero arguments.

Python’s excellent exception handling makes this rollback() feature really useful because there are lots of moving parts at work that could break and throw an exception. The hotcopy process causes several state changes on the source replica such as : set single user mode, stop replication (if the source is a slave), take LVM snapshot, mount snapshot etc…

I don’t like LVM snapshots laying around so we can COW forever! Nor would I want a database server to remain in “single user mode”…

Three ways to extend this project:

I’m sure many like Spinn3r have similar infrastructure goals. The above InnoDB modification is one of several Desirable Innodb Features that Spinn3r will probably throw down cash for…

Check it out at bitbucket.org, my Mysql Innodb Hotcopy program.
This is not even a release candidate… but if anyone wants to look at the code… feel free.

Please leave your thoughts and comments.

Ganglia is an excellent tool. Ganglia makes me happy.
It seems to have an excellently efficient and reliable/highly available design.

Ganglia seems to be mostly self configuring… except for a few modules out there that need configuration parameters. If I need to use these modules I’ll modify them to be self configuring like I did with multidisk.py… Anyone who maintains a fair sized cluster knows with a little code it pays to be lazy.

Ganglia seems to be way better than Cacti and the rest. Initially I was disappointed with the standard set of modules which didn’t allow me to monitor the throughput on two different ethernet interfaces. Perhaps the ganglia-developers don’t run multi-homed servers. Or maybe they just don’t care about how much throughput they use. I wrote a python module to graph usage for an internal and external interface because this will help us project how much we’d be paying a different data-center facility to host our cluster.

Gilad Raphaelli wrote a really cool embedded python module for monitoring mysql metrics… It monitors 100 metrics including various Innodb buffers. Perfect! It also comes with one custom report for queries :

I haven’t had time to look at it yet but Silvan Mühlemann wrote
a custom report php script for mysql

Admittedly I don’t know how best to use gmond’s python module interface even though I’ve written several modules already… I think its supposed to make it easy to write metric monitors with some embedded python code. But I like using gmetric more. It seems to be less code to produce the equivalent monitoring of metrics than the embedded python interface.

Also I’ve noticed people trying to use the python interface in interesting ways… e.g. spawning metric collector threads that populate a cache etc. I was hoping to be able to write ganglia metric modules without having to think about threads, cache and race conditions. Couldn’t a module harness take care of these details?

I’m working on gmetric-daemon which is a simple python forking daemon with a modular interface that calls gmetric (via system() or popen())… It’s not very memory efficient.
Perhaps using gmetric in this way is a silly approach… But right now I think I like it.

Here’s the gitHub repository for the work in progress I call gmetric-daemon

I’m going to try and make available (e.g. opensource via apache software license)
lots more code that I write extending Ganglia.

I might even write modules that send a “passive” Nagios alerts about metrics exceeding thresholds.

I got Postfix e-mail submissions to Bugzilla (3.0.5) working properly.
Perhaps these notes of mine could save someone some trouble when attempting this.

Certainly postfix could accomplish e-mail submissions via a custom transport using a pipe.
I however decided to use a pipe in the /etc/aliases file; mine contains this important line :

bug-submit: "|/var/www/bugz/email_in.pl -vvv 2>/tmp/emailin.log"


Note that for troubleshooting I can take a look at email_in.pl’s STDERR in /tmp/emailin.log;

Log in via the Bugzilla admin account and go to the Email section of the Parameters page.
Change the mailfrom to match the above e-mail alias so that Bugzilla users can add a comment to a bug by replying to Bugzilla’s e-mails.


I’m using SPF to verify sender e-mail addresses.
Here’s part of my /etc/postfix/main.cf containing some configuration for SPF :

alias_maps = hash:/etc/aliases

smtpd_recipient_restrictions =
permit_mynetworks
reject_unauth_destination
check_policy_service unix:private/policy-spf

policyd-spf_time_limit = 3600

and part of my /etc/postfix/master.cf :

policy-spf unix - n n - - spawn
user=nobody argv=/usr/bin/policyd-spf


Next I got DomainKeys working.
SPF and DomainKeys are especially important for this setup because
Bugzilla will not be doing and e-mail spam filtering.
All a spammer would have to do to submit annoying bugs into our Bugzilla
system would be to forge an e-mail from a Bugzilla user’s e-mail address
and send it to bug-submit@xxx.xxx… This is why I want SPF and DomainKeys fully
operational… that way many forgery attempts will be rejected.

The DKIM filters for inbound and outbound mail are started like this :

/usr/local/dkimproxy/bin/dkimproxy.in --listen=127.0.0.1:10025 --relay=127.0.0.1:10026 \

--user=dkim --group=dkim --daemonize --pidfile=/var/run/dkimproxy.in

/usr/local/dkimproxy/bin/dkimproxy.out --listen=127.0.0.1:10027 --relay=127.0.0.1:10028 \

--keyfile=/usr/local/dkimproxy/etc/private.key --selector=selector1 --domain=bugzilla.spinn3r.com \

--user=dkim --group=dkim --signature=dkim --daemonize --pidfile=/var/run/dkimproxy.out

For filtering inbound mail via DKIM edit the master.cf with something like this :

# Before-filter SMTP server. Receive mail from the network and
# pass it to the content filter on localhost port 10025.
#
smtp inet n - n - - smtpd
-o smtpd_proxy_filter=127.0.0.1:10025
-o smtpd_client_connection_count_limit=10
# DKIM
# After-filter SMTP server. Receive mail from the content filter on
# localhost port 10026.
127.0.0.1:10026 inet n - n - - smtpd
-o smtpd_authorized_xforward_hosts=127.0.0.0/8
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=
-o mynetworks=127.0.0.0/8
-o receive_override_options=no_unknown_recipient_checks


for outgoing DKIM edit the master.cf like this:


## outgoing dkim
submission inet n - n - - smtpd
-o smtpd_etrn_restrictions=reject
-o content_filter=dksign:[127.0.0.1]:10027
-o receive_override_options=no_address_mappings
-o smtpd_recipient_restrictions=permit_mynetworks,reject
dksign unix - - n - 10 smtp
-o smtp_send_xforward_command=yes
-o smtp_discard_ehlo_keywords=8bitmime
127.0.0.1:10028 inet n - n - 10 smtpd
-o content_filter=
-o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
-o smtpd_helo_restrictions=
-o smtpd_client_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o smtpd_authorized_xforward_hosts=127.0.0.0/8


Do a postfix reload

Now this here is a highly effect treatment for painful and debilitating traumatic bruises from bicycle accidents etc…. I’ve also used it to treat a bruised and swollen knuckle I got during a partner sword training exercise…

Kou Pi Plaster (kou pi gou) – aka dog skin patch is a leather patch folded in half with some medicine herbal gum on the inside. At room temperature the gum is hard. The patch is heated briefly in the oven to liquify the gum. This gum is very sticky so be careful to shave affected fur/hair before applying. Also be careful not to burn yourself by applying the patch when it is too hot.

In this case I cut a small section of the patch for my smallish minor bicycle accident bruise. I wrapped the area with some bandage tape to hold it to the wound. I don’t want the patch falling off when I ride my bicycle to and from work or practicing martial arts.

I keep two Kou Pi Plasters in my first aid kit at home. However I’ll say that my first treatment choice for severe traumas would be to see the Chinese doctor here in San Francisco, who specializes in tramatic/sport injuries. I haven’t tried his treatment; just heard about it from other martial arts practitioners.

I isolated Google’s IF_IDLE feature from SqlChanges; part of the Google V2 patch.
I diffed this IF_IDLE patch for mysql-5.0.68-percona-highperf but it’ll patch mysql-5.0.67 as well.

This simple and brilliant change allows us to
attempt to kill idle mysql processes while avoiding race conditions.

KILL IF_IDLE <id>

mysql> SHOW PROCESSLIST;
+----+-------------+-----------+-----------+---------+------+----------------------------------+
| Id | User        | Host      | db        | Command | Time | State                            | Info
+----+-------------+-----------+-----------+---------+------+----------------------------------+
|  2 | system user |           | NULL      | Connect | 2192 | Waiting for master to send event | NULL
| 13 | root        | localhost | NULL      | Query   |    0 | NULL                             | show processlist
| 30 | root        | localhost | NULL      | Sleep   |    2 |                                  | NULL
+----+-------------+-----------+-----------+---------+------+----------------------------------+
4 rows in set (0.00 sec)
mysql> KILL IF_IDLE 30;
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW PROCESSLIST;
+----+-------------+-----------+-----------+---------+------+----------------------------------+
| Id | User        | Host      | db        | Command | Time | State                            | Info
+----+-------------+-----------+-----------+---------+------+----------------------------------+
|  2 | system user |           | NULL      | Connect | 2357 | Waiting for master to send event | NULL
| 13 | root        | localhost | NULL      | Query   |    0 | NULL                             | show processlist
+----+-------------+-----------+-----------+---------+------+----------------------------------+
3 rows in set (0.00 sec)

We needed a Mysql Innodb Freeze mechanism for Spinn3r’s database operations, so we decided to isolate Google’s Innodb Freeze feature from Google’s V2 patch. The Innodb Freeze can be toggled on and off like so :

set global innodb_disallow_writes=ON
set global innodb_disallow_writes=OFF

Isolating this patch was easy since its a very small code change.
I simply patched Mysql with the Google V2 patch.
Then I started up Cscope and searched for the string: “innodb_disallow_writes”
From there I was able to trace the various C/C++ symbols; finding all the code for this feature…
I then edited an unpatched Mysql; manually pasting in the Innodb Freeze code changes.
Finally I diffed this manually edited Mysql against the unpatched Mysql producing
this Innodb Freeze patch for Mysql 5.0.37; it’s 323 lines small and only modifies 7 files of the Mysql source.

I have re-targeted the patch for Mysql 5.0.67 which will also patch Percona’s High Performance Mysql 5.0.68: Innodb Freeze patch for Mysql 5.0.67

I’ve briefly tested the patch and verified via md5sum that after setting global innodb_disallow_writes to OFF; data on disk does not change and all writes via the Innodb storage engine are blocked. However if you have set innodb_flush_log_at_trx_commit=0 in the my.cnf then updating an existing row will succeed. However the change will only exist in memory and will not be written to disk until you unfreeze (set global innodb_disallow_writes=OFF).

This is a small step towards Spinn3r’s upgrade to Mysql 5.0.x. I’ll probably end up extracting another feature from the Google V2 patch; and then eventually re-target these for Percona’s High Performance patch Mysql 5.0.68.

I needed the latest OpenSSH installed on a large cluster of Debian servers running Debian stable Etch. Since Debian isn’t going to ship a package with the latest OpenSSH for Debian stable Etch I decided to build this package myself.

This is my recipe to build a Debian package of the latest OpenSSH which installs in prefix=/usr/local. I decided not to repackage Debian’s patched OpenSSH because I didn’t want to deal with all the dependencies. The Debian OpenSSH patch adds a lot of code that I don’t need for my production environment. Also my package is named differently than the Debian one… so it can be installed without removing the old Debian openssh package. For my situation I don’t need a post install script as part of the Debian package… because I’m performing a custom cluster-wide software deploy….


Find a mirror and then download the OpenSSH sourcecode :

http://www.openssh.org/portable.html
wget ftp://ftp5.usa.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-5.1p1.tar.gz

Cryptographically verify the OpenSSH tarball :

wget ftp://ftp3.usa.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-5.1p1.tar.gz.asc
wget ftp://ftp3.usa.openbsd.org/pub/OpenBSD/OpenSSH/portable/DJM-GPG-KEY.asc
gpg --import DJM-GPG-KEY.asc
gpg --fingerprint
gpg --verify openssh-5.1p1.tar.gz.asc  openssh-5.1p1.tar.gz
gpg --delete-keys "Damien Miller (Personal Key) "

Prepare the build :

tar xzf openssh-5.1p1.tar.gz
mv openssh-5.1p1 openssh-5.1p1-spinn3r-r1
cd openssh-5.1p1-spinn3r-r1
dh_make --email david@spinn3r.com --single --native --packagename openssh-5.1p1-spinn3r-r1
cd debian/

Edit the ‘control’ file :

vi control:

Source: openssh-5.1p1-spinn3r-r1
Section: unknown
Priority: extra
Maintainer: root
Build-Depends: debhelper (>= 5), autotools-dev
Standards-Version: 3.7.2

Package: openssh-5.1p1-spinn3r-r1
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: David's attempt at a custom Spinn3r OpenSSH deb package

Edit the ‘rules’ file (e.g. vi rules) :
edited this section to make the install prefix set to /usr/local/ :

config.status: configure
        dh_testdir
        # Add here commands to configure the package.
        ./configure --host=$(DEB_HOST_GNU_TYPE) --build=$(DEB_BUILD_GNU_TYPE) \
--prefix=/usr/local --mandir=\$${prefix}/share/man --infodir=\$${prefix}/share/info \
CFLAGS="$(CFLAGS)" LDFLAGS="-Wl,-z,defs"

later on in this file there seems to be this section that seemed like a good idea to edit with the correct prefix :

        $(MAKE) prefix=$(CURDIR)/debian/openssh-5.1p1-spinn3r-r1/usr/local install

Change to parent directory and edit Makefile.in
cd ..

Edit the Makefile.in (so autoconf generates the Makefile that way we want it). We need to edit the ‘install’ target to NOT generate ssh keys otherwise build process will create a set of keys and the .deb package will be distributed with those keys.
change this:

install: $(CONFIGFILES) ssh_prng_cmds.out $(MANPAGES) $(TARGETS) install-files install-sysconf host-key check-config

to this :

install: $(CONFIGFILES) ssh_prng_cmds.out $(MANPAGES) $(TARGETS) install-files install-sysconf check-config

Build the custom debian package:
dpkg-buildpackage -rfakeroot -uc -b

       -us, -uc
              Do not sign the source package or the .changes file, respectively.

         -b indicates that no source files  are  to  be
              built  and/or  distributed

the above command outputs out a bunch of stuff and then ends with this :

dh_md5sums
dh_builddeb
dpkg-deb: building package `openssh-5.1p1-spinn3r-r1' in `../openssh-5.1p1-spinn3r-r1_5.1p1-spinn3r-r1_amd64.deb'.
 dpkg-genchanges -b
dpkg-genchanges: binary-only upload - not including any source code
dpkg-buildpackage: binary only upload (no source included)
root@fu:~/builds/openssh-5.1p1-spinn3r-r1#

The .deb file should have been written to the parent directory.

Enjoy…

I maintain a cluster of MySQL servers. I mentioned below that Linux has some memory management issues which cause our MySQL servers to swap when they really should not. So my idea was to upgrade the kernel to the Debian Etch Testing 2.6.25 kernel… but patched with Rik van Riel’s Split LRU patch (check it out: http://www.surriel.com/node/6).
Also… I don’t feel any need to compile a super efficient static kernel with only the modules we use. I’m OK with the Debian default initrd module loading kernel.

So what I ended up doing was first installing Debian’s kernel :

apt-get install linux-image-2.6.25-2-amd64


Then I grab the Debian kernel source (Linux kernel patched by Debian) :

apt-get source linux-image-2.6.25-2-amd64


which creates these files in the currect directory :

linux-2.6-2.6.25
linux-2.6_2.6.25-7.diff.gz
linux-2.6_2.6.25-7.dsc
linux-2.6_2.6.25.orig.tar.gz

Patch the kernel with the split-LRU patch :

cd linux-2.6-2.6.25
patch -p1 < ../linux-2.6.25-splitlru.patch


I move the changelog so that make-kpkg doesn’t complain and quit :

cd debian
mv changelog changelog.fu
cd ..


Here’s the secret sauce… I’m lazy and don’t want to do a “make menuconfig”
because I’m happy enough with the way Debian configures the kernel + initrd module loading.
So I copy the Debian config from the Kernel we just installed :

cp /boot/config-2.6.25-2-amd64 .config


Edit the Makefile and put unique id in EXTRAVERSION; this'll show up in a 'uname -a' :
e.g.

EXTRAVERSION = spinn3r.r1


I was building on a dual core machine so I set the CONCURRENCY_LEVEL
to tell the Debian make-kpkg to spawn instances of the compiler for building.
However if I was doing this often I'd use DistCC (http://code.google.com/p/distcc/).

export CONCURRENCY_LEVEL=2


Build the kernel, create debian package with a custom name :

fakeroot make-kpkg --initrd --us --uc --revision=spinn3r kernel_image

After some period of time that'll create a .deb file in the kernel directory's parent directory
which you can installed via a dpkg -i file.deb

I no longer have to use this workaround… since we patched our kernel.
The current 2.6 Linux Kernels seem to have some swap issues.
The Linux Kernel really likes to swap MySQL out to disk.

If for example you do a :


cat /proc/swaps


Often times on MySQL servers, with a data-set which can easily fit within memory,
swap is reported to be in use even though it should not.
Additionally sometimes too much swap space is reported used.

Here’s some related links on the subject :

• Re: Ability to limit or disable page caching?
• Using O_DIRECT on Linux and INNODB to Fix Swap Insanity
  
• MySQL and the Linux swap problem
• Should you have your swap file enabled while running MySQL ?

Here’s my work-around for this situation :

I create two swap files :


dd if=/dev/zero of=/swap01 bs=1MB count=34000
dd if=/dev/zero of=/swap02 bs=1MB count=34000
mkswap /swap01
mkswap /swap02


Now I can run my Perl script to rotate the active swap filesystem between /swap01 and /swap02.
If /swap01 is active, then Perl script does this :


swapon /swap02
swapoff /swap01


This causes the pages written to swap to be reloaded into memory and ensures I’m not using swap. MySQL shouldn’t get swapped in the first place but I feel this is a pretty good workaround. I run this little Perl script from cron every half hour. Notice the locking… :

#!/usr/bin/perl
use strict;
use warnings;

use LockFile::Simple qw(lock trylock unlock);

# set to 1 to turn verbosity off
my $verbose = 0;
my $lock = '/var/lock/rotate_swap.lock';

Main();

sub rs_lock
{
  die "already locked\n" unless trylock($lock);
  $verbose == 1 || print "acquired lock\n";
}

sub rs_unlock
{
  unlock($lock);
  $verbose == 1 || print "released lock\n";
}

sub err
{
    my $msg = shift;
    print "$msg\n";
# remove lock
    rs_unlock();
    exit -1;
}

sub Main
{
    my %swap;
    $swap{'/swap01'} = '/swap02';
    $swap{'/swap02'} = '/swap01';

# verify that valid swap files exist

    my $freecmd_output = `free`;
    my $totalmem;
    if($freecmd_output =~ m/Mem:\s+([^\s]+)/)
    {
	$totalmem = $1;
    }
    else
    {
	err("'free' cannot determine available memory");
    }

    my @stat_field;
    my $swap_size;
    foreach(keys %swap)
    {
	@stat_field = stat($_);
	$swap_size = $stat_field[7];
	$swap_size = $swap_size / 1024;

	if($swap_size > $totalmem)
	{
	    if($verbose == 0)
	    {
		print "swap file: $_ size: $swap_size is greater than free mem size: $totalmem\n";
	    }
	}
	else
	{
	    err("swap file: $_ size: $swap_size is not greater than free mem: $totalmem");
	}
    }

    eval
    {
# grab a mutex for this swap /var/lock/rotate_swap.lock
	rs_lock();

# make sure at least ONE swap partition is up and running...

	my $status = `cat /proc/swaps`;
	unless($status =~ /Priority\n.+/)
	{
	    err("No swap units available!");
	}

# determine the TARGET swap partition.
	my $target_swap;
	my @line;
	@line = split(/\n/,$status);

	my @field = split(/\s+/,$line[1]);
	my $current_swap = $field[0];

	if(!defined($swap{$current_swap}))
	{
	    $target_swap = '/swap01';
	}
	else
	{
	    $target_swap = $swap{$current_swap};
	}

	$verbose == 1 || print "currently swap $current_swap\n";

# attempt to mount it

	unless(system("swapon $target_swap") == 0)
	{
	    err("swapon failed for : $target_swap");
	}

	$verbose == 1 || print "enabled swap $target_swap\n";

# attempt to umount the stable swap partition

	unless(system("swapoff $current_swap") == 0)
	{
	    err("swapoff failed for : $current_swap");
	}

	$verbose == 1 || print "disabled swap $current_swap\n";

    }; # end eval {...

    # unlock
    rs_unlock();

    if ($@)
    {
        ### catch block
        die "caught unexpected error: $!\n";
    }
}

__END__