Linux Swap Issue

I no longer have to use this workaround… since we patched our kernel.
The current 2.6 Linux Kernels seem to have some swap issues.
The Linux Kernel really likes to swap MySQL out to disk.

If for example you do a :


cat /proc/swaps


Often times on MySQL servers, with a data-set which can easily fit within memory,
swap is reported to be in use even though it should not.
Additionally sometimes too much swap space is reported used.

Here’s some related links on the subject :

• Re: Ability to limit or disable page caching?
• Using O_DIRECT on Linux and INNODB to Fix Swap Insanity
  
• MySQL and the Linux swap problem
• Should you have your swap file enabled while running MySQL ?

Here’s my work-around for this situation :

I create two swap files :


dd if=/dev/zero of=/swap01 bs=1MB count=34000
dd if=/dev/zero of=/swap02 bs=1MB count=34000
mkswap /swap01
mkswap /swap02


Now I can run my Perl script to rotate the active swap filesystem between /swap01 and /swap02.
If /swap01 is active, then Perl script does this :


swapon /swap02
swapoff /swap01


This causes the pages written to swap to be reloaded into memory and ensures I’m not using swap. MySQL shouldn’t get swapped in the first place but I feel this is a pretty good workaround. I run this little Perl script from cron every half hour. Notice the locking… :

#!/usr/bin/perl
use strict;
use warnings;

use LockFile::Simple qw(lock trylock unlock);

# set to 1 to turn verbosity off
my $verbose = 0;
my $lock = '/var/lock/rotate_swap.lock';

Main();

sub rs_lock
{
  die "already locked\n" unless trylock($lock);
  $verbose == 1 || print "acquired lock\n";
}

sub rs_unlock
{
  unlock($lock);
  $verbose == 1 || print "released lock\n";
}

sub err
{
    my $msg = shift;
    print "$msg\n";
# remove lock
    rs_unlock();
    exit -1;
}

sub Main
{
    my %swap;
    $swap{'/swap01'} = '/swap02';
    $swap{'/swap02'} = '/swap01';

# verify that valid swap files exist

    my $freecmd_output = `free`;
    my $totalmem;
    if($freecmd_output =~ m/Mem:\s+([^\s]+)/)
    {
	$totalmem = $1;
    }
    else
    {
	err("'free' cannot determine available memory");
    }

    my @stat_field;
    my $swap_size;
    foreach(keys %swap)
    {
	@stat_field = stat($_);
	$swap_size = $stat_field[7];
	$swap_size = $swap_size / 1024;

	if($swap_size > $totalmem)
	{
	    if($verbose == 0)
	    {
		print "swap file: $_ size: $swap_size is greater than free mem size: $totalmem\n";
	    }
	}
	else
	{
	    err("swap file: $_ size: $swap_size is not greater than free mem: $totalmem");
	}
    }

    eval
    {
# grab a mutex for this swap /var/lock/rotate_swap.lock
	rs_lock();

# make sure at least ONE swap partition is up and running...

	my $status = `cat /proc/swaps`;
	unless($status =~ /Priority\n.+/)
	{
	    err("No swap units available!");
	}

# determine the TARGET swap partition.
	my $target_swap;
	my @line;
	@line = split(/\n/,$status);

	my @field = split(/\s+/,$line[1]);
	my $current_swap = $field[0];

	if(!defined($swap{$current_swap}))
	{
	    $target_swap = '/swap01';
	}
	else
	{
	    $target_swap = $swap{$current_swap};
	}

	$verbose == 1 || print "currently swap $current_swap\n";

# attempt to mount it

	unless(system("swapon $target_swap") == 0)
	{
	    err("swapon failed for : $target_swap");
	}

	$verbose == 1 || print "enabled swap $target_swap\n";

# attempt to umount the stable swap partition

	unless(system("swapoff $current_swap") == 0)
	{
	    err("swapoff failed for : $current_swap");
	}

	$verbose == 1 || print "disabled swap $current_swap\n";

    }; # end eval {...

    # unlock
    rs_unlock();

    if ($@)
    {
        ### catch block
        die "caught unexpected error: $!\n";
    }
}

__END__