Enabling Floating Table Headers in Django Admin

To anyone who’s used Django’s amazing admin interface to browse large record lists, you’ve probably felt the frustration of having to scroll up and down the list to remind yourself of a column’s header name. Fortunately, there’s a trivial change you can make to Django’s base admin template to make your life easier. Enter the jQuery floating header plugin. As the name suggests, it auto-magically makes the header of tables float when you scroll down the table, so that the header is always in view.

To incorporate this library into Django’s admin, follow these steps:

1. Add jquery.floatheader.js to your media. I put mine in <workspace>/media/js/lib. I’ve arranged my URL patterns so that admin and non-admin media start from the same root URL. This allows me to easily use a relative “..” path to resolve to one or the other without knowing the root.

2. In your default templates directory, ensure there’s an admin subdirectory, and in there create a file called “base_site.html” with the content:

{% extends "admin/base.html" %}
{% load i18n admin_modify adminmedia %}
{% block extrahead %}{{ block.super }}
    <script language="javascript" src="{% admin_media_prefix %}../js/lib/jquery.js"></script>
    <script language="javascript" src="{% admin_media_prefix %}../js/lib/jquery.floatheader.js"></script>
    <script type="text/javascript">
        (function($){
            $(document).ready(function($){
                $('<insert your table pattern here>').floatHeader();
            });
        })(jQuery);
    </script>
{% endblock %}

3. Replace “<insert your table pattern here>” with a CSS selector of the table you’d like to float. I find Firebug useful for inspecting the CSS classes to identify the ones I want. For example, “‘.tabular fieldset table” is what you’d use to float the header on tabular inlines. You can include additional lines in ready() for each pattern.

To summarize, this modification defines an extra Javascript include in every admin page in order to load the jquery.floatheader.js plugin and then applies it to every table matching the pattern you specify.

Note, I’m also explicitly including jQuery even though it’s already available in admin via the variable django.jQuery. Unfortunately, I couldn’t find an easy way to access it in this template, as it seems to be defined later in the parent template, so attempts to access it here result in an “undefined variable” error. Django’s version of jQuery tends to lag behind the most recent version, so I tend not to rely on it regardless.

Otherwise, that’s all there is to it. I found this so surprisingly useful and easy to add to my current projects, I’m surprised this functionality isn’t included by default in admin.

Online Variance Calculation in Python

Since I couldn’t easily find this code anywhere, I figured I’d post it here for quick reference:


"""
2012.1.25 CKS
Incremental calculation of both the mean and variance.
http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
"""
import unittest
 
## Dumb slow mean/variance formulas.
 
def mean(seq):
    return sum(seq)/float(len(seq))
 
def variance(seq):
    m = mean(seq)
    return sum((v-m)**2 for v in seq)/float(len(seq))
 
## Incremental mean/variance formulas.
 
class Stat(object):
 
    def __init__(self):
        self.mean_sum = 0
        self.mean_count = 0
        self.last_variance = 0
 
    def __iadd__(self, value):
        last_mean = self.mean
        self.mean_sum += value
        self.mean_count += 1
        if last_mean is not None:
            self.last_variance = self.last_variance + (value  - last_mean)*(value - self.mean)
        return self
 
    @property
    def mean(self):
        if self.mean_count:
            return self.mean_sum/float(self.mean_count)
 
    @property
    def variance(self):
        return self.last_variance/float(self.mean_count)
 
class Test(unittest.TestCase):
 
    def test(self):
        nums = range(1,7)
        s = Stat()
        for n in nums:
            s += n
            print 'mean:',s.mean
            print 'variance:',s.variance
        self.assertAlmostEqual(s.mean, mean(nums), 1)
        self.assertAlmostEqual(s.variance, variance(nums), 2)
 
if __name__ == '__main__':
    unittest.main()

Persisting Django’s Test Database

After running Django unittests, it may sometimes be useful to manually inspect the test database after unittests are complete. I was working on a complicated network model, trying to resolve an elusive bug in a unittest, and in this specific instance, I thought running a manually written SQL query would be a bit more helpful than dealing with Django’s ORM. Since most people don’t really care about the test database, Django’s unittest framework is understandably hard-coded to destroy it after the tests have ran. Although there’s no simply flag to disable this, after digging around in the code, I found it’s relatively easy enough to change with a couple class method overrides.

First, for convenience, I define a PERSIST_TEST_DATABASE = 1 flag in my settings.py.

Then, my tests.py takes the general form:

from django.conf import settings
from django.test import TestCase
 
if settings.PERSIST_TEST_DATABASE:
    # Disable destruction of the test database.
    from django.db import transaction, connection, connections, DEFAULT_DB_ALIAS
    from django.test.testcases import restore_transaction_methods, connections_support_transactions
    connection.creation.destroy_test_db = lambda *args, **kwargs: None
 
class Test(TestCase):
 
    def _fixture_teardown(self):
        if not settings.PERSIST_TEST_DATABASE or not connections_support_transactions():
            return super(TestCase, self)._fixture_teardown()
 
        # If the test case has a multi_db=True flag, teardown all databases.
        # Otherwise, just teardown default.
        if getattr(self, 'multi_db', False):
            databases = connections
        else:
            databases = [DEFAULT_DB_ALIAS]
 
        restore_transaction_methods()
        for db in databases:
            transaction.commit(using=db)
            transaction.leave_transaction_management(using=db)

My two overrides are connection.creation.destroy_test_db, which I completely disable, and TestCase._fixture_teardown, which is essentially the same as the original, but tweaked to commit instead of rollback the transaction.

Passwd: Authentication token manipulation error

I was doing some work with an inherited Kerberos setup, when I noticed my Kerberos-based SVN login suddenly stopped working. Attempting to ssh into the machine revealed the cause:

WARNING: Your password has expired.
You must change your password now and login again!

Fair enough. I personally find password expiration a huge annoyance that provides no real security, but sure, why not. So I proceed with the default prompts, only to encounter:

Changing password for user myuser.
Kerberos 5 Password:
New password:
Retype new password:
passwd: Authentication token manipulation error

Fortunately (or unfortunately), this is a common if unhelpfully ambiguous error. Several people have posted theories for the underlying cause as well as fixes. Of course, none of the solutions worked for me. Having done a little maintenance work on the system before, I had root access to the machine, so one solution I found was to login as root and run `kpasswd myuser`. Apparently, even though it’s Kerberos detecting that my password has expired, it was using the default Linux `passwd` tool, and not the Kerberos-equivalent `kpasswd`.

Restricting Write Access to Django Admin

Django has a great auto-generated administration interface. It’s saved me countless hours of development time and makes basic data maintenance a breeze. Although it’s designed to be extensible, it’s not without it’s quirks and hurtles. Fortunately, most of these can be overcome with a little code inspection. One of these quirks is the superficial inability to make certain models read-only for specific user groups.

To understand why this may not make sense for certain applications, we have to look at Django’s two basic permissions, “staff” permission, which allows a user to login to admin, but not necessarily do anything else, and “superuser” permission, which gives the user access to all Django models.

For one application I worked on, staff users needed to be able to login to admin and read and possibly edit User records. Since the User model stores each user’s permissions, the main problem we immediately run into is that giving a user permission to edit the User model gives them blanket permission to edit anyone’s permissions. What this means is that a staff user can give themselves superuser permission, remove another user’s superuser permission, assign an incorrect group permission, and generally wreak havoc. Now, in keeping with The Zen of Admin, if you’ve given a user staff permission, then they can probably be trusted not to screw around with these permissions. However, this is essentially an honor system, which may not always give you peace of mind. It’s still a great temptation and potential opening for accidental misuse.

Fortunately, we can fix this problem by disallowing staff users from editing user permissions with a relatively simple change to Django’s UserAdmin. Specifically, we can define a custom UserAdmin, inheriting from the default UserAdmin, by defining the following in <your_app>/admin.py:


from django.contrib import admin
from django.contrib.admin.util import flatten_fieldsets
from django.contrib.auth.admin import UserAdmin as _UserAdmin
from django.core.exceptions import PermissionDenied
from django.shortcuts import get_object_or_404
 
class UserAdmin(_UserAdmin):
    _readonly_fields = [] # Default fields that are readonly for everyone.
 
    def get_readonly_fields(self, request, obj):
        readonly = list(self._readonly_fields)
        if request.user.is_staff and not request.user.is_superuser:
            if obj.is_superuser:
                # Prevent a staff user from editing anything of a superuser.
                readonly.extend(flatten_fieldsets(self.declared_fieldsets))
            else:
                # Prevent a non-superuser from editing sensitive security-related fields.
                readonly.extend(['is_staff', 'is_superuser', 'user_permissions', 'groups'])
        return readonly
 
    def user_change_password(self, request, id):
        # Disallow a non-superuser from changing the password of a superuser.
        user = get_object_or_404(self.model, pk=id)
        if not request.user.is_superuser and user.is_superuser:
            raise PermissionDenied
        return super(UserAdmin, self).user_change_password(request, id)
 
admin.site.unregister(User)
admin.site.register(User, UserAdmin)

This prevents staff users from editing any user permissions, but still allows them to edit other user data, and prevents them from editing anything for a superuser.

How to Extract a Webpage’s Main Article Content: The Unicode Edition

When I originally wrote html2text.py, my focus was only on extracting English text from webpages, so I didn’t give much thought to handling Unicode. Ignoring anything but ASCII would suffice. However, I was recently commissioned to extend the script’s functionality to use Unicode, so it could extract text in nearly any language found on the Web. The modifications turned out to be relatively straight-forward, made easier with the use of the handy chardet package to detect encoding. I also cleaned up the command line interface, to make the various parameters more accessible.

You can download the code here.

Update 2011-12-17: To avoid naming conflicts with similar scripts, I’ve renamed the script to webarticle2text and published it to github.com.

How To Install MBROLA Voices For Use With Festival On Ubuntu

Festival is a great text-to-speech engine. MBROLA also contains some impressive voice files that have notably higher quality than Festival’s default voices. Both of these systems are packaged in Ubuntu, and even though Festival can use MBROLA’s voice files, as of Ubuntu 10.04, there doesn’t seem to be a package to easily allows you to do this.

There’s an excellent guide on how to accomplish this manually. However, since it was written, a few more packages are now included in Ubuntu, allowing several steps to be skipped. Since most of the MBROLA voices are available as packages, the only steps needed to use them from Festival are to download a Festival-to-MBROLA wrapper and move the MBROLA voice files into Festival’s directory.

The following script installs MBROLA’s male British English voice for use with Festival. All MBROLA voices are represented by a “two-letter number” combination (e.g. en1). To install a different voice, change the VOICE variable to suit.

#!/bin/bash
set -e
 
VOICE=en1
VLANG=english
 
# Install all required packages.
sudo apt-get install festival mbrola mbrola-${VOICE} esound-clients festlex-oald
 
# Download wrappers for mbrola voices.
cd ~/Desktop
wget http://www.cstr.ed.ac.uk/downloads/festival/1.95/festvox_${VOICE}.tar.gz
tar xvf festvox_${VOICE}.tar.gz
rm -f festvox_${VOICE}.tar.gz
 
# Copy MBROLA voices into Festival's voice directory.
sudo mkdir -p /usr/share/festival/voices/${VLANG}/${VOICE}_mbrola/${VOICE}
sudo mv festival/lib/voices/${VLANG}/${VOICE}_mbrola/* /usr/share/festival/voices/${VLANG}/${VOICE}_mbrola/
sudo cp /usr/share/mbrola/voices/${VOICE} /usr/share/festival/voices/${VLANG}/${VOICE}_mbrola/${VOICE}
 
# Cleanup.
rm -rf ~/Desktop/festival
 
# Test.
echo '(voice_'${VOICE}'_mbrola)(SayText "Installation complete!")' | esddsp festival --pipe

Ubuntu Flash Drive Encryption Script

I was researching methods to encrypt USB flash drives, to protect personal data in the event they’re lost or stolen, and I found a few good write-ups on the subject using tools available in Ubuntu. Although the current distribution of Gnome includes pretty good support for auto-detecting and prompting for the password for encrypted drives, I couldn’t find any completely baked tools for encrypting the drives. So I wrote a simple script to streamline the process.

#!/bin/bash
set -e
 
echo "Drive Encrypter"
 
# Determine the device name.
mount # Display all mounted devices to help the user find the device name.
read -p "What is device name (e.g. /dev/<devicename>, from the mount list above)? " -e DEVICE
 
# Verify all required packages are installed.
echo "Checking packages..."
sudo apt-get install cryptsetup
sudo modprobe dm-crypt
 
# Overwrite with random data in order to slow down an attack on the encryption.
echo "Erasing device..."
umount /dev/$DEVICE
sudo badblocks -c 10240 -s -w -t random -v /dev/$DEVICE
 
echo "Creating encrypted partition..."
sudo cryptsetup --verbose --verify-passphrase luksFormat /dev/$DEVICE
sudo cryptsetup luksOpen /dev/$DEVICE $DEVICE
sudo mkfs.ext3 /dev/mapper/$DEVICE
sudo cryptsetup luksClose $DEVICE
 
echo "Done."
echo "Remove the drive and then plug it back in, and you should be prompted for a password to access the device."
#Note, you may have to run this to access the device from a normal account:
#sudo chown -R user:user /media/mountpoint

How To Remotely Backup A Web Server With No Install

I was researching simple methods for backing up a basic Linux web server, and I found an excellent script that handles files and databases, and also dumps everything to a remote server. I was looking for something a bit less complicated and overkill than Bacula, but more comprehensive than simply archiving the filesystem, and this fills the niche nicely.

Taking this solution one step further, I simplified the install requirements a bit by rewriting the script to run the backup remotely by logging in via SSH and pulling the data directly from the source machine. Instead of having to install the script and cron job on each machine you want to backup, you just install it once on the destination server, and ensure it has SSH access to each target. I also modified it to record a list of all installed packages, which would be useful if the system had to be reconstructed. Intuitively, it seems easier to deal with a single file, so I also wrap all backup parts into a single archive.

To start, you’ll want to ensure the script’s user has SSH access to the target machine by locally running something like:

ssh-copy-id tunneluser@12.345.678.90

You’ll also want to ensure it has permission to access all the data it needs to backup. On the target machine, you’d run something like:

sudo /usr/sbin/useradd tunneluser
sudo /usr/sbin/usermod -a -G apache tunneluser
sudo passwd tunneluser
sudo mysql -e "GRANT USAGEG ON *.* TO tunneluser@localhost IDENTIFIED BY 'mysqlpassword';GRANT ALL PRIVILEGES ON *.* TO tunneluser@localhost;"

Finally, the script itself:

#!/bin/bash
# Simple script for remotely backing up all data on a webserver.
# Adapted from http://www.cyberciti.biz/faq/how-to-back-up-a-web-server/
 
# Define remote login.
SSHUSER="tunneluser"
SSHHOST="12.345.678.90"
 
# Define remote and local backup destinations.
LOCAL_BACKUP_ROOT="/var/backups"
NOW=$(date +"%F")
BACKUP_ROOT="/tmp/backup/$NOW"
BACKUP_FILENAME="backup-$SSHHOST-$NOW.tar.gz"
BACKUP_PATH="$BACKUP_ROOT/$BACKUP_FILENAME"
 
# Run a script on the remote host.
ssh -T $SSHUSER@$SSHHOST <<EOI
 
TERM=dumb
export TERM
 
# File directories to backup.
DIRS="/var/www/html/ /etc"
 
# Paths for binary files
TAR="/bin/tar"
PGDUMP="/usr/bin/pg_dump"
MYSQLDUMP="/usr/bin/mysqldump"
GZIP="/bin/gzip"
LOGGER="/usr/bin/logger"
 
# Set Pgsql username
PGSQLUSER="tunneluser"
 
# Set MySQL username and password
MYSQLUSER="tunneluser"
MYSQLPASSWORD="mysqlpassword"
 
# Backup file name hostname.time.tar.gz
BFILE="web.tar.gz"
PFILE="pg.sql.gz"
MFILE="mysql.sq.gz"
PKGFILE="installed-software.log"
 
# Store todays date.
NOW=\$(date +"%F")
echo \$NOW
 
# Store backup path.
BACKUP_ROOT="$BACKUP_ROOT"
[ ! -d \$BACKUP_ROOT ] && mkdir -p \${BACKUP_ROOT}
BACKUP_FILENAME="$BACKUP_FILENAME"
 
# Log backup start time in /var/log/messages
\$LOGGER "*** Backup started @ \$(date) ***"
 
# Backup package list.
echo "Backing up package list..."
rpm -qa > \${BACKUP}/\${PKGFILE} # Redhat
#dpkg --get-selections > \${BACKUP}/\${PKGFILE} # Debian
 
# Backup webserver directories.
echo "Backing up directories..."
\$TAR -zcf \${BACKUP_ROOT}/\${BFILE} \${DIRS}
 
# Backup PgSQL.
echo "Backing up PgSQL..."
\$PGDUMP -x -D -U\${PGSQLUSER} | \$GZIP -c > \${BACKUP_ROOT}/\${PFILE}
 
# Backup MySQL.
echo "Backing up MySQL..."
\$MYSQLDUMP -u \${MYSQLUSER} -h localhost -p\${MYSQLPASSWORD} --all-databases --lock-all-tables | \$GZIP -9 > \${BACKUP_ROOT}/\${MFILE}
 
# Create final file and cleanup.
echo "Cleaning up..."
\$TAR -zcf \${BACKUP_FILENAME} \${BACKUP_ROOT}/\${BFILE} \${BACKUP_ROOT}/\${MFILE} \${BACKUP_ROOT}/\${PKGFILE} \${BACKUP_ROOT}/\${PFILE}
rm -Rf \${BACKUP_ROOT}
 
# Log backup end time in /var/log/messages
\$LOGGER "*** Backup Ended @ \$(date) ***"
 
exit
EOI
 
echo "Downloading archive..."
[ ! -d $LOCAL_BACKUP_ROOT ] && mkdir -p ${LOCAL_BACKUP_ROOT}
scp -rp $SSHUSER@$SSHHOST:$BACKUP_PATH ${LOCAL_BACKUP_ROOT}/${BACKUP_FILENAME}
 
echo "Backup complete."

Implementing a Simple Machine Learning Intelligence to Play Rock-Paper-Scissors in Javascript

Rock-paper-scissors is not a complicated game. However, there are some that not only enjoy the game, but take pride in professing skill at it. But is it possible to have skill at rock-paper-scissors?

In the real world, where a human plays against a human, “cheating” is relatively easy. A player can gain an advantage by subtly delaying their release just long enough to read their opponent’s body language and choose a winning gesture. For sake of argument, let’s ignore this and assume the players play fairly, where each is completely unaware of their opponent’s move.

In this case, optimal play between two players making truly random gestures ends in a tie (on average). The only way to repeatedly win is to somehow predict your opponent’s gesture, which is impossible if your opponent is truly random. Only if your opponent has a bias can statistics be used to potentially model and predict their gesture.

To demonstrate this, I implemented three simple prediction routines.

To play a round against each routine, just click a link representing the gesture you’d like to make. The routine will then calculate its own gesture (not looking at your current choice of course), and then display the result.

1. Conditional-0
All the opponent’s past gestures are counted, and these counts are used as weights when pseudo-randomly selecting a gesture. For example, if you’ve picked rock once, paper four times, and scissors five times, then there will be a 10% chance that the routine will choose paper, 40% for scissors, and 50% for rock.

2. Conditional-1
This method tracks the last two gestures made by the opponent, and counts the number of each transition. These counts are again used to weight the random selection of a gesture. This allows it to see simple sequential patterns where an opponent’s next gesture is dependent on their last gesture.

3. Conditional-2
Similar to the previous method, this one tracks a combination of the opponent’s last gesture, and its own next-to-last gesture. This allows it to see simple dependence between the opponent’s gesture and its own.

It’s a strange game in that the player with the most entropy wins in the long run. To see the code click here.