Tim Retout's www presence

SmoothWall Express - ntpd

Part one of an occasional series about SmoothWall Express (SWE).

The SmoothWall Express source tree contains two NTP daemons - both ntpd and OpenNTPD. SWE uses the openntpd daemon, but installs the ntpdate and ntpq utilities from the ntpd package.

This is justified because openntpd is apparently more lightweight than ntpd. But note that ntpq does not work with openntpd.

On my machine, the saving in memory is not massive. As observed in the thread, openntpd needs to run two processes (with one running as root), while ntpd can use Linux capabilities. From a cursory look at the Debian bug tracker, ntp has more bugs, but they don't seem as important as openntpd's. I suspect ntp gets more testing. The latest portable release of openntpd was in May 2006.

If I ruled the world, I'd choose ntpd over openntpd.

Posted: 29 Oct 2009 00:38 | Tags: smoothwall, swe, ntpd

The story so far

I spent the weekend in Rugby, not doing very much. I had this morning off work to travel back to Southampton, and it was surprisingly busy.

This afternoon I put the transaction scope guards I'd researched on Saturday into my project at work, and then refactored bits of it. According to the project plan I'm meant to be finished soon, but I keep finding more important things to do than implement the final SQL functions.

For instance, at the end of last week I found and fixed a small but nasty bug that was causing corruption of the database. Now that's out of the way, my life becomes much easier, in theory.

Posted: 26 Oct 2009 21:18 | Tags: life

The Future Is Here

While depositing a cheque using the machine at my local Nationwide branch yesterday, a remarkable thing occurred; the machine looked at the cheque and told me how much it was for.

Posted: 25 Oct 2009 17:22 | Tags: future, ocr, nationwide

Transaction Scope Guards

I've been writing some Perl DBI code which involves some fairly involved error handling; I've been looking for a way to roll back transactions neatly when certain errors happen.

I very nearly reinvented the concept of a 'transaction scope guard' which I now find is implemented in DBIx::Class (with Scope::Guard implementing a more general version). A lexical variable can be used to detect in which cases a transaction should be ended, because the object it points to will get DESTROYed when it goes out of scope. Some rough code to illustrate the concept is below.

# $fallback controls whether we use nested transactions,
# which is slower but lets us commit all the other lines in the batch.
sub process_batch {
    my $fallback = shift;

    # This catches any exception handling that makes us leave the function.
    # The DESTROY method of $transaction contains $db->rollback()
    my $transaction = $db->txn_scope_guard();

    for (1..2000) {
        # File access error will get thrown to outside of function,
        # so the transaction will be rolled back.
        my $line = get_next_line();
        last unless $line;

        my $parsed;
        eval {$parsed = parse_line($line)};
        if ($@) {
            # Handle line parsing errors here without interfering with
            # DB error handling.
            warn "Could not parse $line\n";
            next;
        }

        # here's the actual db code
        eval {
            $db->savepoint() if $fallback;
            $db->process_one_line($parsed);
        }
        if ($@) {
            if ($fallback) {
                # Just roll back to savepoint.  Transaction continues.
                $db->rollback_to_savepoint();
            } else {
                # Propagate error outside of function, ending transaction.
                die $@;
            }
        }
    }

    $transaction->commit();
}

The advantage is that rollback code can be kept in one place, and not repeated all over the various error handling cases.

I had actually gone off this idea, because I couldn't see any documentation defining exactly when the DESTROY method gets called in Perl. But given it's got into DBIx::Class, it must be fine! I also prefer their API over what I was considering implementing; the transaction object that gets returned will handle only commit(), not any other database calls.

Posted: 25 Oct 2009 01:11 | Tags: perl, dbi

Postcodes

I just signed up to a petition about postcodes.

The context is that Royal Mail asked someone to stop giving not-for-profit websites access to postcode data. From a database copyright perspective, Royal Mail is within its rights to do this; but it seems wrong that socially useful non-commercial sites are unable to make use of postcodes without paying a high license fee.

Compare this to the US, where this data is freely available.

However, why stop with non-commercial sites? The economic boost to small businesses from a freely-available postcode database would likely exceed the revenue made via licensing.

Meanwhile, of course, OpenStreetMap will continue to improve its postcode coverage until the whole question becomes moot.

Posted: 06 Oct 2009 22:29 | Tags: postcodes, osm

Code Reuse

At work, I have been refactoring old Perl code. Part of me feels that this was tangential to the main aims of the project I've been assigned, but another part of me can list all the bugs I've found/fixed and the advantages in terms of maintainability, so on balance I think it was a good idea.

Something I like even more than tidying code is reducing the amount of code required. (I've been doing a lot of that as well.) Breaking code into reusable modules is the essence of what I'm trying to achieve - later projects within the company (and perhaps beyond) should not have to reinvent what I'm writing. I'm attempting to replace boilerplate code with existing modules from CPAN where I can - I'm fortunate in that this seems to be possible licensing-wise within our product.

Now, there is some code which loops over a list of plugins, and forks off a background process to deal with each of them. Which CPAN module should I use? There are a few which look promising (like the classic Proc::Daemon), but force the parent process to exit. Not good for a loop. There's one which does dodgy hacks with parsing the output of 'ps', rather than using 'kill 0 => $pid' like every other module. Some handle pidfiles (which I want, provided I can customize the name, but what if I didn't?), and some don't (which is fine, I'll just use one of the separate pidfile modules). But why do CPAN modules reimplement the pidfile handling themselves anyway?

One of the best approaches I've seen so far is MooseX::Daemonize. Moose's role system looks like it would be very useful, but I'm a bit hesitant to introduce Moose to the company just yet. Maybe I should.

My point is, there is all this code, but subtle problems are preventing its reuse. I see the same thing going on at work; there are some utility libraries, but they contain large numbers of mostly unrelated functions. Developers often do not spot code which might be useful later, so they are left as a subroutine in a script. Ironically, I am reimplementing these functions in a more modular fashion, and hopefully this will catch on.

Posted: 04 Oct 2009 23:27 | Tags: perl