Category Archives: Software

GStreamer’s playbin, threads and queueing

I’ve been working on a project that uses GStreamer to play back audio files in an automatically-determined order. My implementation uses a playbin, which is nice and easy to use. I had some issues getting it to continue playback on reaching the end of a file, though.

According to the documentation for the about-to-finish signal,

This signal is emitted when the current uri is about to finish. You can set the uri and suburi to make sure that playback continues.

This signal is emitted from the context of a GStreamer streaming thread.

Because I wanted to avoid blocking a streaming thread under the theory that doing so might interrupt playback (the logic in determining what to play next hits external resources so may take some time), my program simply forwarded that message out to be handled in the application’s main thread by posting a message to the pipeline’s bus.

Now, this approach appeared to work, except it didn’t start playing the next URI, and the pipeline never changed state- it was simply wedged. Turns out that you must assign to the uri property from the same thread, otherwise it doesn’t do anything.

Fortunately, it turns out that blocking that streaming thread while waiting for data isn’t an issue (determined by experiment by simply blocking the thread for a while before setting the uri.

Newlib’s git repository

Because I had quite the time finding it when I wanted to submit a patch to newlib, there’s a git mirror of the canonical CVS repository for newlib, which all the patches I saw on the mailing list were based off of. Maybe somebody else looking for it will find this note useful:

git clone git://sourceware.org/git/newlib.git

See also: the original mailing list announcement of the mirror’s availability.

“Four”ier transform

Today’s Saturday Morning Breakfast Cereal:
20130201
I liked the joke and am familiar enough with the math of working in unusual bases that I felt a need to implement a quick version of this in Python. Code follows.

#!/usr/bin/env python

def fourier(x, b):
    """Attempts to find a fourier version of x, working down from base b.

    Returns the fouriest base."""
    mostFours = 0
    bestBase = -1

    for base in range(b, 1, -1):
        fours = 0
        t = x
        while t != 0:
            if (t % base) == 4:
                fours += 1
            t //= base

        # Prefer lower bases
        if fours >= mostFours:
            print(baseconvert(x, base) + "_{0}".format(base))
            mostFours = fours
            bestBase = base

    return bestBase

BASE_CHARS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
def baseconvert(x, base):
    s = ""
    while x != 0:
        s += BASE_CHARS[x % base]
        x //= base
    return ''.join(reversed(s))

if __name__ == '__main__':
    from sys import argv, exit
    if len(argv) < 2:
        print("""Usage: {0} number

Computes the "four"ier transform of , printing the optimizations to
reach the "fouriest" form.""".format(argv[0]))
        exit(1)

    x = int(argv[1])
    # Base 36 is the largest sensible base to use
    base = fourier(x, 36)

    if base == -1:
        print("{0} is four-prime!".format(x))

This is Python 3.x code, using explicit integer division. It should work under the 2.x series if you change line 34 to use “/=” rather than “//=”. It can only go up to base 36, because I didn’t want to deal with bases that are hard to represent in reasonable ways. Up to base 64 is an option, but in that case I would have wanted to use MIME base 64, which puts digits at positions 52 through 61, which would be confusing to read. Thus it only supports up to base 36, but could be adjusted with relative east to do larger bases.

Running a few examples:

$ python fourier.py 624
HC_36
HT_35
IC_34
IU_33
JG_32
K4_31
143_23
1B4_20
440_12
4444_5

$ python fourier.py 65535
1EKF_36
1IHF_35
1MNH_34
1R5U_33
1VVV_32
2661_31
2COF_30
2JQO_29
2RGF_28
38O6_27
3IOF_26
44LA_25
B44F_18
14640_15
4044120_5

$ python fourier.py 3
3 is four-prime!

A few quirks: it prefers lower bases, so bases that match earlier attempts in fouriness will be printed, despite having equal fouriness. I’ve decided to call values that have no representations containing a ‘4’ character “four-prime”, which is probably going to be a rare occurrence, but the program handles it okay.

Generalization of the algorithm is certainly possible, and basically requires changing the condition on line 14 to match different digit values. For example, a hypothetical “Three”ier transform would replace the ‘4’ on line 14 with a ‘3’.

Further reading

There’s a rather interesting discussion of the topic over on Reddit, as well as a few other implementations. (Thanks to Merth for pointing those out to me.)

Of Cable Modems and the Dumb Solution

I was studying in Japan last semester (which explains somewhat why I haven’t posted anything interesting here in a while). That’s a whole different set of things to blog about, which I’ll get to at some point with any luck (maybe I’ll just force myself to write one post per day for a bit, even though these things tend to take at least a few hours to write..).

Background

At any rate, back in Houghton I live with a few roommates in an apartment served by Charter internet service (which I believe is currently DOCSIS2). The performance tends to be quite good (it seems that the numbers that they quote for service speeds are guaranteed minimums, unlike most other ISPs), but I like to have complete control over my firewall and routing.

In the past, such freedom has been achieved through my trusty WRT54GL, but the 4-megabyte Flash chip in that device makes it hard to fit in a configuration that includes IPv6 support, which is increasingly important to me. As I had an Intel Atom-based board sitting around some time ago, I decided to turn that into a full-time router/firewall running pfSense. The power available with pfSense is probably overkill for my needs, but it ensures I’ll be able to stay up to date and potentially do fancy things with my network configuration at some future date.

Returning to the matter at hand: the whole system was working just fine for a while, but I got a report from my roommates that the internet connection had stopped working, but came up okay with a bargain-basement consumer router (a Linksys/Cisco E900). From what information I was able to get from my roommates, it sounded like a hardware failure in the secondary network card, which is used for the WAN uplink (not exactly surprising, since it’s a 100-megabit PCI Ethernet card I pulled out of something else some time ago).

Debugging

On my recent return to the apartment, one of my priorities was getting the pfSense system up and running again as the main router/firewall. While the E900 was performing fine, pfSense allows me to get a few additional things out of the connection. Most notably, Charter provide a 6rd relay for ISP-provided IPv6 (compared to something like the public IPv6 tunnel service available from Hurricane Electric), which is quite desirable to me.

After performing a basic test, the pfSense box did indeed fail to get a public IP address from Charter when put in place as the primary gateway. At that point, I decided to break out a network analyzer (Wireshark in this case) and see how the DHCP solicitations on the WAN interface differed between the E900 and my pfSense configuration. What follows is Wireshark’s dissection of a single DHCP Discover message from each system.

Linksys E900

Ethernet II, Src: Micro-St_60:86:0c (8c:89:a5:60:86:0c), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
    Source: Micro-St_60:86:0c (8c:89:a5:60:86:0c)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255 (255.255.255.255)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x10 (DSCP 0x04: Unknown DSCP; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0001 00.. = Differentiated Services Codepoint: Unknown (0x04)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 328
    Identification: 0x0000 (0)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 128
    Protocol: UDP (17)
    Header checksum: 0x3996 [correct]
    Source: 0.0.0.0 (0.0.0.0)
    Destination: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: bootpc (68), Dst Port: bootps (67)
    Source port: bootpc (68)
    Destination port: bootps (67)
    Length: 308
    Checksum: 0x9918 [validation disabled]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x1a5f4329
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 0.0.0.0 (0.0.0.0)
    Client MAC address: Micro-St_60:86:0c (8c:89:a5:60:86:0c)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type
        Length: 1
        DHCP: Discover (1)
    Option: (12) Host Name
        Length: 10
        Host Name: Needlecast
    Option: (55) Parameter Request List
        Length: 4
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (6) Domain Name Server
    Option: (61) Client identifier
        Length: 7
        Hardware type: Ethernet
        Client MAC address: Micro-St_60:86:0c (8c:89:a5:60:86:0c)
    Option: (255) End
        Option End: 255
    Padding
pfSense 2.0.2

Ethernet II, Src: 3com_8a:b9:6b (00:50:da:8a:b9:6b), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
    Source: 3com_8a:b9:6b (00:50:da:8a:b9:6b)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255 (255.255.255.255)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x10 (DSCP 0x04: Unknown DSCP; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0001 00.. = Differentiated Services Codepoint: Unknown (0x04)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 328
    Identification: 0x0000 (0)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 16
    Protocol: UDP (17)
    Header checksum: 0xa996 [correct]
    Source: 0.0.0.0 (0.0.0.0)
    Destination: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: bootpc (68), Dst Port: bootps (67)
    Source port: bootpc (68)
    Destination port: bootps (67)
    Length: 308
    Checksum: 0x3a68 [validation disabled]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x06303c2b
    Seconds elapsed: 0
    Bootp flags: 0x0000 (Unicast)
        0... .... .... .... = Broadcast flag: Unicast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 0.0.0.0 (0.0.0.0)
    Client MAC address: 3com_8a:b9:6b (00:50:da:8a:b9:6b)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type
        Length: 1
        DHCP: Discover (1)
    Option: (61) Client identifier
        Length: 7
        Hardware type: Ethernet
        Client MAC address: 3com_8a:b9:6b (00:50:da:8a:b9:6b)
    Option: (12) Host Name
        Length: 7
        Host Name: pfSense
    Option: (55) Parameter Request List
        Length: 8
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (28) Broadcast Address
        Parameter Request List Item: (2) Time Offset
        Parameter Request List Item: (121) Classless Static Route
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (12) Host Name
    Option: (255) End
        Option End: 255
    Padding

(Apologies to anybody who finds the above ugly, but I only have so much patience for CSS while blogging.)

There are a few differences there, none of which seem really harmful. Given it was working without incident before, however, I guessed that maybe some upstream configuration had changed and become buggy. In particular, I thought that either the BOOTP broadcast flag (line 32 of both packet dissections) needed to be set for some reason, or the upstream DHCP server was choking on some of the parameters pfSense was requesting.

In an effort to pin down the problem, I manually made some DHCP requests with dhclient configured to match what I was seeing from the E900. The configuration I used with dhclient looked like this (where xl0 is the identifier BSD assigns to my WAN interface):

interface "xl0" {
	send host-name "Needlecast";
	request subnet-mask, routers, domain-name, domain-name-server;
	send dhcp-client-identifier 1:8c:89:a5:60:86:0c;
}

This yielded packets that, when examined in Wireshark, only differed by some of the hardware addresses and the BOOTP broadcast flag. At that point I was rather stuck. Newer releases of dhclient support an option to force the broadcast flag in requests, but FreeBSD (which pfSense is derived from) does not provide a new enough version to have that option, and I didn’t want to try building it myself. In addition, I know that my ISP doesn’t lock connections to MAC addresses, so I shouldn’t have to spoof the MAC address of the E900 (indeed, nothing needed to be changed when switching from pfSense to the E900, so the other direction shouldn’t need anything special).

Since I was stuck, it was time to start doing things that seemed increasingly unlikely. One comment on the pfSense forum related to a similar issue mentioned that cable modems tend to be simple DOCSIS-to-Ethernet bridges, so there’s some sort of binding to the client MAC address in the upstream DOCSIS machinery, which rebooting the modem should reset. So I hooked everything up normally, cycled power to the modem and booted up pfSense, and…

…it worked.

I had spent a few evenings working on the problem, and the fix was that simple. I was glad it was finally working so I could reconfigure internet-y goodness (QoS, DDNS updating, 6rd tunneling, VPN) on it, but there was certainly also frustration mixed in there.

Lessons

So what’s the lesson? I suppose we might say that “you’re never too knowledgeable to try rebooting it”. It’s common advice to less savvy users to “try rebooting it”, but I think that’s an oft-neglected solution when more technically-inclined individuals are working on a problem. On the other hand, maybe I’ve just learned some details about DOCSIS systems and the solution in this case happened to be rebooting.

<witty and relevant image goes here>

Treating configuration as code with Python’s import hooks

Rationale

I was reading up on web frameworks available when programming in Haskell earlier today, and I liked the use of domain-specific languages (DSLs) within frameworks such as the routing syntax in Yesod. Compared to how routes are specified in Django (as a similar example that I’m already familiar with), the DSL is both easier to read (because it doesn’t need to be valid code in the hosting language) and faster (since it ends up getting compiled into the application as properly executable code).

A pattern I find myself using rather often in Python projects is to have a small module (usually called config) that encapsulates an INI-style configuration file. It feels like an ugly solution though, since it generally just exports a ConfigParser instance. Combined with consideration of DSLs in Haskell, that got me thinking: what if there were an easier way that made INI configuration files act like Python source such that they could just be imported and have the contents of the file exposed as simple Python types (thus hiding some unnecessary complexity)?

Implementation

I was aware of Python’s import hook mechanisms, so I figured that it should be a good way to approach this problem, and it ended up being a good excuse to learn more about the import hook mechanism. Thus, the following code provides a way to expose INI-style configuration as Python modules. It should be compatible with Python 3 after changing the import of ConfigParser on line 1 to configparser, but I only tested it on Python 2.7.

import ConfigParser, imp, os, sys

class INILoader(object):
    def __init__(self, prefix):
        self.prefix = prefix

    def load_module(self, name):
        if name in sys.modules:
            return sys.modules[name]

        module = imp.new_module(name)
        if name == self.prefix:
            # 'from config import foo' gets config then config.foo,
            # so we need a dummy package.
            module.__package__ = name
            module.__path__ = []
            module.__file__ = __file__
        else:
            # Try to find a .ini file
            module.__package__, _, fname = name.rpartition('.')
            fname += '.ini'
            module.__file__ = fname
            if not os.path.isfile(fname):
                raise ImportError("Could not find a .ini file matching " + name)
            else:
                load_ini_module(fname, module)

        sys.modules[name] = module
        return module

    def find_module(self, name, path=None):
        if name.startswith(self.prefix):
            return self
        else:
            return None

def load_ini_module(f, m):
    """Load ini-style file ``f`` into module ``m``."""
    cp = ConfigParser.SafeConfigParser()
    cp.read(f)
    for section in cp.sections():
        setattr(m, section, dict(cp.items(section)))

def init(package='config'):
    """Install the ini import hook for the given virtual package name."""
    sys.meta_path.append(INILoader(package))

Most of this code should be fairly easy to follow. The magic of the import hook itself is all in the INILoader class, and exactly how that works is specified in PEP 302.

Usage

So how do you use this? Basically, you must simply run init(), then any imports from the specified package (config by default) will be resolved from an .ini file rather than an actual Python module. Sections in a file are exposed as dictionaries under the module.

An example is much more informative than the preceding short description, so here’s one. I put the code on my Python path as INIImport.py and created foo.ini with the following contents:

[cat]
sound=meow
[dog]
sound=woof
[cow]
sound=moo

It has three sections, each describing an animal. Now I load up a Python console and use it:

>>> import INIImport
>>> INIImport.init()
>>> from config import foo
>>> foo.cat
{'sound': 'meow'}
>>> foo.dog['sound']
'woof'

This has the same semantics as a normal Python module, so it can be reloaded or aliased just like any other module:

>>> import config.foo
>>> foo == config.foo
True
>>> reload(config.foo)
<module 'config.foo' from 'foo.ini'>

The ability to reload this module is particularly handy, because my normal configuration module approach doesn’t provide an easy way to reload the file.

Improvements, Limitations

Some addition improvements come to mind if I were to release this experiment as production-quality code. Notably, additional path manipulations for finding .ini files would be useful, such as taking a path argument to init(), supplying a set of directories to search within. Having a way to remove the import hook that it installs would also be good, and straightforward to implement. There’s no way to get all the sections in the module, so it would also be useful to export the sections somehow– perhaps by having the module support the mapping protocol (so all the sections could be retrieved with module.items(), for example).

The main limitation of this scheme is that it has no way to determine the desired type of loaded configuration values, so everything is a string. This is a typical limitation when using the ConfigParser module, but compared to a more robust configuration scheme such as defining objects in a Python file (such as Django does), this might be an annoying loss of expressiveness. The values can always be coerced to the required type when retrieving them, but that’s a bit of unnecessary extra code in whatever uses the configuration.

It may also be useful to provide a way to write configuration back to a file when modifying a config module, but my simplistic implementation makes no attempt at such. Doing so would not be terribly difficult, just involving some wrapper objects to handle member assignment for sections and items, then providing a mechanism for saving the current values back to the original file.

Postlude

This made for an interesting experiment, and it should be a handy example for how to implement import hooks in Python. You may use this code freely within your own work, but I’d appreciate if you leave a note here that it was useful, and link back to this post.

Chainloading Truecrypt

I recently purchased a new laptop computer (a Lenovo Thinkpad T520), and wanted to configure it to dual-boot between Windows and Linux.  Since this machine is to be used “on the go”, I also wanted to have full encryption of any operating systems on the device. My choices of tools for this are Truecrypt on the Windows side, and dm_crypt with LUKS on Linux. Mainly due to rather troublesome design on the Windows side of this setup, it was not as easy as I might have hoped. I did eventually get it working, however.

Admonishment

Truecrypt was “Discontinued” in 2014, but still works okay. VeraCrypt is substantially a drop-in replacement if you’re looking for a piece of software that is still actively maintained. As of this update (early 2017) the only non-commercial option for an encrypted Windows system booted from UEFI is Windows’ native BitLocker (with which dual-booting is possible but it won’t be possible to read the encrypted Windows partition from Linux), but if you’re booting via legacy BIOS these instructions should still work for TrueCrypt or VeraCrypt.

Windows

Installing Windows on the machine was easy enough, following the usual installation procedure. I created a new partition to install Windows to filling half of the disk, and let it do its thing. Downloading and installing Truecrypt is similarly easy. From there, I simply chose the relevant menu entry to turn on system encryption.

The first snag appeared when the system encryption wizard refused to continue until I had burned an optical disk containing the recovery information (in case the volume headers were to get corrupted). I opted to copy the iso file to another location, with the ability to boot it via grub4dos if necessary in the future (or merely burn a disc as necessary). The solution to this was to re-invoke the volume creation wizard with the noisocheck option:

C:\Program Files\TrueCrypt>TrueCrypt Format.exe /noisocheck

One reboot followed, and I was able to let TrueCrypt go through and encrypt the system. It was then time to set up Linux.

Linux

Basic setup of my Linux system was straightforward. Arch (my distribution of choice) offers good support for LUKS encryption of the full system, so most of the installation went smoothly.

On reaching the bootloader installation phase, I let it install and configure syslinux (my loader of choice simply because it is easier to configure than GRUB), but did not install it to the MBR. With the installation complete, I had to do some work to manually back up the MBR installed by Truecrypt, then install a non-default MBR for Syslinux.

First up was backing up the Truecrypt MBR to a file:

# dd if=/dev/sda of=/mnt/boot/tc.bs count=1

That copies the first sector of the disk (512 bytes, containing the MBR and partition table) to a file (tc.bs) on my new /boot partition.

Before installing a Syslinux MBR, I wanted to ensure that chainloading the MBR from a file would work. To that end, I used the installer to chainload to my new installation, and used that to attempt loading Windows. The following incantation (entered manually from the syslinux prompt) eventually worked:

.com32 chain.c32 hd0 1 file=/tc.bs

Pulling that line apart, I use the chainloader to boot the file tc.bs in the base of my /boot partition, and load the first partition on my first hard drive (that is, where Windows is installed). This worked, so I booted once more into the installer to install the Syslinux MBR:

# dd if=/usr/lib/syslinux/mbr.bin of=/dev/sda bs=1 count=440 conv=notrunc

This copies 440 bytes from the given file to my hard drive, where 440 bytes is the size of the MBR. The input file is already that size so the count parameter should not be necessary, but one cannot be too careful when doing such modification to the MBR.

Rebooting, that, sadly, did not work. It turns out that the Syslinux MBR merely scans the current hard drive for partitions that are marked bootable, and boots the first one. The Truecrypt MBR does the same thing, which is troublesome– in order for Truecrypt to work the Windows partition must be marked bootable, but Syslinux is unable to find its configuration when this is the case.

Enter albmbr.bin. Syslinux ships several different MBRs, and the alternate does not scan for bootable partitions. Instead, the last byte of the MBR is set to a value indicating which partition to boot from. Following the example from the Syslinux wiki (linked above), then, I booted once more from my installer and copied the altmbr into position:

# printf 'x5' | cat /usr/lib/syslinux/altmbr.bin - | dd bs=1 count=440 conv=notrunc of=/dev/sda

This shell pipeline echoes a single byte of value 5, appends it to the contents of altmbr.bin, and writes the resulting 440 bytes to the MBR on sda. The 5 comes from the partition Syslinux was installed on, in this case the first logical partition on the disk (/dev/sda5).

With that, I was able to boot Syslinux properly and it was a simple matter to modify the configuration to boot either Windows or Linux on demand. Selected parts of my syslinux.cfg file follow:

UI menu.c32

LABEL arch
    MENU LABEL Arch Linux
    LINUX /vmlinuz-linux
    APPEND root=/dev/mapper/Homura-root cryptdevice=/dev/sda6:HomuHomu ro
    INITRD /initramfs-linux.img

LABEL windows
    MENU LABEL Windows 7
    COM32 chain.c32
    APPEND hd0 1 file=/tc.bs

Further resources

For all things Syslinux, the documentation wiki offers documentation sufficient for most purposes, although it can be somewhat difficult to navigate. A message from the Syslinux mailing list gave me the key to making Syslinux work from the MBR. The Truecrypt documentation offered some interesting information, but was surprisingly useless in the quest for a successful chainload (indeed, the volume creation wizard very clearly states that using a non-truecrypt MBR is not supported).

Static libpng on win32 with CMake

Working on mkg3a upgrades for libpng more, I was getting unusual crashes with the gnuwin32 libpng binaries (access violations when calling png_read_int()).  It turned out that the libpng dll was built against an incompatible C runtime, so I had to build static libraries.  With the official libpng source distribution (and zlib), building static libraries was reasonably easy.  Using the MSVC make tool in the libpng source tree, I first had to build zlib. The default build (for some reason) doesn’t build the module containing _inflate_fast, so I had to add inffast.obj to the OBJS in zlib/win32/Makefile.msc (this manifested as an unexported symbol error when linking a program against zlib). Building it then was easy, using nmake in the Visual Studio toolkit:

zlib-1.2.5> nmake -f win32/Makefile.msc

With zlib built, copy zlib.h and zlib.lib out of the source directory and into wherever it will be used.

For libpng, we first have to modify the makefile, since the one included uses unusual options. Change CFLAGS to read CFLAGS=/nologo /MT /W3 -I..zlib for some sane options. The include path also needs to be updated to point to your zlib.h. In my case, that makes it -I..include. The rest of the procedure for building libpng is very similar to that for zlib:

lpng158> nmake -f scripts/Makefile.msc

Building against libpng then requires png.h, pngconf.h, pnglibconf.h and png.lib. To build against these libraries, I simply put the include files in an ‘include’ directory, the .lib files in a ‘lib’ directory, and pointed cmake at it.

Warnings about runtime libraries when linking a program against these static libraries is an indication that you’ll probably see random crashes, since it means theses static libraries are using a different version of the runtime libraries than the rest of your program. I was this problem manifested as random heap corruption. Changing CFLAGS (in the makefiles) to match your target configuration as set in Visual Studio and rebuilding these libraries will handle that problem.

Locating packages with cmake

When building programs with cmake on non-UNIX systems, it can be a pain to specify the location of external libraries. I’ve been upgrading mkg3a to support using libpng to load icons in addition to the old bmp loader, but that means I need to link against libpng, and also zlib (since libpng depends on zlib to handle the image compression). Compiling it all on Windows, however, is not an easy task, since there’s no standard search path for libraries like there is on UNIX systems (eg /usr/include for libraries, /usr/lib for libraries..). I didn’t find any good resources on how to make it work in my own searches, so here’s a quick write-up of the process in the hopes that it’ll be useful to somebody else.

I grabbed the zlib and libpng static libraries from gnuwin32 and extracted them near my mkg3a source tree, in the same directory. Setting up to build, then, my directory tree looks something like the following (some files omitted for brevity):

+ build
- libs
 - include
  + libpng12
  | png.h
  | pngconf.h
  | zconf.h
  | zlib.h
 - lib
  | libpng.lib
  | zlib.lib
 + manifest
- mkg3a
 | CMakeLists.txt
 | config.h.in
 | README

So I have a libs directory containing the headers and library files to link against, build is my build tree, and mkg3a is the source tree.

In order to tell cmake where to find zlib and libpng now, we can use the CMAKE_PREFIX_PATH variable, which is a path relative to the source directory. In this case, the following command will pick up the libraries in libs and generate project files for Visual Studio 2010 (note we’re executing from within the build tree):

H:Desktopbuild> cmake -G "Visual Studio 10" -D CMAKE_PREFIX_PATH=../libs ../mkg3a

If the build tree were instead under the source tree (mkg3a/build/ instead of just build/), the value for CMAKE_PREFIX_PATH would not need to change, since it is specified relative to the source directory.

In short: set CMAKE_PREFIX_PATH to help it find packages when they’re not in the usual system locations. It’s much easier to combine all your external libraries into one directory (libs in my example), but you could also specify a list of paths and keep them separate.

rtorrent scripting considered harmful

As best I can tell, whomever designed the scripting system for rtorrent did so in a manner contrived to make it as hard to use as possible.  It seems that = is the function application operator, and precedence is stated by using a few levels of distinct escaping. For example:

# Define a method 'tnadm_complete', which executes 'baz' if both 'foo' and 'bar' return true.
system.method.insert=tnadm_complete,simple,branch={and="foo=,bar=",baz=}

With somewhat more sane design, it might look more like this:

system.method.insert(tnadm_complete, simple, branch(and(foo(),bar()),baz()))

That still doesn’t help the data-type ambiguity problems (‘tnadm_complete’ is a string here, but not obviously so), but it’s a bit better in readability. I haven’t tested whether the escaping with {} can be nested, but I’m not confident that it can.

In any case, that’s just a short rant since I just spent about two hours wrapping my brain around it. Hopefully that work turns into some progress on a new project concept, otherwise it was mostly a waste. As far as the divergence meter goes, I’m currently debugging a lack of communication between my in-circuit programmer and the microcontroller.

Incidentally, the rtorrent community wiki is a rather incomplete but still useful reference for this sort of thing, while gi-torrent provides a reasonably-organized overview of the XMLRPC methods available (which appear to be what the scripting exposes), and the Arch wiki has a few interesting examples.

A few small projects

Going through some of my old projects this evening, I came across a couple little tools I wrote.  I’ve uploaded them here in the hope that others will find them useful.  They are the GCNClient GUI and RX BRR calculator.

I make no guarantees of the utility of these pieces of software, but they may be useful as examples in how to perform some task in the .NET framework (both are written in C# for .NET), or just for performing the very specific tasks which they are designed to perform.