Author Archives: tari

D-meter updates

Posted on by

I’ve been able to do some more work on the divergence meter now. The university’s labs made short work of the surface-mount soldering, but there were some hitches in the assembly and testing phase, in which I discovered some of the part footprints were wrong, and it was a bit of trouble getting the programmer working.

I was able to work around most of the bad footprints, but some of them were barely salvageable, since the through-holes were too small. I was able to drill them out on the drill press in the lab, but that left me with very small contact areas to solder to, so I had a few hideous solder joints.

After getting the power supply portions of the board soldered came getting the MSP430 talking to my MSP430 Launchpad, which I’m using as a programmer. Initial attempts to program the micro were met with silence (and mspdebug reporting no response from the target), but the problem turned out to be due to using cables that were too long- I had simply clipped test leads onto the relevant headers, yielding a programming cable that was around 1 meter long, while the MSP430 Hardware Tools User’s Guide (SLAU278) indicates that a programming cable should not exceed 20 cm in length. I assembled a shorter cable in response (by soldering a few wires onto the leads of a female 0.1″ socket) and all was well.

The most recent snag in assembly was the discovery that I had botched some of the MSP430′s outputs. I had connected the boost converter’s PWM input to Timer A output 0 on the micro, but I discovered while writing the code to control the boost converter that it’s impossible to output PWM on output module 0, due to the assignment of SFRs for timer control. The user’s manual for the chip even mentions this, but I simply failed to appreciate it.

I could have cut the a few traces and performed a blue wire fix, but it seemed like a very poor solution, and I was still concerned about the poor contact on the other vias I had to drill out, so I bit the bullet and created a new revision of the board with correct footprints for all the parts, and a more comprehensive ground plane (hopefully reducing inductive spiking on the optocoupler control lines). I’ve now sent revision 1.1 out to be made, so improved boards will be here in a few weeks. Until then, I’ll be working on the software a bit more, and hopefully updating this post with photographs.

Static libpng on win32 with CMake

Working on mkg3a upgrades for libpng more, I was getting unusual crashes with the gnuwin32 libpng binaries (access violations when calling png_read_int()).  It turned out that the libpng dll was built against an incompatible C runtime, so I had to build static libraries.  With the official libpng source distribution (and zlib), building static libraries was reasonably easy.  Using the MSVC make tool in the libpng source tree, I first had to build zlib. The default build (for some reason) doesn’t build the module containing _inflate_fast, so I had to add inffast.obj to the OBJS in zlib/win32/Makefile.msc (this manifested as an unexported symbol error when linking a program against zlib). Building it then was easy, using nmake in the Visual Studio toolkit:

zlib-1.2.5> nmake -f win32/Makefile.msc

With zlib built, copy zlib.h and zlib.lib out of the source directory and into wherever it will be used.

For libpng, we first have to modify the makefile, since the one included uses unusual options. Change CFLAGS to read CFLAGS=/nologo /MT /W3 -I..\zlib for some sane options. The include path also needs to be updated to point to your zlib.h. In my case, that makes it -I..\include\. The rest of the procedure for building libpng is very similar to that for zlib:

lpng158> nmake -f scripts/Makefile.msc

Building against libpng then requires png.h, pngconf.h, pnglibconf.h and png.lib. To build against these libraries, I simply put the include files in an ‘include’ directory, the .lib files in a ‘lib’ directory, and pointed cmake at it.

Warnings about runtime libraries when linking a program against these static libraries is an indication that you’ll probably see random crashes, since it means theses static libraries are using a different version of the runtime libraries than the rest of your program. I was this problem manifested as random heap corruption. Changing CFLAGS (in the makefiles) to match your target configuration as set in Visual Studio and rebuilding these libraries will handle that problem.

Ika Okonomiyaki

Posted on by
Thawing

This gallery contains 13 photos.

I cooked a bit of an experiment tonight: okonomiyaki with squid.  Preparing the squid took a long time, but I was very pleased with how it turned out in the end. To prepare the squid, simply pull to separate the body from the head.  Most of the innards will stay attached to the head.  Chop [...]

High-availability /home revisited

Posted on by

About a month ago, I wrote about my experiments in ways to keep my home directory consistently available. I ended up concluding that DRBD is a neat solution for true high-availability systems, but it’s not really worth the trouble for what I want to do, which is keeping my home directory available and in-sync across several systems.

Considering the problem more, I determined that I really value a simple setup. Specifically, I want something that uses very common software, and is resistant to network failures. My local network going down is an extremely rare occurence, but it’s possible that my primary workstation will become a portable machine at some point in the future- if that happens, anything that depends on a constant network connection becomes hard to work with.

If an always-online option is out of the question, I can also consider solutions which can handle concurrent modification (which DRBD can do, but requires using OCFS, making that solution a no-go).

Rsync

rsync is many users’ first choice for moving files between computers, and for good reason: it’s efficient and easy to use.  The downside in this case is that rsync tends to be destructive, because the source of a copy operation is taken to be the canonical version, any modifications made in the destination will be wiped out.  I already have regular cron jobs running incremental backups of my entire /home so the risk of rsync permanently destroying valuable data is low.  However, being forced to recover from backup in case of accidental deletions is a hassle, and increases the danger of actual data loss.

In that light, a dumb rsync from the NAS at boot-time and back to it at shutdown could make sense, but carries undesirable risk.  It would be possible to instruct rsync to never delete files, but the convenience factor is reduced, since any file deletions would have to be done manually after boot-up.  What else is there?

Unison

I eventually decided to just use Unison, another well-known file synchronization utility.  Unison is able to handle non-conflicting changes between destinations as well as intelligently detect which end of a transfer has been modified.  Put simply, it solves the problems of rsync, although there are still situations where it requires manual intervention.  Those are handled with reasonable grace, however, with prompting for which copy to take, or the ability to preserve both and manually resolve the conflict.

Knowing Unison can do what I want and with acceptable amounts of automation (mostly only requiring intervention on conflicting changes), it became a simple matter of configuration.  Observing that all the important files in my home directory which are not already covered by some other synchronization scheme (such as configuration files managed with Mercurial) are only in a few subdirectories, I quickly arrived at the following profile:

root = /home/tari
root = /media/Caring/sync/tari

path = incoming
path = pictures
path = projects
path = wallpapers

Fairly obvious function here, the two sync roots are /home/tari (my home directory) and /media/Caring/sync/tari (the NAS is mounted via NFS at /media/Caring), and only the four listed directories will be syncronized. An easy and robust solution.

I have yet to configure the system for automatic syncronization, but I’ll probably end up simply installing a few scripts to run unison at boot and when shutting down, observing that other copies of the data are unlikely to change while my workstation is active.  Some additional hooks may be desired, but I don’t expect configuration to be difficult.  If it ends up being more complex, I’ll just have to post another update on how I did it.

Update Jan. 30: I ended up adding a line to my rc.local and rc.shutdown scripts that invokes unison:

su tari -c "unison -auto home"

Note that the Unison profile above is stored as ~/.unison/home.prf, so this handles syncing everything I listed above.

Locating packages with cmake

Posted on by

When building programs with cmake on non-UNIX systems, it can be a pain to specify the location of external libraries. I’ve been upgrading mkg3a to support using libpng to load icons in addition to the old bmp loader, but that means I need to link against libpng, and also zlib (since libpng depends on zlib to handle the image compression). Compiling it all on Windows, however, is not an easy task, since there’s no standard search path for libraries like there is on UNIX systems (eg /usr/include for libraries, /usr/lib for libraries..). I didn’t find any good resources on how to make it work in my own searches, so here’s a quick write-up of the process in the hopes that it’ll be useful to somebody else.

I grabbed the zlib and libpng static libraries from gnuwin32 and extracted them near my mkg3a source tree, in the same directory. Setting up to build, then, my directory tree looks something like the following (some files omitted for brevity):

+ build\
- libs\
 - include\
  + libpng12\
  | png.h
  | pngconf.h
  | zconf.h
  | zlib.h
 - lib\
  | libpng.lib
  | zlib.lib
 + manifest\
- mkg3a\
 | CMakeLists.txt
 | config.h.in
 | README

So I have a libs directory containing the headers and library files to link against, build is my build tree, and mkg3a is the source tree.

In order to tell cmake where to find zlib and libpng now, we can use the CMAKE_PREFIX_PATH variable, which is a path relative to the source directory. In this case, the following command will pick up the libraries in libs and generate project files for Visual Studio 2010 (note we’re executing from within the build tree):

H:\Desktop\build> cmake -G "Visual Studio 10" -D CMAKE_PREFIX_PATH=../libs ../mkg3a

If the build tree were instead under the source tree (mkg3a/build/ instead of just build/), the value for CMAKE_PREFIX_PATH would not need to change, since it is specified relative to the source directory.

In short: set CMAKE_PREFIX_PATH to help it find packages when they’re not in the usual system locations. It’s much easier to combine all your external libraries into one directory (libs in my example), but you could also specify a list of paths and keep them separate.

rtorrent scripting considered harmful

Posted on by

As best I can tell, whomever designed the scripting system for rtorrent did so in a manner contrived to make it as hard to use as possible.  It seems that = is the function application operator, and precedence is stated by using a few levels of distinct escaping. For example:

# Define a method 'tnadm_complete', which executes 'baz' if both 'foo' and 'bar' return true.
system.method.insert=tnadm_complete,simple,branch={and="foo=,bar=",baz=}

With somewhat more sane design, it might look more like this:

system.method.insert(tnadm_complete, simple, branch(and(foo(),bar()),baz()))

That still doesn’t help the data-type ambiguity problems (‘tnadm_complete’ is a string here, but not obviously so), but it’s a bit better in readability. I haven’t tested whether the escaping with {} can be nested, but I’m not confident that it can.

In any case, that’s just a short rant since I just spent about two hours wrapping my brain around it. Hopefully that work turns into some progress on a new project concept, otherwise it was mostly a waste. As far as the divergence meter goes, I’m currently debugging a lack of communication between my in-circuit programmer and the microcontroller.

Incidentally, the rtorrent community wiki is a rather incomplete but still useful reference for this sort of thing, while gi-torrent provides a reasonably-organized overview of the XMLRPC methods available (which appear to be what the scripting exposes), and the Arch wiki has a few interesting examples.

Divergence meter progress

Posted on by

One project which I’ve been working on since about October and just got around to creating a project page for is the divergence meter.

There’s not a lot to see there yet, but I’ve recorded my notes on what the design needs and the outline for the control and power supply module.  I ordered the PCB in early December in the hopes that they would be available for me to work on while in Wauwatosa during the semester break.  That didn’t pan out, so unfortunately the whole project won’t move until next week, when I return to Houghton and can get my boards from the mailbox.

My batch of nixie tubes arrived earlier than expected, however, and I got the components to populate the board in mid-November.  All I need is the boards and some time to solder, while hoping I don’t completely botch the job of soldering a 38-TSOP package, especially since that chip (the MSP430F2272) cost me $5.  Photos follow.

One I find the time to assemble the control board, the software should come together pretty quickly.  Just a matter of time now..

Experiments with a high-availability /home

Posted on by

I was recently experimenting with ways to configure my computing setup for high availability of my personal data, which is stored in a Btrfs-formatted partition on my SSD. When my workstation is booted into Windows, however, I want to be able to access my data with minimal effort. Since there’s no way to access a Btrfs volume natively from within Windows, I had to find another approach. It seemed like automatically syncing files out to my NAS was the best solution, since that’s always available and independent of most other things I would be doing at any time.

Candidates

The obvious first option for syncing files to the NAS is the ever-common rsync. It’s great at periodic file transfers, but real-time syncing of modifications is rather beyond the ken of rsync.  lsync provides a reasonable way to keep things reasonably in-sync, but it’s far from an elegant solution.  Were I so motivated, it would be reasonable to devise a similar rsync wrapper using inotify (or similar mechanisms) to only handle modified files and possibly even postpone syncing changes until some change threshold is exceeded.  With existing software, however, rsync is a rather suboptimal solution.

From a cursory scan, cluster filesystems such as ceph or lustre seem like good options for tackling this problem.  The main disadvantage of the cluster filesystem approach, however, is rather high complexity. Most cluster filesystem implementations require a few layers of software, generally both a metadata server and storage server. In large deployments that software stack makes sense, but it’s needless complexity for me.  In addition, ensuring that data is correctly duplicated across both systems at any given time may be a challenge.  I didn’t end up trying this route so ensuring data duplication may be easier than it seems, but a cluster filesystem ultimately seemed like needless complexity for what I wanted to do.

While researching cluster filesystems, I discovered xtreemfs, which has a number of unique features, such as good support for wide-area storage networks, and is capable of operating securely even over the internet.  Downsides of xtreemfs are mostly related to the technology it’s built on, since the filesystem itself is implemented with Linux’s FUSE (Filesystem in USErspace) layer and is implemented in Java.  Both those properties make it rather clunky to work with and configure, so I ended up looking for another solution after a little time spent attempting to build and configure xtreemfs.

The solution I ultimately settled upon was DRBD, which is a block-level replication tool.  Unlike the other approaches, DRBD sits at the block level (rather than the filesystem level), so any desired filesystem can be run on top of it.  This was a major advantage to me, because Btrfs provides a few features that I find important (checksums for data, and copy-on-write snapshotting). Handling block-level syncing is necessarily somewhat more network-intensive than running at the file level, but since I was targeting use over a gigabit LAN, network usage was a peripheral concern.

Implementation

From the perspective of normal operation, a DRBD volume looks like RAID 1 running over a network.  One host is marked as the primary, and any changes to the volume on that host are propagated to the secondary host.  If the primary goes offline for whatever reason, the secondary system can be promoted to the new primary, and the resource stays available. In the situation of my designs for use of DRBD, my workstation machine would be the primary in order to achieve normal I/O performance while still replicating changes to the NAS. Upon taking the workstation down for whatever reason (usually booting it into another OS), all changes should be on the NAS, which remains active as a lone secondary.

DRBD doesn’t allow secondary volumes to be used at all (mainly since that would introduce additional concerns to ensure data integrity), so in order to mount the secondary and make it accessible (such as via a Samba share) the first step is to mark the volume as primary. I was initially cautious about how bringing the original primary back online would affect synchronization, but it turned out to handle such a situation gracefully. When the initial primary (workstation) comes back online following promotion of the secondary (NAS), the former primary is demoted back to secondary status, which also ensures that any changes while the workstation was offline are correctly mirrored back. While the two stores are resyncing, it is possible to mark the workstation as primary once more and continue normal operation while the NAS’ modifications sync back.

Given that both my NAS and workstation machines run Arch Linux, setup of DRBD for this scheme was fairly simple. First order of business was to create a volume to base DRBD on. The actual DRBD driver is part of mainline Linux since version 2.6.33, so having the requisite kernel module loaded was easy. The userspace utilities are available in the AUR, so it was easy to get those configured and installed. Finally, I created a resource configuration file as follows:

resource home {
  device /dev/drbd0;
  meta-disk internal;

  protocol A;
  startup {
    become-primary-on Nakamura;
  }

  on Nakamura {
    disk /dev/Nakamura/home;
    address ipv4 192.168.1.10:7789;
  }
  on Nero {
    disk /dev/loop0;
    address ipv4 192.168.1.8:7789;
  }

}

The device option specifies what name the DRBD block device should be created with, and meta-disk internal specifies that the DRBD metadata (which contains such things as the dirty bitmap for syncing modified blocks) should be stored within the backing device, rather than in some external file. The protocol line specifies asynchronous operation (don’t wait for a response from the secondary before returning saying a write is complete), which helps performance but makes the system less robust in the case of a sudden failure. Since my use case is less concerned with robustness and more with simple availability and maintaining performance as much as possible, I opted for the asynchronous protocol. The startup block specifies that Nakamura (the workstation) should be promoted to primary when it comes online.

The two on blocks specify the two hosts of the cluster. Nakamura’s volume is backed by a Linux logical volume (in the volume group ‘Nakamura’), while Nero’s is hosted on a loop device. I chose to use a loop device on Nero simply because the machine has a large amount of storage (6TB in RAID5), but no unallocated space, so I had to use a loop device. In using a loop device I ended up ignoring a warning in the DRBD manual about running it over loop block devices causing deadlocks– this ended up being a poor choice, as described later.

It was a fairly simple matter of bringing the volumes online once I had written the configuration. Load the relevant kernel module, and use the userland utilities to set up the backing device. Finally, bring the volume up. Repeat this series of steps again on the other host.

# modprobe drbd
# drbdadm create-md home
# drbdadm up home

With the module loaded and a volume online, status information is visible in /proc/drbd, looking something like the following (shamelessly taken from the DRBD manual):

$ cat /proc/drbd
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by buildsystem@linbit, 2008-12-18 16:02:26
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r---
    ns:0 nr:8 dw:8 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

The first few lines provide version information, and the two lines beginning with ’0:’ describe the state of a DRBD volume. Of the rest of the information, we can see that both hosts are online and communicating (Connected), both are currently marked as secondaries (Secondary/Secondary), and both have the latest version of all data (UpToDate/UpToDate). The last step in creating the volume is to mark one host as primary. Since this is a newly-created volume, marking one host as primary requires invalidation of the other, prompting resynchronization of the entire device. I execute drbdadm primary --force home on Nakamura to mark that host as having the canonical version of the data, and the devices begin to synchronize.

Once everything is set, it becomes possible to use the DRBD block device (/dev/drbd0 in my configuration) like any other block device- create filesystems, mount it, or write random data to it. With a little work to invoke the DRBD initscripts at boot time, I was able to get everything working as expected. There were a few small issues with the setup, though:

  • Nero (the NAS) required manual intervention to be promoted to the primary role. This could be improved by adding some sort of hooks on access to promote it to primary and mount the volume. This could probably be implemented with autofs for a truly transparent function, or even a simple web page hosted by the NAS which prompts promotion when it is visited.
  • Deadlocks! I mentioned earlier that I chose to ignore the warning in the manual about deadlocks when running DRBD on top of loop devices, and I did start seeing some on Nero. All I/O on the volume hosting the loop device on Nero would stall, and the only way out was by rebooting the machine.

Conclusion

DRBD works for keeping data in sync between two machines in a transparent fashion, at the cost of a few more software requirements and a slight performance hit. The kernelspace tools are in mainline Linux so should be available in any reasonably recent kernel, but availability of the userspace utilities is questionable. Fortunately, building them for oneself is fairly easy. Provided the drbd module is loaded, it is not necessary to use the userspace utilities to bring the volume online- the backing block device can be mounted without DRBD, but the secondary device will need to be manually invalidated upon reconnect. That’s useful for ensuring that it’s difficult for data to be rendered inaccessible, since the userspace utilities are not strictly needed to get at the data.

I ultimately didn’t continue running this scheme for long, mainly due to the deadlock issues I had on the NAS, which could have been resolved with some time spent reorganizing the storage on that host. I decided that wasn’t worth the effort, however. To achieve a similar effect, I ended up configuring a virtual machine on my Windows installation that has direct access to the disks which have Linux-hosted data, so I can boot the physical Linux installation in a virtual machine. By modifying the initscripts a little, I configured it to start Samba at boot time when running virtualized in order to give access to the data. The virtualized solution is a bit more of a hack than DRBD and is somewhat less robust (in case of unexpected shutdown, this makes two operating systems coming down hard), but I think the relative simplicity and absence of a network tether are a reasonable compromise.

Were I to go back to a DRBD-backed solution at some time, I might want to look into using DRBD in dual-primary mode. In most applications only a single primary can be used since most filesystems are designed without the locking required to allow multiple drivers to operate on them at the same time (this is why NFS and similar network filesystems require lock managers). Using a shared-disk filesystem such as OCFS (or OCFS2), DRBD is capable of having both hosts in primary mode, so the filesystem can be mounted and modified on both hosts at once. Using dual primaries would simplify the promotion scheme (each host must simply be promoted to primary when it comes online), but would also require care to avoid split-brain situations (in which communications are lost but both hosts are still online and processing I/O requests, so they desync and require manual intervention to resolve conflicts). I didn’t try OCFS2 at all during this experiment mainly because I didn’t want to stop using btrfs as my primary filesystem.

To conclude, DRBD works for what I wanted to do, but deadlocks while running it on a loop device kept me from using it for long. The virtual machine-based version of this scheme performs well enough for my needs, despite being rather clunky to work with. I will keep DRBD in mind for similar uses in the future, though, and may revisit the issue at a later date when my network layout changes.

Update 26.1.2012: I’ve revisited this concept in a simpler (and less automatic) fashion.

Some beautiful music

I very rarely use the work ‘beautiful’, but in the case of this song, I think it’s apt. Kimi no shiranai monogatari by Supercell, the closing theme for Bakemonogatari (which is well worth watching in its own right, hosting some other catchy music as well as having a unique visual style).


(Turn on captions and up the video quality for great justice)

A delicious sound, and I also like the allusion to Tanabata (in referring to the Summer Triangle).  Taken in the context of Bakemonogatari, the power of the lyrics is only increased.

Continue reading →

A few small projects

Posted on by

Going through some of my old projects this evening, I came across a couple little tools I wrote.  I’ve uploaded them here in the hope that others will find them useful.  They are the GCNClient GUI and RX BRR calculator.

I make no guarantees of the utility of these pieces of software, but they may be useful as examples in how to perform some task in the .NET framework (both are written in C# for .NET), or just for performing the very specific tasks which they are designed to perform.