Tag Archives: windows

Considering my backup systems

With the recent news that Crashplan were doing away with their “Home” offering, I had reason to reconsider my choice of online backup backup provider. Since I haven’t written anything here lately and the results of my exploration (plus description of everything else I do to ensure data longevity) might be of interest to others looking to set up backup systems for their own data, a version of my notes from that process follows.

The status quo

I run a Linux-based home server for all of my long-term storage, currently 15 terabytes of raw storage with btrfs RAID on top. The choice of btrfs and RAID allows me some degree of robustness against local disk failures and accidental damage to data.

If a disk fails I can replace it without losing data, and using btrfs’ RAID support it’s possible to use heterogenous disks, meaning when I need more capacity it’s possible to remove one disk (putting the volume into a degraded state) and add a new (larger) one and rebalance onto the new disk.

btrfs’ ability to take copy-on-write snapshots of subvolumes at any time makes it reasonable to take regular snapshots of everything, providing a first line of defense against accidental damage to data. I use Snapper to automatically create rolling snapshots of each of the major subvolumes:

  • Synchronized files (mounted to other machines over the network) have 8 hourly, 7 daily, 4 weekly and 3 monthly snapshots available at any time.
  • Staging items (for sorting into other locations) have a snapshot for each of the last two hours only, because those items change frequently and are of low value until considered further.
  • Everything else keeps one snapshot from the last hour and each of the last 3 days.

This configuration strikes a balance according to my needs for accident recovery and storage demands plus performance. The frequently-changed items (synchronized with other machines and containing active projects) have a lot of snapshots because most individual files are small but may change frequently, so a large number of snapshots will tend to have modest storage needs. In addition, the chances of accidental data destruction are highest there. The other subvolumes are either more static or lower-value, so I feel little need to keep many snapshots of them.

I use Crashplan to back up the entire system to their “cloud”1 service for $5 per month. The rate at which I add data to the system is usually lower than the rate at which it can be uploaded back to Crashplan as a backup, so in most cases new data is backed up remotely within hours of being created.

Finally, I have a large USB-connected external hard drive as a local offline backup. Also formatted with btrfs like the server (but with the entire disk encrypted), I can use btrfs send to send incremental backups to this external disk, even without the ability to send information from the external disk back. In practice, this means I can store the external disk somewhere else completely (possibly without an Internet connection) and occasionally shuttle diffs to it to update to a more recent version. I always unplug this disk from power and its host computer when not being updated, so it should only be vulnerable to physical damage and not accidental modification of its contents.

Continue reading Considering my backup systems

Static libpng on win32 with CMake

Working on mkg3a upgrades for libpng more, I was getting unusual crashes with the gnuwin32 libpng binaries (access violations when calling png_read_int()).  It turned out that the libpng dll was built against an incompatible C runtime, so I had to build static libraries.  With the official libpng source distribution (and zlib), building static libraries was reasonably easy.  Using the MSVC make tool in the libpng source tree, I first had to build zlib. The default build (for some reason) doesn’t build the module containing _inflate_fast, so I had to add inffast.obj to the OBJS in zlib/win32/Makefile.msc (this manifested as an unexported symbol error when linking a program against zlib). Building it then was easy, using nmake in the Visual Studio toolkit:

zlib-1.2.5> nmake -f win32/Makefile.msc

With zlib built, copy zlib.h and zlib.lib out of the source directory and into wherever it will be used.

For libpng, we first have to modify the makefile, since the one included uses unusual options. Change CFLAGS to read CFLAGS=/nologo /MT /W3 -I..zlib for some sane options. The include path also needs to be updated to point to your zlib.h. In my case, that makes it -I..include. The rest of the procedure for building libpng is very similar to that for zlib:

lpng158> nmake -f scripts/Makefile.msc

Building against libpng then requires png.h, pngconf.h, pnglibconf.h and png.lib. To build against these libraries, I simply put the include files in an ‘include’ directory, the .lib files in a ‘lib’ directory, and pointed cmake at it.

Warnings about runtime libraries when linking a program against these static libraries is an indication that you’ll probably see random crashes, since it means theses static libraries are using a different version of the runtime libraries than the rest of your program. I was this problem manifested as random heap corruption. Changing CFLAGS (in the makefiles) to match your target configuration as set in Visual Studio and rebuilding these libraries will handle that problem.

Locating packages with cmake

When building programs with cmake on non-UNIX systems, it can be a pain to specify the location of external libraries. I’ve been upgrading mkg3a to support using libpng to load icons in addition to the old bmp loader, but that means I need to link against libpng, and also zlib (since libpng depends on zlib to handle the image compression). Compiling it all on Windows, however, is not an easy task, since there’s no standard search path for libraries like there is on UNIX systems (eg /usr/include for libraries, /usr/lib for libraries..). I didn’t find any good resources on how to make it work in my own searches, so here’s a quick write-up of the process in the hopes that it’ll be useful to somebody else.

I grabbed the zlib and libpng static libraries from gnuwin32 and extracted them near my mkg3a source tree, in the same directory. Setting up to build, then, my directory tree looks something like the following (some files omitted for brevity):

+ build
- libs
 - include
  + libpng12
  | png.h
  | pngconf.h
  | zconf.h
  | zlib.h
 - lib
  | libpng.lib
  | zlib.lib
 + manifest
- mkg3a
 | CMakeLists.txt
 | config.h.in
 | README

So I have a libs directory containing the headers and library files to link against, build is my build tree, and mkg3a is the source tree.

In order to tell cmake where to find zlib and libpng now, we can use the CMAKE_PREFIX_PATH variable, which is a path relative to the source directory. In this case, the following command will pick up the libraries in libs and generate project files for Visual Studio 2010 (note we’re executing from within the build tree):

H:Desktopbuild> cmake -G "Visual Studio 10" -D CMAKE_PREFIX_PATH=../libs ../mkg3a

If the build tree were instead under the source tree (mkg3a/build/ instead of just build/), the value for CMAKE_PREFIX_PATH would not need to change, since it is specified relative to the source directory.

In short: set CMAKE_PREFIX_PATH to help it find packages when they’re not in the usual system locations. It’s much easier to combine all your external libraries into one directory (libs in my example), but you could also specify a list of paths and keep them separate.