タリ

A few CPU hardware bugs

Wed, 28 Jan 2026 20:06:00 +1100

Catherine (Whitequark)’s recent observations on poorly -engineered firmware reminded me of a few mistakes I’ve seen in vendors’ CPUs; some unimportant and others surprisingly bad. Since I’ve never seen these widely discussed, here’s some discussion and links to supporting evidence to make them more widely known, since I think they’re interesting.

Intel’s misspelled CPUIDs

I’m aware of two situations where Intel have sold CPUs that report misspelled names in some of the strings returned by the CPUID instruction. This seems embarrassing for an organization of Intel’s size, but probably doesn’t hurt anybody’s ability to use the CPUs in question.

GenuineIotel

A web search for “GenuineIotel” reveals some discussions regarding this apparent typo, where some processors such as the Xeon E3-1231 v3 return the string “GenuineIotel” (instead of the usual “GenuineIntel”) for the CPU manufacturer ID. This one is well-known enough to be mentioned in the list of manufacturer IDs on Wikipedia.

It’s possible this misspelling is actually caused by some kind of random bit error, since the characters ’n’ and ‘o’ differ by only one bit; an unpredictable error that sets that bit could change GenuineIntel to GenuineIotel.

ore i5

Another error that seems more likely to be human error in the CPU is in the Core i5-1245U CPU, which returns a processor brand string Intel(R) ore(TM) i5-1245U which is simply missing the ‘C’ in Core(TM) i5. Web searches for “Intel(R) ore(TM)” show a number of results which could be errors introduced by non-technical users attempting to copy down text from their screen when asking for tech support, but the Ubuntu certified configuration of the Dell Latitude 5430 with this CPU attests to this error actually being present in at least some machines using that CPU.

It’s possible this misspelling is not part of the physical CPU design and is instead part of the system firmware because at least on many AMD CPUs the CPU name is normally set by the system firmware. Probably either the CPU design or its microcode encode this misspelling, or Intel’s firmware package that vendors use is the ultimate source. In either case, it seems embarrassing for them that such an error made it out into machines purchased by members of the public because it seems very likely to be the result of human error.

ITE’s pipeline bug

This one is an actual hardware bug that I discovered at work, but in an embedded processor which most people will never see.

ITE Tech is a Taiwanese chip company that sells a variety of specialized ICs, including a selection of PC embedded controllers (which are used for tasks like making the keyboard work and managing battery charging in most laptops). IT81202 is one of those, with on-chip peripherals for communicating with an x86 processor and plenty of memory for private use by its RISC-V CPU.

It turns out there’s a pipeline bug in the IT81202 CPU, where instructions modifying some registers immediately following a multiply (mul instruction) may have no effect. The workaround for this is to cripple the system, telling your compiler that the CPU doesn’t support multiplication or division instructions. Some of the performance can be regained by providing implementations of the library functions that provide integer multiply/divide operations that work in terms of the mul and div instructions, which works because inserting no-op instructions after them prevents the issue.

To me, this issue doesn’t seem as embarrassing as Intel’s wrong CPUIDs. Pipelined CPUs are hard to build, and at the time they designed the IT81202 CPU RISC-V wasn’t widely used in industry yet so they probably had a pretty immature core implementation. In addition, that’s an embedded processor which very few people will ever need to write software for so an invasive workaround like that isn’t a big deal. This one seems more like a cautionary tale to be aware of than any reason to mock the vendor!

WEB1999: the web of 1999 in math class

Wed, 22 Jan 2025 16:00:00 +1100

I often find that imposed limitations make it easier to create things: it’s easy to aim for perfection if you can expend as much effort as you like on something and thus end up with nothing that you’ll ever call good enough to share. Over on Cemetech in the final months of 2023,¹ we held a programming contest: write a screensaver, any kind of screensaver. I’m not often one to do any kind of competitive programming, but as a prompt for a constrained project this was a good one for me. As a result, I wrote a program I called WEB1999, that won second place.

A screensaver is a nice project because there’s a lot of room in which to play and many of the hard aspects of programming (interacting with humans) can be completely ignored! I wasn’t originally planning to write anything to enter, but after thinking to myself what I might expect other people to enter, I came up with an idea that piqued my interest and didn’t seem too hard to build.

In this post you will find...

Get the program

If you want to play with the program instead of or in addition to simply reading about it, a copy of WEB1999 suitable for loading onto a TI-84+ CE or TI-83 Premium CE calculator can be obtained from the releases section of my Gitlab project:
https://gitlab.com/taricorp/web1999/-/releases

The source code can also be found on Gitlab, or a copy can also be obtained from the Cemetech archives or ticalc.org. All of the release packages include the full source code and documentation, so you shouldn’t miss anything regardless of where you choose to get a copy from.

WEB1999 in action

WEB1999 in action.

Inspired heavily by the Realistic Internet Simulator (“Kill the Pop-ups”), WEB1999 invokes the spirit of pop-up advertising and Internet culture around the turn of the millennium in the form of a program that runs on Texas Instruments’ TI-84+ CE color-screen graphing calculators.

The contest judges had some very nice things to say:

An excellent instance of a screensaver modeled after both the golden age of screensavers and the golden age of the internet. Callbacks to both classic internet culture and the calculator community of yore, with a few easter eggs to boot.

An HTML readme that looks like it’s from ‘99? Nice!

I did spend some significant effort writing the documentation for the program, but I didn’t intentionally design the README with retro style; it mostly just uses the default HTML structure provided by Pandoc!

It looks great. The posterization and dithering really gives the feel of the 90s, and the use of an even more limited palette is a clever way to work within the bounds. Capturing the nostalgia of having just way too many popups.

As described further in the following sections, not only did I have a lot of fun using historical inspiration and realistic limitations to restrict myself, but there were also a few interesting aspects to the implementation that I’ll describe!

Continue reading for notes on the program’s design and interesting aspects of its implementation, plus additional improvements I’d like to make.

Implementation notes

WEB1999 has two major tasks that it needs to handle while running:

Periodically open a new window.
Move the cursor to an open window and close it.

Opening windows is fairly easy, but how that interacts with moving the cursor and interacting with windows is rather more complex. When there are multiple windows onscreen, some UI elements might be obscured because the windows are stacked. This presents a challenge to the cursor, because it should not “interact” with UI elements that are not visible (not that it actually interacts, it just needs to close a window when that’s plausibly possible) and consequently needs to choose a window and an element of that window to move to and interact with.

As it turns out, it’s much easier as a software author to assume an intelligent user than it is to automate behavior that looks intelligent. On the upside however, automating everything means none of the interaction actually needs to work, and shortcuts can be taken there.

Cursor behavior

The Large Test Window illustrates the important UI elements.

Starting with the goal of closing a window by pressing the ‘X’ button in its upper-right corner, it’s simple to close a window when it’s alone onscreen. Simply by moving the cursor to the ‘X’ button, we can plausibly interact with it and then remove the window.

If the ‘X’ button is offscreen however, it’s impossible to interact with. It would be reasonable to ensure that the button is never placed offscreen when creating a new window, but that would be less visually interesting. So to handle such a situation, the cursor must move to the window’s titlebar and “drag” the entire window to the left until its close button is visible. In the video below, the program only ever has one window to consider, but it moves the cursor to that window and closes it, moving the window if needed.

When there are multiple windows onscreen, they can be modelled as a simple list where the first window in the list is on top. By drawing them on the screen from the bottom up, it’s easy to ensure windows higher in the stack obscure the lower ones.

Window occlusion

When there are multiple windows onscreen, some of them might be obscuring important parts of others like the close button or even hiding the lower one completely.

Although the topmost window should always be interactable, I didn’t think always going to the topmost window would be interesting behavior, and it doesn’t seem to match what a human would do in this situation: a human would probably choose a window located near the cursor and close it, unless it became obscured before it could be closed. As it turns out, that human behavior is fairly straightforward to express as long as it’s clear which windows are not obscured.

This was one of the situations where I took a shortcut, since although a window’s close button or titlebar should be interactable if any part of it is visible, it was much simpler to treat a UI element as visible only if none of it was obscured. While looping over all windows stacked on top of a candidate window, the ones that cannot be closed or moved can be ignored.

This code sample (adapted from the real program’s getNewTarget function) illustrates generally how the logic to select a window works, assuming one that we want to interact with is returned by getAWindow. wm is the list of windows sorted with the topmost first:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


extern Window wm[MAX_WINDOWS];

Window& candidate = getAWindow();
Rect closeTarget = candidate.getCloseTarget();
bool closeOccluded = false;
Rect moveTarget = candidate.getMoveTarget();
bool moveOccluded = false;

while (occluderIdx-- > 0) {
 // Occlusion is simply based on having any overlap, because partial overlap
 // involves much more complicated geometry.
 auto occBounds = wm[occluderIdx]->getBounds();
 closeOccluded |= occBounds.overlapsWith(closeTarget);
 moveOccluded |= occBounds.overlapsWith(moveTarget);
}

// If both targets are occluded, this window is not interactable.
if (closeOccluded && moveOccluded) {
 dbg_printf("Window %p is fully occluded, will not interact.\n",
 &candidate);
 continue;
}

In the real program, this runs in a loop until an interactable window is found with candidates chosen by their distance from the cursor: the program will choose to interact with the window that is closest to the cursor, as long as the region that the cursor needs to be in to either move or close the window is not occluded by another window.

This code sample also illustrates a few of the abstractions I wrote to hide some of the details of the math. A Window’s location is described by a Rect which has a position onscreen, width and height; there are also shortcut functions to get the Rects corresponding to where the window’s titlebar is (which can be interacted with to bring the window to the top and move it), as well as its close button. The overlapsWith method on Rects checks whether any part of a rectangle overlaps with some other rectangle.

Distance metrics

The next moderately interesting question to answer was how to determine which window is closest to the cursor, since it’s now understood how to determine if a given window is actually an acceptable candidate to interact with.

The obvious approach is to use basic geometry and get the length of a vector between the cursor and the closest point of a window to the cursor. If cx and cy are the cursor’s X and Y coordinates while wx and wy are the coordinates of a point on a window, the distance is clearly sqrt((cx-wx)*(cx-wx) + (cy - wy)*(cy - wy)): simply apply the Pythagorean theorem.

On an eZ80 processsor where I could be doing this computation with fairly high frequency, I didn’t think getting a true hypotenuse length would be acceptably performant. There’s no hardware support for computing the square root of numbers, and I wasn’t very interested in trying to implement a fast square root.

Instead, I noticed that the actual distance between two points doesn’t actually matter, as long as I can know which of multiple points (points on windows that we might want to interact with) is closest to a chosen point (the cursor’s current location). Since the relationship between any two distances (which is longer) before computing the square root and even before squaring the coordinate offsets is the same as after doing those operations, I simplified distance computations to work in terms of the rectilinear distance instead:

1
2
3


static int distance_NonLinear(gfx_point_t a, gfx_point_t b) {
 return abs(a.x - b.x) + abs(a.y - b.y);
}

With a distance metric, the getNewTarget function can iterate over all windows onscreen and find the one nearest to the cursor. The actual distance is not known, but we do know the nearest window is found. As discussed in the previous section, windows where none of the interaction targets are visible (abstracted out as the isFullyOccluded function) are ignored.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


Window* nearest = nullptr;
int nearestMetric = INT_MAX;

for (auto candidateIdx = 0; candidateIdx < wm.size(); candidateIdx++) {
 Window* candidate = wm[candidateIdx];
 if (isFullyOccluded(candidate)) {
 continue;
 }

 auto candidateDest = closeTarget.getNearestPoint(cursor.getLocation());
 auto candidateDist = distance_NonLinear(cursor.getLocation(), candidateDest);
 if (candidateDist < nearestMetric) {
 nearest = candidate;
 nearestMetric = candidateDist;
 }
}

return nearest;

With that logic in mind, here’s a prototype in action where new windows were periodically being spawned and an appropriate one was being closed, with movement of the window as required.

The cursor’s speed seems inconsistent here because this particular prototype restarted moving the cursor every time it considered whether a new window should be targeted (when a window was created), and (as described in the following sections) at the time I recorded this video the cursor moved over short distances moved much more slowly than a human would.

Cursor movement

In the previous video, the cursor is already moving along a line between its original location and its destination. This is very easy for a human to do when using a mouse, but took some effort to implement in software in a way that was reasonably performant on a (e)Z80 processor.

The first algorithm I thought of when considering how to determine where to move the cursor was to use Bresenhams’ line algorithm, which I have previously heard of used for fast line-drawing functions on TI calculators. After spending some time familiarizing myself with Bresenham’s algorithm, I realized there was an issue with simply applying that algorithm: I need the cursor to move over time, not merely draw the line between two points.

At this point I decided it made sense to take advantage of the eZ80 being more capable than most 8-bit processors, in that it is able to do arithmetic on integers that are up to 24 bits wide and can multiply (but not divide) 16-bit integers in hardware. So I reached for fixed-point math.

What I ended up with was a Mover class encapsulating the details of computing how the cursor should move, which is constructed with two points to move between and can be advanced over time to sweep between the start and end points.

1
2
3
4
5
6
7
8


class Mover {
public:
 Mover(gfx_point_t start, gfx_point_t end);

 // Return the new position after moving at the given speed,
 // setting *done to true if the new position is the endpoint.
 gfx_point_t advance(uint8_t speed, bool *done);
};

Because it seemed reasonable to move the cursor in up to 128 steps, I chose to express the progress from start to end point as an 8-bit unsigned value, with 1 bit before the decimal point and 7 after: the 8-bit integer 0 is 0.0, and 0x80 is 1.0. In this way, moving the cursor at minimum nonzero speed involves adding 1/128 (integer value 1) to the progress along the line at each step.

Inspired by Bresenham’s line algorithm, for each of the x and y coordinates I multiplied the required distance to be moved by the fraction of the distance to be moved (that fixed-point number) to get the distance to move the cursor in one step. To prevent drift, the fractional part is saved and accumulates between calls, adding to the distance moved when it exceeds 1 pixel. The code looks like this, where speed is a fixed-point fraction of the total distance to cover, progress is the fraction of the distance covered already, dx and dy are the total distance to move on the X and Y axes, and errx and erry are the accumulated error for each axis:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


gfx_point_t Mover::advance(uint8_t speed) {
 // If speed would cause overshoot, clamp progress to 1.0.
 if (speed + progress > 0x80) {
 speed = 0x80 - progress;
 }
 gfx_point_t out;

 // Fixed-point multiplication, taking the integer part.
 out.x = (dx * speed) / 0x80;
 // Take the fractional part and add to accumulated error.
 errx += abs(dx * speed) & 0x7f;
 // If error is greater than 1.0, advance towards the destination
 // and subtract 1.0 from the error.
 if (errx >= 0x80) {
 out.x += signum(dx);
 errx -= 0x80;
 }

 // Do the same for the y axis...

 return out;
}

Notably, this implementation shouldn’t require any actual division because the divisions are always by 128, which can be expressed as an arithmetic right shift by 7 bits instead. Multiplication is still needed, but eZ80’s hardware multiplier should be sufficient to make that fast.

Cursor pacing

As I noted in relation to the last video, the initial approach I chose to selecting the speed at which the cursor should move yielded awkward results where it tended to move much slower than a human would if it only needed to move a short distance. This had two reasons:

The amount of time required to move between two points was constant: the speed used to advance the Mover was always 1/128, so the cursor would always take 128 ticks to move between two positions.
When a new window was created, cursor movement was reset. If the new destination was close to the cursor’s current position, it would always take 128 ticks to reach the new position which could make it seem to suddenly change speed to move very slowly when a new window appeared.

Clearly, constant-time cursor movement doesn’t look like what a human would do. I expect that a human attempting to move a mouse cursor to a given location onscreen would tend to move faster (cover more pixels each second) for longer moves, and might slow down as they near the target. In general, the time taken to move a given distance onscreen would be roughly constant whereas what I had implemented was that the fraction of the distance covered would be roughly constant.

Because the Mover still works in fractions of the distance to cover however, there needs to be a conversion between speed in pixels and the fraction of the total distance to cover in each tick. The obvious approach is to compute the distance to cover, then divide that distance by the desired speed in pixels per tick and multiply by 1.0 (as a fixed-point value; 128): speed_px * 128 / distance_px. For example, if the cursor moves at a constant speed of 4 pixels per tick and it needs to move 64 pixels total, then it’s obvious that it should move one sixteenth of the total distance per tick (because 64 divided by 4 is 16): 4 * 128 / 64 = 8, which is equal to 1/16 when interpreted as a fixed-point value with 7 fractional bits.

That obvious approach bumps into both of the issues that we’ve already used tricks to improve the performance of, though: it both needs to accurately compute the distance to traverse (which was previously replaced with the rectilinear distance in distance_NonLinear) and divide by that number (whereas we chose to do fixed-point arithmetic so the only required division can be expressed as a bitshift).

Noticing that accurately expressing the fraction of the desired distance to move would always require an accurate computation of the Euclidean distance between two points on the screen (which I didn’t want to do, because computing the square root of a number would probably be slow), I instead chose to manually write a piecewise function that looked about right:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// d is the rectilinear distance to travel
if (d == 0) {
 return 128;
} else if (d < 3) {
 return 64;
} else if (d < 9) {
 return 32;
} else if (d < 16) {
 return 16;
} else if (d < 48) {
 return 8;
} else if (d < 96) {
 return 4;
} else if (d < 152) {
 return 2;
} else {
 return 1;
}

In practice, this means that moves of 0 distance will take one tick to complete, 2 or less take 2 ticks, 8 or less take 4 ticks and so forth. I initially wrote that function to work in powers of two exclusively (branches for values of d less than 1, 2, 4, 8, … taking 1, 2, 4, 8, 16, … ticks to complete) and then manually adjusted the points where the speed would change by just watching the program run while printing debug output (how far the cursor was going to move next and the chosen speed), so when I thought a cursor movement looked bad I could make a note and eventually converge on good-looking speeds.

This speed computation function is clearly not very accurate, but because it’s meant to behave like a hypothetical human operating a computer mouse it’s possibly even better this way than if it were accurate: a human’s speed won’t always be exactly the same, and it builds in some bias toward moving slightly slower for very short moves where a human might be more careful to move accurately.

Window placement

The last somewhat awkward thing in the previous video is that the placement of windows on the screen doesn’t seem as random as it should be: they tend to be toward the right side of the screen, whereas it seems more likely that pop-up windows would attempt to place themselves near the center of the screen for maximum ~~annoyance~~ visibility.

To choose the position of a window when it was created, I was simply generating uniform (or near enough to uniform) random screen coordinates for it to be displayed at (0-319 on the X axis and 0-239 on the Y), then clamping values to ensure there would always be something interactable visible (the titlebar to drag the window, or its close button).

However, the coordinates of a window were the position of its Rect, which is defined to be the top left corner. That meant that windows would never hang off the left side of the screen, and could hang almost all the way off the right which would tend to look like a bias toward the right side of the screen.

The obvious first thing to change was to make the randomly-selected X coordinate represent the center of a window, rather than the left edge. While I did make that change, I also considered how to make windows prefer to be placed toward the center of the screen while still appearing to be random. (I left the Y coordinate as it was, because a window with its top edge offscreen cannot be interacted with at all.)

I assumed that a standard normal distribution tends to look random in a way that is satisfactory to the eye, so after doing some research I decided to approximate a normal distribution with an Irwin-Hall distribution of degree twelve. With some tweaking of the values output from the distribution with random numbers in the range 0-255 as input (generated as three 32-bit integers and split into twelve bytes), I generated an approximation of a normal distribution centered at 208:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


static int genXCoord() {
 // This uses an Irwin-Hall distribution approximating a normal distribution in
 // range 0-512 to generate coordinates with approximate normal distribution
 // between -96 and 416 which is a pretty good range for window X coordinates.
 int sum = 0;

 for (auto i = 0; i < 3; i++) {
 uint32_t x = random();
 sum += x & 0xff;
 sum += (x >> 8) & 0xff;
 sum += (x >> 16) & 0xff;
 sum += (x >> 24) & 0xff;
 }
 return (sum / 6) - 96;
}

A division operation snuck in here, which at a glance seems like an odd choice given the effort I went through earlier to avoid any division. In this case I decided it was okay, because:

This function is called only when a new window is created, where it’s plausible that a computer might “lag” a little bit because it’s busy rendering the window.
Because the division is by a constant integer, it’s possible (and easy) to convert that to a multiplication and bit-shift instead. I don’t recall checking if the compiler I used to build WEB1999 actually did such a transformation (though Clang does support doing so), but recognizing that it could is good enough for me.

Selecting window kinds

A final interesting piece of WEB1999’s implementation is how it selects which kind of window to display when a new one is to be created. I wanted there to be as many different pop-ups as I had time and memory available to make, but also never wanted the same pop-up to be displayed more than once on the screen at any given time: every visible window should be unique.

I achieved this by indirection in the window manager and dynamic dispatch. The WindowManager owns an array of Window* which are the windows being displayed onscreen, or nullptr if the window that goes in a slot is not open. Each slot in this array is assigned to a particular kind of window, which is a subclass of Window. Those slots are assigned manually in the source code, something like this (where in the actual implementation, use of a macro avoids repeating most of the code for each case):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


static Window *instances[10];
int x; // Random number representing a valid index in instances

int created_type;
switch(x) {
createWrap:
 case 0:
 if (instances[0] == nullptr) {
 instances[0] = new HotSingles();
 created_type = 0;
 break;
 }
 // Fall through to try the next kind if there's already a
 // HotSingles instance.
 case 1:
 // Try to create a DragonballFanpage in the same way,
 // in instances[1].
 case 2:
 // ...

 // After 10 window kinds, wrap back around
 goto createWrap;
}

Note that if a window of the randomly-chosen type already exists, this function will attempt each of the others in sequence until one that doesn’t yet exist is created. That requires a check beforehand to ensure that there is at least one available slot in instances to avoid an infinite loop, which I’ve omitted from the above sample code. The use of a goto statement at all is also rather unconventional, but I found it to be a useful shortcut that allowed me to guarantee that this function would always complete in a reasonable amount of time. It would be more random to continuously generate random numbers until an empty instances slot were found, but then in an unluckly case the computer could spend a long time searching for a free slot.

The created_type value, not yet discussed, is important to how the window manager tracks the order of windows (which one is topmost). Because the order of instances is determined by the type of each of its members, a level of indirection is required to refer to each instance in the on-screen order. This is where created_type comes in, as well as a second array of the same size as instances:

1
2
3
4
5
6


static uint8_t order[10];

// Move all entries in order down
memmove(order + 1, order, sizeof(*order) * 9);
// Place the newly-created window on top
order[0] = created_type;

Then in order to iterate over windows in their onscreen order, code walks though order and uses those values to index into instances:

1
2
3
4
5


// Iterate through windows, topmost first
for (int i = 0; i < slots_in_use; i++) {
 Window *w = instances[order[i]];
 // Do something with w
}

Although somewhat unusual in structure (similar to Duff’s device but less syntatically confusing), I think this approach nicely avoids creating multiples of any given window type while also avoiding any need for more complex data structures, depending on randomness to bound execution time, or wasting extra memory on pointers.

One mildly annoying limitation to this solution is that because instances of each window type are downcast to Window pointers and get freed (with the delete operator) when closed, Window must have a virtual destructor in order to avoid undefined behavior when the instances are deleted even though no further action is required in the destructor:

1
2
3
4
5
6
7
8
9


class Window {
public:
 /*
 * Window instances are meant to be POD, but delete on a superclass pointer for an
 * instance of a subclass is undefined behavior even for POD types unless there is
 * a virtual destructor.
 */
 virtual ~Window() = default;
}

How I wrote the code isn’t the only thing I find worthy of comment in WEB1999, though. I also have thoughts to share on how I designed the windows that it can display and why they look the way they do!

Design notes

I included many of my design notes in the documentation that comes along with the program, but in the interest of easy reading I’ll reproduce much of that here alongside images that are much easier to study than they are when the program is in motion.

Window styles and color

With an idea in mind to build a “kill the pop-ups”-style screensaver, the first code I had to write was something that could display a window which I could then add interesting things to. Since I had retro PCs in mind, the window decorations and background color are based on the default styles of Windows 95, 98, and Me; grey borders with an outset/inset shading effect, and a blue titlebar (changing to grey when a window is inactive).

The general concept for displaying windows.

I also limited myself to approximately the web-safe color palette with only 216 colors available, although the conscious choice to limit the color palette to that one came a bit later in the process. The CE calculators have displays capable of doing 16-bit color (5-6-5 RGB), but programmers usually run it in an 8-bit palettized mode instead (where each byte refers to one of 256 colors) because it’s considerably faster on the calculator’s relatively slow Z80-derived processor that also lacks any purpose-built hardware to make it more efficient to display images.

As I was developing graphics to display in the program, I had some difficulty dealing with the limited 256-color palette and ensuring that my graphics looked like I intended. Once I realized I could lean into the theme with a web-safe palette and use existing tools (the GNU Image Manipulation Program) to make my images look the same on my computer and a calculator, it was an easy choice! GIMP offers a web-safe palette as an option “out of the box” alongside a choice of several dithering methods, so it was rather fun and easy to develop graphics once I committed to that approach.

Once I nailed down the general look and feel, the ideas I had for what to put in each kind of window fell into several categories.

Malware and malvertising

Malvertising lives in 2024, much as it did in 1999.

These windows are inspired by malware and malvertising (advertisements designed to take advantage of people who interact them, often by roping them into scams) that feel to me like they were common around 1999. It turns out that LimeWire as appears on the left is a little bit anachronistic because LimeWire didn’t reach its peak until 2001, but malware on peer-to-peer networks is a perennial concern for their users. The other two advertisements here are much more generic and would be at home on a web page today with some style adjustments.

Pornographic (or nearly-so) advertisements are not uncommon on web sites peddling illegal (or questionably-legal) goods, so the “Hot singles” window alludes to that concept without compromising the all-ages friendliness of my program. As a bonus, using the “broken image” icon as appeared in Netscape Navigator (in place of an image that would not be appropriate for all viewers) nicely captures the feeling of personal computing prior to the year 2000. I was also quite pleased with the idea of text that looks like it’s on fire, which feels strongly reminiscent of WordArt (which I feel has become much less used since the 90s).

Finally, the image of a sports car combined with a dubious claim of being the millionth visitor to a web page is an easy way for baddies to collect information from gullible targets. Dangling an apparent prize in front of a user raises plenty of questions to somebody who takes a moment to think about it (Why is a prize being given to the millionth visitor specifically?), but those who get excited and only think carefully later could find their personal information in the hands of n’er-do-wells before they realize!

Memes

Internet users still enjoy memes today, though these may not be immediately recognized by younger viewers.

Meme culture remains strong today, and while brainstorming ideas for things to include in WEB1999 I recalled several “classic” memes that worked out nicely. “All your base are belong to us” seems comfortably classified as a classic that many people will still recognize today, and in browsing a very old collection of memes I had stored I was reminded of the “demotivator” genre inspired by Despair’s parodies of motivational posters; “Mistakes” pictured above was one of the earliest examples of their work I found in old versions of their web site on the Internet Archive’s Wayback Machine that I thought would still look okay when reduced to 256 colors and greatly reduced in resolution.

Finally, although I might not classify “Strong Bad Sings” as a meme, it comes from Homestar Runner which I get the impression became popular with many of the same people who would have been plugged into meme culture around the turn of the millennium.

Mass media

The Star Wars prequels were just getting started at the end of the 90s, and young people watching cartoons were very into Dragonball Z.

In developing ideas for windows, an early one I hit upon was the unprofessional style of pages that might be found on GeoCities. Thanks to the efforts of the Archive Team, I was able to easily browse semi-random pages from GeoCities, and one that I found had exactly the title I used here: “Dragonball Oasis.” To capture the feeling of a web page designed by a young fan of anime an excess of enthusiasm but little concern for easy readability, I thought it would be fun to make some of the text be rainbow-colored with the color changing for every letter. This page is also part of a webring, which is a concept that has all but completely died since the early 2000s.

Later on, I realized that Star Wars Episode I was released in 1999, so was perfectly contemporary with the intended timeframe this program represents. By browsing old versions of the official Star Wars web page, I was inspired to ask viewers what kind of droid they are best represented by, and took some liberty to combine that idea with the sketchy concept of a “free personality test” which I feel has a proud and dubious history on the Web.

Advertisements just like these were a common sight in the late 90s.

Taking a step back from the malicious and scammy elements I took inspiration from, I also borrowed ideas from a few of the most well-known 90s advertising campaigns. AOL became notorious for sending out enormous quantities of CDs to a public who grew increasingly hostile (or perhaps indifferent) to AOL, so an advertisement like this that advertises many hours of free service would have been a common sight. “Got Milk” was another campaign that appeared across many forms of media; there were few people in the United States who were never exposed to those ads, so this ought to be familiar to anybody of at least a certain age.

Bytecoded graphics and future work

At this point, discussion has reached the limit of what I implemented in time to submit WEB1999 to the contest I wrote it for. After that however, I had more ideas that seemed interesting to play with even if they weren’t going to be included in my contest submission. This section discusses those ideas, which involve a variety of technical decisions that I think are worthy of discussion.

In the version of WEB1999 I submitted to the contest, there were 11 different windows. I might have added more, but the program was nearing the limit for reasonable size. On CE calculators the tooling supports compressing programs so they can be larger than 64 kilobytes, but even a compressed program is limited by the available user RAM once decompressed (up to 154 KB) and I found the program was nearing that limit.

Having designed those 11 window kinds, I had a pretty good idea of what sorts of graphical operations are useful for this application. Since it seemed interesting to try to allow windows to be provided separately from the program itself, I set out to define windows as a simple domain-specific language rather than as C++ classes that must be compiled in to the program.

That approach could allow window definitions to be stored independently from the main program and act more like plug-ins than an integral part of WEB1999, so it would be easy for users to pick and choose which windows they want to see as well as create their own.

Bytecodes

Knowing what operations are useful, I came up with the following bytecode instructions that the program should be able to interpret. A window would be described by its size, text to be shown in the titlebar, and a series of bytecode instructions. Each instruction includes up to four operands that specify things like a position in the window’s content area or colors to use:

Fill(c): fills the entire window content area with a specified color.
Color(c): sets the color used by all following shape drawing commands, until a new color is set.
Rect(x, y, w, h): draw an unfilled rectangle with the top left corner at the specified x and y coordinates, with specified width and height.
FillRect(x, y, w, h): same as Rect() but fills the rectangle with the same color as its outline.
Horiz(x, y, w): draw a horizontal line start at specified (x,y) coordinates, with width w.
Vert(x, y, h): same as Horiz() but draws a vertical line with height h.
Circle(x, y, r): draw an unfilled circle centered at (x,y) with radius r.
TextFG(c): set the text foreground color to c.
TextColors(f, b): set the test foreground color to f and background color to b.
StrXY(x, y, s): display the string s at (x,y).
Str(s): display the string s immediately to the right of the last displayed text.
Sprite(x, y, i): display the image i at (x,y)
SpriteRLET(x, y, i): display the image i (with transparency and RLE compression) at (x,y)

Clearly some of these commands partially duplicate the functionality of others, but I found it useful to have simplified versions of some commands like setting the text color (where I often wanted to change the foreground color but not the background color), or encode data differently (RLET sprites are faster to display and require less memory to store, provided a sprite has moderately large areas of contiguous transparency).

The encoding of bytecode that the program consumes is as a stream of single-byte opcodes (basically the names of each of the supported functions), each followed by its parameters encoded in a way defined by the opcode. The Fill and Color operations for example are followed by a single byte specifying the palette index to use, whereas Sprite is followed by a 16-bit X coordinate, 8-bit Y coordinate, 8-bit image width and height, then width×height bytes of image data.

Each of these bytecode functions translates pretty directly to graphics functions as they exist in the calculator libraries, so that aspect of the implementation is mostly uninteresting. But to simplify writing windows in terms of the bytecode I spent some effort building tools, and those are also worth discussing.

Compilation and preview

I quickly found while translating some of the window types to bytecode that it was difficult to manually write the bytecodes. Even with the assistance of some macros, my initial approach of writing the bytecode as byte arrays in the program’s C++ source clearly wouldn’t scale to creating more windows with ease. The “Hot Singles” window was the first one I converted, and it looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


extern const uint8_t HotSingles_descriptor[] = WINDOW_BYTECODE(
 180, 126, "Singles near you!",
 FILL(C_WHITE),
 RECT(C_BORDER_TOP, 0, 0, 96, 64),
 SPRITE(broken_image, (96 / 2) - (broken_image_width / 2),
 (64 / 2) - (broken_image_height / 2)),
 SPRITE_RLET(hotsingles_left, 5, 66),
 SPRITE_RLET(hotsingles_right, 63, 66),
 STR_XY("Waiting to meet today!", 10, 94),
 TEXT_COLOR(C_GREEN),
 STR_XY("CHAT", 118, 22),
 STR_XY("NOW!", 120, 30)
);

Although not too difficult to read, I needed to edit other project files to import image files in a format suitable for embedding in a program (variables like broken_image and hotsingles_left refer to image data) and although it’s not difficult to understand how the window is meant to look from reading this code, making changes while developing it is tedious because that required rebuilding and relaunching the program.²

To improve on those things, I chose to define a domain-specific language that expresses the bytecode operations nicely and integrate that with a way to generate a live preview of the window definition code so I could get a WYSIWYG experience.

My first approach to a domain-specific language was building a small Python library that allowed me to write a window definition as a sequence of Python function calls. Something like this:

1
2
3
4


from w9bcc import *

FillWindow(C_WHITE)
Rectangle(C_BORDER_TOP, 0, 0, 96, 64)

That was a little easier to write, but I wasn’t satisfied by the options I had to load images or conveniently specify colors while still selecting a color that actually exists in the palette used by the program. In particular, although I could choose existing libraries to load images and quantize them to the same palette as used by WEB1999 I thought it would be difficult to build a Python interpreter and those libraries into a format that would be easy to distribute so anybody could write window bytecode. In particular, my ideal was that it should be possible for anybody to write window definitions just by loading an editor in their web browser and save the resulting bytecode in a format suitable for sending to a calculator. So, instead of continuing further down the Python path I decided to make it so bytecode would be written in Lua using some WEB1999-specific libraries.

Although I had never worked with Lua before, it seemed like a good choice for this application because it’s easy to embed in larger programs and it’s not too complex as a language so it should be fairly accessible to other people. The result looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


require 'w9bcc'

local WW = 168
WindowTitle("Singles near you!")
WindowSize{width = 168, height = 105}

FillWindow('white')

SetDrawColor('silver')
DrawRectangle{x = 0, y = 0, width = 96, height = 64}
local broken_image = Sprite:load('broken_image.png')
broken_image:draw_at{x = (96 / 2) - (broken_image.width / 2),
 y = (64 / 2) - (broken_image.height / 2)}

It’s not difficult to see a similarity between this Lua and the Python sample above, but it has solutions to the difficulties I already discussed:

Colors are specified as strings, which are interpreted according to the rules of CSS. That means that where it’s convenient basic names (like white or green) can be used, or colors can be specified with their RGB components (#fdcb12) or even by hue, saturation and lightness (HSL; hsl(50 80% 40%)).
Images can be loaded as Sprite objects, which are converted to the appropriate palette when loaded. Their dimensions are exposed as properties of the Sprite object, which is useful to compute the position to draw at when an image is meant to be centered at some position.

The Lua interpreter that runs this code is embedded in a Rust program, because I needed to write code to handle image loading and color conversion. It was reasonably easy to do those tasks in Rust and expose the relevant functions to Lua as a library, with a few magic functions to emit data into the final bytecode stream (hidden behind functions like FillWindow in the above example).

In order to get WYSIWYG editing (and to satisfy the goal that everything should be usable by anybody with access to a web browser), I embedded the Rust program in a web application (which I knew would be feasible when I started using Rust for this application). It has a text editor and a canvas to display a window in (both things that are easy to do in a web page but moderately difficult to do in programs generally), and when the text changes it attempts to run the bytecode compiler. If successful, it then interprets the bytecode to display the results. Since this is a web application, you can go try it out for yourself!

The bytecode editor is a simple split view between code and a preview of the result, with status and error information also shown.

One annoyance with this simple web application is in how images need to be loaded. Because the app does not have access to files in general, they need to be manually loaded via a file picker before they can be loaded with Sprite:load. I think this could be improved with the web file system API, but because my bytecode compiler can also be built as a regular program and run on the command line it’s not annoying enough for me to have investigated: it’s reasonably easy to load images into the webapp for testing, then save a copy of the images and final bytecode source as the “final” version which can be recompiled at any time without using the web app.

Interpreting the bytecode here reuses the C++ decoder that I built for the actual program (here called w99bci), which is compiled separately from the Rust bytecode compiler (w99bcc). A small Javascript driver program takes bytecode from the compiler and passes it to the renderer which is exposed to Javascript using Emscripten’s binding generation tools. I had to write versions of the calculator graphics functions that worked in terms of a 2d canvas rendering context, but that wasn’t terribly difficult. Displaying text in the same way it would appear on a calculator was most complex, because I needed to embed a copy of the calculator’s bitmap font and decode text into a series of images to display. The Javascript driver looks something like this:

1
2
3
4
5
6
7
8
9


// interpreter is the bytecode compiler (Rust with embedded Lua interpreter)
const interpreter = new Interpreter()
// renderer takes bytecode and displays to a canvas (C++)
const renderer = Renderer.create(document.getElementById('rendererCanvas'))

// editor refers to the text input widget
const script = editor.getValue()
const result = await interpreter.evaluate(script)
renderer.render(result.bytecode)

The C++ renderer has a small amount of code that’s emscripten-specific, but most of the class implementation is the same between the web and calculator. When constructing a renderer to be embedded in a web page, the caller is expected to pass in a handle to the DOM canvas element, palette data, and data representing the bitmaps corresponding to how characters are displayed in the calculator’s font. The palette and font data are loaded from image files in the caller in my application.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


#ifdef __EMSCRIPTEN__
Renderer::Renderer(emscripten::val canvasElement, emscripten::val paletteValues,
 emscripten::val fontBitmaps)
 : x(0),
 y(0),
 w(canvasElement["width"].as<uint16_t>()),
 h(canvasElement["height"].as<uint8_t>()) {
 using namespace std::string_literals;
 auto ctx = canvasElement.call<emscripten::val>("getContext", "2d"s);

 gfx_Begin(ctx, paletteValues, fontBitmaps);
}

void Renderer::renderJS(const emscripten::val &bytecode) {
 std::vector<uint8_t> bytes =
 emscripten::convertJSArrayToNumberVector<uint8_t>(bytecode);
 if (bytes.size() == 0 || bytes.back() != OP_EOF) {
 EM_ASM({throw new Error('Bytecode must end with OP_EOF, but did not')});
 }

 render(bytes.data());
}

EMSCRIPTEN_BINDINGS(Renderer) {
 emscripten::class_<Renderer>("Renderer")
 .constructor<emscripten::val, emscripten::val, emscripten::val>()
 .function("render", &Renderer::renderJS);
}

#else /* !__EMSCRIPTEN__ */
Renderer::Renderer(Rect contentArea)
 : x(contentArea.x),
 y(contentArea.y),
 w(contentArea.width),
 h(contentArea.height) {}
#endif

To implement graphics in terms of the library functions used on a calculator, I added more C++ code that maintains the same kind of global state that the calculator has, and depends on the application passing in handles to the requisite Javascript objects (the canvas rendering context notably):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


static std::vector<RGB> palette;
static emscripten::val ctx = emscripten::val::undefined();
static emscripten::val fontBitmaps = emscripten::val::undefined();
static RGBA textFGColor = {.opaque = true, .rgb = {0, 0, 0}};
static RGBA textBGColor = {.opaque = false, .rgb = {}};
static int textX;
static int textY;

void gfx_Begin(emscripten::val canvasContext, emscripten::val paletteRGB,
 emscripten::val fontBitmapsIn) {
 ctx = canvasContext;
 fontBitmaps = fontBitmapsIn;

 for (size_t i = 0; i < paletteRGB["length"].as<size_t>(); i++) {
 uint8_t r = paletteRGB[i][0].as<uint8_t>();
 uint8_t g = paletteRGB[i][1].as<uint8_t>();
 uint8_t b = paletteRGB[i][2].as<uint8_t>();
 palette.push_back({.r = r, .g = g, .b = b});
 }
}

void gfx_FillRectangle(int x, int y, int w, int h) {
 ctx.call<void>("fillRect", x, y, w, h);
}

Since when running on a calculator a program needs to call gfx_Begin to switch display modes anyway, it’s easy to do canvas initialization in the same function for w99bci. Functions like gfx_FillRectangle implement the actual drawing commands, using the global state set up in gfx_Begin.

Limitations

Currently the web-based editor can generate bytecode from an easy-to-write Lua program, but can’t be saved as a program that can be loaded by WEB1999 running on a calculator. The main reason for this is that I’m not very satisfied with the amount of memory they would require: when window definitions are built into the main program, they automatically get compressed with the rest of the code and data and decompressed when the program is run. When loading external bytecode however, the program would either need to:

Decompress bytecode on the fly whenever a window needs to be displayed, which could be slow.
Require that external bytecode not be compressed, potentially requiring a lot of memory.
Decompress all bytecode that will be used into RAM at startup.

None of those are very satisfactory to me. I don’t think on-the-fly decompression will be fast enough, not allowing compression of external bytecode seems like it would be too large, and decompressing bytecode at startup wouldn’t solve the issue of the program currently being limited by the amount of available RAM. Possibly a hybrid solution such as allowing a window definition to be either compressed or uncompressed and decompressing them at startup if needed would be acceptable, but I haven’t spent the time to explore those ideas much.

At the time of this writing, the latest revisions to WEB1999 have converted all of the windows I created to be expressed in terms of bytecode. This is integrated into the procedure for compiling the program, where w99bcc is run as needed to translate Lua source into data that gets compiled into the program and interpreted using the w99bci component. This represents useful steps toward being able to load external bytecode as well, but since I haven’t developed a satisfactory solution to the bytecode size-related concerns it’s unable to load external bytecode at this time.

Conclusions

I had fun writing WEB1999, and it was nice to have a calculator programming challenge that caught my interest. It had been some years since I wrote any software intended to run on a calculator, and I enjoyed working within those limitations.

If you own or have access to a TI-84 Plus CE calculator, I’d encourage you to load a copy of WEB1999 onto it for on-demand entertainment. For everybody else, I hope there were some interesting aspects to its implementation that I’ve described or that the historical discussion around the windows I designed was some combination of entertaining and informative!

Oops, it took me more than a year to write this post. ↩︎
Referring to sprites by name as in broken_image also wouldn’t work for images that weren’t built into the program, because in this example they’re actually expressed as pointers to data compiled into the program. ↩︎

Video Squisher: minimal command-line processing on the web

Fri, 29 Dec 2023 02:24:54 +0000

I’ve recently been asked to reduce the size of video files with some regularity, taking in a video file and generating something with reduced file size (“squishing” the video; hence “video squisher”). This is an easy task to accomplish with Handbrake, and since the transcodes I was asked to do were consistent in their needs, I was able to set up a preset in Handbrake to make these conversions very simple.

Unfortunately, there were a few steps that weren’t as easy to automate: namely, grabbing the original video and sharing the transcoded version later. Rather than need to do anything myself for each video, I sought to make my process available for “self-service”, probably as some kind of web-based tool instead. Since the imposition of receiving a file, running it through Handbrake, and sharing the result is fairly small though, I wanted to make this tool as simple as possible.

I believe I succeeded and that the results are interesting enough to share because I discovered a few new tricks that made it easier, so in this post I will describe what I built to meet this need and how it was made.

Requirements

Since manually processing a few videos is fairly easy, whatever I might build to automate video conversion would also need to be a simple tool requiring little effort to build and run. However, it also needs to be comprehensible to an unsophisticated user so I arrived at these requirements:

There must be a single server component with no external dependencies such as a database server.
All input and output must occur through a web browser.
Video transcoding progress should be reported to the user in real time.

Given I had been using Handbrake to manually transcode videos, I expected that automation of the process would involve running Handbrake’s CLI in a subprocess and streaming its console output back to the client.

Client prototyping: enter SSE

Of the requirements I had set, the goal of reporting real-time progress seemed most challenging so I investigated that first. I fairly quickly stumbled upon a web API that I wasn’t familiar with which seemed to meet my needs: Server-sent events (SSE). This API involves a client making a single HTTP request to a server, which then responds with a stream of events in a structured data format (using content type text/event-stream). At a high level, the flow of using an EventSource in a browser is to:

Send an HTTP request to a server
Handle messages as they are sent from the server. Each message has up to four fields that the client can interpret:
- data: UTF-8 text of arbitrary length.
- event: optional application-defined string identifying what kind of event this is.
- id: optional application-defined ID for a message, allowing the client to resume from the last message it received if the connection is lost by sending a request containing this ID.
- retry: a number indicating how long the server wishes the client to wait before attempting to reconnect, if the connection is lost.
When satisfied with the received data, the client may close the connection.

Overall, SSE is similar to older technologies like HTTP long polling or the newer websockets. Notably however, SSE operates over regular HTTP connections unlike websockets (which require that the server understand the websocket protocol and how to “upgrade” an HTTP request to a websocket), and SSE inherently presents a potentially-unbounded sequence of events to the client whereas long-polling may require the client to make a new request for each message to be received.¹

In the interest of making the server implementation simple, SSE seemed like a good choice because I would only need to ensure that the server returned a valid stream of events. SSE communication is unidirectional, excepting the initial request (messages are only streamed from the server back to the client) whereas websocket messages can be sent both ways (also from the client to the server) which introduces a small challenge because it needs to be possible to send data back to the server. For the application I’m interested in of receiving a file and sending back progress then a result, sending the file contents in an initial request ought to be sufficient.

Sending file data

There’s one problem with the idea of using server-sent events: the EventSource constructor provided by web browsers for the client to open a connection doesn’t offer any way to send data alongside the original request, which I need in order to upload a file to be processed. Conveniently, others have noticed the same limitation and worked around it in the form of the fetch-event-source package which also describes the limitations of the standard EventSource, saving me some explanation:

You cannot pass in a request body: you have to encode all the information necessary to execute the request inside the URL, which is limited to 2000 characters in most browsers.

You cannot pass in custom request headers

You can only make GET requests - there is no way to specify another method.

If the connection is cut, you don’t have any control over the retry strategy: the browser will silently retry for you a few times and then stop, which is not good enough for any sort of robust application.

With fetch-event-source in hand, I prototyped a basic client allowing a user to select a file, which would then be uploaded to the server and events handled as sent back. The HTML is simple, substantially just an input field and some javascript to be run when the user confirms the selected file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


<!DOCTYPE html>
<html lang="en">
 <head>
 <title>SSE prototype</title>
 </head>
 <body>
 <form id="inputForm">
 <label>
 Select a file to process:
 <input type="file" id="fileSelect" required>
 </label>
 <input type="submit" value="Go!">
 </form>
 <script type="module" src="client.mjs"></script>
 </body>
</html>

Since I don’t care much about compatibility with old browsers that don’t support javascript modules, I was able to write “modern” javascript in client.mjs to plumb data through:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


// Use fetch-event-source, pulling from a CDN so I don't need to bother
// with serving a copy alongside my application.
import { fetchEventSource } from 'https://cdn.jsdelivr.net/npm/@microsoft/fetch-event-source@2.0.1/+esm';

async function handleFile(evt) {
 // Don't do normal form submission; this function does something else.
 evt.preventDefault();

 // Get the File selected by the user.
 const file = document.getElementById('fileSelect').files[0];

 // Open an event source, POSTing the file data to /process as a blob
 // of bytes and interpreting the response as an event stream.
 await fetchEventSource('/process', {
 method: 'POST',
 header: {
 'Content-Type': 'application/octet-stream',
 },
 body: file,
 // For this prototype, just log every message that's received
 onmessage(ev) {
 console.log(ev);
 }
 });
}

// Do the EventSource thing when the form is submitted.
document.getElementById('inputForm').onsubmit = handleFile;

Server implementation

Without a server the above client prototype is useless, so I next had to write a server application that can accept the uploaded file and send back events. Since I’m familiar with the tools available in Python’s standard library, I opted to implement it in terms of http.server.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


import http.server

BUF_COPY_SZ = 1 << 20

class Handler(http.server.SimpleHTTPRequestHandler):
 def send_event(self, data: str = None, ty: str = None):
 """
 Send an event in a stream, with specified type (unset if None)
 and data (no data if None).
 """
 if data is not None:
 # Send the event type if specified.
 if ty is not None:
 self.wfile.write(f'event: {ty}\n'.encode('utf-8'))
 # Line breaks are field separators in an event stream, so multiline
 # data must be split into multiple data: lines.
 for line in data.splitlines():
 self.wfile.write(f'data: {line}\n'.encode('utf-8'))
 # Messages are separated by blank lines
 self.wfile.write(b'\n')
 else:
 # No data sends a line with no meaning, useful to keep the
 # connection alive by sending some data.
 self.wfile.write(b':\n')
 # Ensure the whole message gets sent immediately.
 self.wfile.flush()

 def do_POST(self):
 # SimpleHTTPRequestHandler will call this to handle POST
 # requests.

 # To receive the uploaded file, we need to know how large it is.
 upload_size = self.headers.get('Content-Length')
 if upload_size is None:
 self.send_error(400, 'Missing length', 'Content-Length must be set for POSTs')
 return
 upload_size = int(upload_size)

 # Input seems okay, so start sending an event stream.
 self.send_response(200)
 self.send_header('Content-Type', 'text/event-stream')
 self.send_header('Cache-Control', 'no-store')
 self.end_headers()

 # Save the uploaded data into a temporary file that can be
 # processed later, copying it in 1MB chunks. Send back
 # 'uploadprogress' events as we go, to demonstrate sending
 # events.
 with tempfile.NamedTemporaryFile(mode='wb', dir='/var/tmp') as infile:
 for o in range(0, upload_size, BUF_COPY_SZ):
 buf = self.rfile.read(min(BUF_COPY_SZ, upload_size - o))
 infile.write(buf)
 self.send_event(f'{o / upload_size:.02}', 'uploadprogress')
 self.send_event('1', 'uploadprogress')

# When run as a script, serve HTTP on an arbitrary port.
if __name__ == '__main__':
 PORT = 9428

 httpd = http.server.HTTPServer(('', PORT), Handler)
 print("serving on port", PORT)
 httpd.serve_forever()

This server will respond to GET requests by returning the contents of a file that exists (behavior provided by SimpleHTTPRequestHandler), allowing it to serve my HTML and javascript files. Any POST request expects to receive some uploaded file data and returns an event stream by calling send_event from inside do_POST for each request.

Notably, no per-request information is saved anywhere: if the connection is interrupted there is no mechanism to resume, because managing the state of each active stream and reaping the ones that have completed would be more complex. Associating a stream directly with a connection by keeping only local state ensures resources will always be cleaned up when a connection closes.

Testing the principle

Running the server (python server.py) and loading it in a web browser, I was able to upload a file and see events come back as expected. However, when the server closed the connection after completing its work I found that the client would attempt to reconnect (as if the connection had been interrupted). This makes sense because in general it’s not possible to determine whether a given connection closure was intentional, but it is a little bit annoying.

After spending some time studying the implementation of fetch-event-source (not actually very complex!), I decided that the best way to cleanly close a stream in the browser was to throw an exception:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


class Done extends Exception {}

try {
 await fetchEventSource('foo', {
 onmessage(ev) {
 // Decide if done..
 throw new Done();
 }
 onerror(err) {
 throw err;
 },
 });
} catch (e) {
 if (e instanceof Done) {
 // Finished successfully
 } else {
 // Unexpected error
 throw e;
 }
}

There might be a slightly better way to handle this, but I wasn’t able to quickly discern it. The Promise returned by fetchEventSource() resolves successfully in some cases, so there’s probably a detail that wasn’t obvious to me.² In any case, depending on exceptions to close a connection works okay.

It’s worth realizing here that my implementation of this application is completely unable to reconnect in case of connection loss because the server is entirely stateless: it is impossible to resume a stream because any information about the transcode process will be lost when the connection to the server for a given request is closed. For personal use and limited applications I don’t mind, but a public tool might want to be more robust and pay the associated complexity costs.

Actual transcoding

Having proven the concept of streaming events, the remaining piece of the server is to run something that transcodes the received video then returns the new file and streams progress output while it’s running. Given the input file is a NamedTemporaryFile called infile, running Handbrake’s CLI and getting access to the output isn’t hard:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


with tempfile.NamedTemporaryFile(mode='rb', dir='/var/tmp') as outfile:
 args = [
 # Output mp4 with leading MOOV
 '--format', 'av_mp4', '--optimize',
 # 5 megabits per second video, two-pass encoding
 '--multi-pass', '--vb', '5000',
 '--crop-mode', 'none',
 '--encoder', 'x264', '--encoder-preset', 'slow', '--turbo',
 # Don't process audio, just copy
 '--aencoder', 'copy',
 '--input', infile.name,
 '--output', outfile.name,
 ]
 with subprocess.Popen(['HandBrakeCLI'] + args,
 stdin=subprocess.DEVNULL,
 stdout=subprocess.PIPE,
 stderr=subprocess.STDOUT,
 text=True) as proc:
 self.handle_subprocess(proc)
 assert proc.poll() is not None, "Subprocess should have exited"
 self.send_event(str(proc.returncode), 'returncode')

Here I’ve simply created a second temporary file for Handbrake to write its output to, then run it in a subprocess. A handle_subprocess method on the request handler (not yet implemented!) will be responsible for relaying output back to the client by calling send_event, and once the subprocess exits it sends back a returncode event to indicate whether the transcode process completed successfully.

Real-time output streaming

Normally to capture the output from a subprocess using Python’s subprocess module, you’d want to call communicate on the process to wait for completion and return its output as a string. This application wants to return output immediately as it arrives rather than all at once after the subprocess exits, so that’s clearly not sufficient: we instead need to send a message whenever any new output appears, and ideally send empty messages rather than waiting for a long time without printing anything to ensure the stream’s connection doesn’t time out due to inactivity.

I know that communicate must do something similar to what I want to do by collecting output as it arrives, because that method ensures processes which take input on standard input won’t get stuck by doing the same thing: waiting for output while a subprocess is waiting for input would deadlock the entire thing, so it must be able to opportunistically grab output from a subprocess and pass input in. Looking at how the Python standard library implements communicate, I found it worked similarly to how I expected it would (using something like the select() system call) and implemented something similar myself in handle_subprocess:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


def handle_subprocess(self, proc: subprocess.Popen):
 with selectors.DefaultSelector() as sel:
 # If not set to nonblocking, the read() of stdout can block
 # when we try to read more data than is present. When nonblocking,
 # it only returns whatever data is available.
 os.set_blocking(proc.stdout.fileno(), False)
 sel.register(proc.stdout, selectors.EVENT_READ)

 while True:
 # Wait up to 1 second for some data
 ready = sel.select(timeout=1)
 if ready:
 for key, events in ready:
 # If there's data available, read it.
 if key.fileobj == proc.stdout:
 data = proc.stdout.read(16384)
 # We were told there's data ready, but if we read
 # nothing that means the output has been closed
 # (the subprocess exited) and we reached the end.
 if not data:
 return
 self.send_event(data, ty='stdout')
 else:
 # select() timed out; send an empty event
 self.send_event()

This assumes the subprocess’ standard output is opened in text mode (we passed text=True to subprocess.Popen()), and thus sends chunks of text from its standard output out as messages by calling send_event.

The main tricks here are:

Setting proc.stdout to non-blocking mode so we won’t ever wait for new data to arrive.
Using read(n) to read only up to n bytes of data, rather than reading to the end of the stream.
Polling with selectors to respond immediately when data becomes available to read, or give up after a timeout.

To test this, I used curl to manually send requests to the server because that was somewhat easier than clicking several things in a web browser for every test and it directly prints out the results:

$ curl -X POST \
--data-binary @myvideo.mp4 \
--header "Content-Type: application/octet-stream" \
http://localhost:9429/squish
event: uploadprogress
data: 0.0
event: uploadprogress
data: 0.99
event: uploadprogress
data: 1
event: stdout
data: [13:23:57] Compile-time hardening features are enabled
event: stdout
data: Cannot load libnvidia-encode.so.1
event: stdout
data: [13:23:57] hb_display_init: attempting VA driver 'iHD'
...

Somewhat interestingly, I had originally expected that calling read with a parameter (to limit the number of bytes read) should prevent it from blocking, but I found that my server was emitting large chunks of data much less frequently than expected, and wasn’t generating empty (keepalive) messages at all. It turned out that the chunks were 16384 bytes (the length which was being passed to read) and the reads were actually blocking, which I fixed by calling os.set_blocking. It might be possible to make this logic a little bit simpler after that discovery, but I’ve found it to work okay.

This implementation ended up working nicely, so the remaining piece is to return the output file to the client and save it in the web browser.

Sending files back

The output from Handbrake is created as a NamedTemporaryFile, which can be read like a normal file once the Handbrake subprocess exits. In order to let the client do some progress reporting for the download, I first send a resultsize message indicating how many bytes of output there are:

1
2
3
4
5
6
7
8


outfile.seek(0, os.SEEK_END)
output_len = outfile.tell()
outfile.seek(0, os.SEEK_SET)
self.send_event(f'{output_len}', 'resultsize')

# Read chunks of the output file and send back to the client
while chunk := outfile.read(BUF_COPY_SZ):
 self.send_event(base64.b64encode(chunk).decode('ascii'), 'result')

The data in each result message is encoded with base64 because event streams only accept text, not binary data. To ensure the video data being returned is valid in an event stream, I’ve chosen to base64-encode it because that’s easy to decode in the browser.

With all the server parts implemented, I had to extend the javascript running on the client to handle all of the event types in the onmessage function. I added an element to the HTML with ID outputBox to contain the encoder’s output, which will be streamed, and added code to handle each event kind as they arrive on the stream (inside the event source’s onmessage function):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


const outputBox = document.getElementById('outputBox');
let resultData = null;
let resultDataOffset = null;

switch (ev.event) {
 case 'uploadprogress':
 // Report progress of initial upload
 break;
 case 'stdout':
 // Add the new line of data to the output box
 outputBox.append(ev.data, '\n');
 // Scroll it to the bottom so the new output is visible
 outputBox.scrollTo(0, outputBox.scrollHeight);
 break;
 case 'returncode':
 // Interpret the return code as a number
 const code = Number(ev.data);
 // Abort if it wasn't successful
 if (code !== 0) {
 throw new Error('Encode failed');
 }
 break;
 case 'resultsize':
 // Interpret data as a number, allocating storage to contain
 // all the data that will be returned. These values will be
 // build up in following 'result' messages.
 resultData = new Uint8Array(Number(ev.data));
 resultDataOffset = 0;
 break;
 case 'result':
 // base64-decode the data
 const chunk = atob(ev.data);
 // Append bytes of the data chunk to resultData (using codePointAt()
 // because the "binary" string returned by atob() is a weird kind of
 // string).
 for (let i = 0; i < chunk.length; i++) {
 resultData[resultDataOffset++] = chunk[i].codePointAt(0);
 }

 // If all the expected data arrived, save the data as if it
 // were a regular downloaded file.
 if (resultDataOffset === resultData.length) {
 // Put the bytes (resultData) into a File
 const f = new File([resultData], file.name + '.mp4', {
 type: 'video/mp4'
 });

 // Create an HTML anchor which downloads the created file when
 // clicked.
 const link = document.createElement('a')
 link.download = f.name;
 link.href = URL.createObjectURL(f);
 // Free the resultData, since the created object URL
 // now contains all the data that matters.
 resultData = null;
 resultDataOffset = null;

 // Click the generated link to trigger download, and free the
 // object URL of data after that. This needs to use setTimeout()
 // because the download won't actually start until no scripts are
 // executing, so we delay calling revokeObjectURL() until after
 // the download has actually begun so the data is still present
 // to save.
 const clickHandler = () => {
 setTimeout(() => {
 URL.revokeObjectURL(link.href);
 link.removeEventListener('click', clickHandler);
 }, 150);
 };
 link.addEventListener('click', clickHandler);
 link.click();

 // All done; close the event stream by throwing an exception
 // which we know means everything is done.
 throw new Done();
 }
}

The handling of binary data and base64-decoding here is somewhat awkward, but works well enough. For very large videos it could be rather inefficient, but at least for videos with size around 100 megabytes I found the performance to be acceptable.

In the actual code I also added a <progress> element to the HTML, which gets updated for each uploadprogress event (indicating how much data has been uploaded) and each result event (indicating what fraction of the total result has been received). I haven’t included that code here, just because it’s not important to the more interesting concepts of how the system works.

Results

The result in a browser looks like this, featuring the same file input field to choose a file, a progress bar that shows upload progress and later download progress, with all of the status output from the encoder appearing in real time beneath:

As already noted, I added a progress bar for both upload and download progress which wasn’t included in the code samples above. This could probably also be hooked up to the progress reporting that Handbrake does itself (especially in its JSON output mode), but that would require more than zero parsing of its output so I didn’t bother; it’s okay to “manually” read the status from text rather than only look at a progress bar.

The other particularly useful-seeming improvement that I’ve considered but not been interested in implementing could be to load the input video in a <video> element on the client before uploading it. The information from the browser could be used to determine its length, which could be used as an input to a simple algorithm that computes a required bitrate for the video output given a target file size. Being able to estimate the input bitrate in that way would solve the current problems where a low-bitrate input could be uselessly transcoded to a higher bitrate!

I’ve published the complete source code to this tool at https://gitlab.com/taricorp/videosquisher, which might be useful to others. Beyond that, I don’t anticipate doing any further work on this because it meets my needs. The options passed to the video encoder might be changed at some point in the future if I decide they’re worth changing, and there’s some possibility I might later add the features noted in the previous paragraph; however right now I have no plans to make further improvements.

I chose to write about this tool simply because I thought its implementation used some interesting techniques that seem worth thinking about: certainly it could be adapted to other applications where a user might run a command-line tool with a fairly fixed set of options to consume one file and generate another, so this could be a convenient base on which to build similar tools!

Today it should be possible to implement SSE in terms of other APIs available in web browsers, particularly streaming responses with the Fetch API: this blurs the lines between long-polling and SSE significantly, and makes clearer that SSE is essentially a form of long-polling using a particular data format. When server-sent events were introduced as an API around 2004 however, techniques for streaming responses in web browsers in the same way were still young and uncommon in practice. ↩︎
http.server speaks HTTP 1.0 and closes the connection when it’s done responding to a request by default, which might confuse event source clients and cause this awkwardness. Using a chunked transfer encoding and speaking HTTP 1.1 might prevent that (by making it clear when the stream ends), but is somewhat more complex to handle on the server. ↩︎

Doing what Nintendon't with the Hero's Path

Fri, 03 Nov 2023 21:00:00 +1100

The Legend of Zelda: Tears of the Kingdom is an excellent video game, which I’ve only recently (about three weeks ago, at the time of writing) played to a satisfactory conclusion; it took me around 90 hours of gameplay over nearly 150 real-time days to reach that point, representing a pretty significant amount of my free time. I previously wrote about what I liked about Breath of the Wild, its predecessor and although I found a few aspects of Tears of the Kingdom’s (henceforth “TOTK”) plot less compelling than Breath of the Wild (“BOTW”), the scale of the newer game and its overall presentation left me extremely satisfied on its completion. Between them, I’d say these most recent Zelda games are strong contenders for the title of “best videogame” with no qualifiers.

Other people have written much more eloquently about what makes TOTK such an exquisite game, so I won’t spend more words on that. My reason for writing is a thought I had on completion of the game relating the Hero’s Path feature and what I might be able to do with its underlying data beyond the scope of the game’s features.

Hero’s Path

The Hero’s Path is a feature of both BOTW and TOTK that records the player’s position in the game world at intervals and allows it to be viewed on the map, recording up to around 250 hours of gameplay. This is often a convenient feature during gameplay because it becomes easier to explore the world by being able to see areas that have or have not been visited, and especially in being able to play it back (animating the player position over the world) can be a neat tool for reminiscence.¹

The Hero’s Path displayed on my TOTK save’s map at the time I completed the game.

Since the Hero’s Path captures such a large amount of gameplay, it seems like it may be interesting to mine for data or simply view in different ways. With those concepts in mind, I set out to learn how it’s stored in the game’s save data in order to extract the data and do new things with it.

File format investigation

To even begin investigating the data format, I first needed to get some save data. Nintendo would seemingly rather you never have access to the actual data stored on a Switch and only be able to make copies on their servers to back up your saves. Fortunately for me, independent programmers building homebrew software have been at it for years now and I have a Switch that’s vulnerable to CVE-2018-6242 ("Fusée Gelée"), so it is fairly straightforward to load up JKSV on my game system and make a copy of my save game for TOTK.

First look

With data in hand, the first thing to do is look at the files that make up the save. It turns out to be structured with a few independent slots and some additional data that seems shared across every slot:

Directory	File(s)	Size
album	000_Photo.jpg	63 KB
	000_Thumb.jpg	8 KB
	001_Photo.jpg	63 KB
	Pattern continues with increasing numbers. Some photos have accompanying .figi files (039_FigureInfo.figi).
	DeathMountainHatago.jpg	61 KB
	Additional "Hatago" images follow, with names seemingly corresponding to locations in the game world.
	LinkHousePicture_1.jpg	64 KB
picturebook	Animal_Bear_A_Detail.jpg	62 KB
	Animal_Bear_A_Icon.jpg	6.5 KB
	Many other detail and icon pairs follow, in categories like Animal, Enemy, Item and Weapon.
slot_00	caption.sav	11 KB
	direct_file_save_related.sav	1 KB
	footprint.sav	600 KB
	progress.sav	2.2 MB
slot_01	same files as slot_00
slot_0*	pattern continues up to slot_05
storage	CacheStorageKey.dat	9 bytes

At only 9 bytes, it doesn’t seem like the CacheStorageKey file contains anything interesting; I haven’t investigated it at all. The other directories look more interesting.

album

The album directory is pretty clearly the in-game album, which allows the player to take photos at almost any time during gameplay; this can be identified easily simply by looking at the JPEG files. A full-resolution image (1280x720) is stored alongside a thumbnail (256x144).

000_Photo.jpg in my save is recognizable as one of the default photos created when starting a new game, meant to be a photo taken by Princess Zelda during the prologue.

Not every photo has a corresponding FigureInfo file, but I infer that the .figi files contain information about the photo’s subject and its pose at the time the photo was taken. This information must be used for the ability to make monster sculptures by speaking with Kilton in Tarrey Town, since in the game context it converts a photo of a monster back into a 3D model in the same pose. Regenerating that information only from an image would be exceptionally difficult, so I expect the .figi files contain the relevant information captured at the same time as the photo.

Monster sculptures arrayed in their designated space at Tarrey Town. From this angle it’s not obvious, but some of the poses are a bit awkward which demonstrates that each monster probably does not have curated poses and they instead take the same pose as they had at the time an image was captured.

The “Hatago” and “LinkHouse” images are presumably the images which can be seen in each stable² (filled by completing the various “A Picture for the name Stable” side quests) and in certain rooms available for the player-built “dream home”, both of which accept pictures taken with the in-game camera.

picturebook

Similar to the album, the picturebook directory appears to be the contents of the Hyrule Compendium, where after a photo is taken of an object of a particular type a few sentences of additional information can be viewed at any time alongside the original photo of that object. Cropped versions of the photos are visible when browsing the Compendium in the game, so it’s logical that each object has Detail and Icon images.

A sample compendium entry for one of the several hundred items and objects that can be viewed therein. The image at the center of the screen is used with reduced resolution as the icon, and the detailed image fills the entire screen behind the rest of the UI.

As with the album, the full image for each entry is 1280x720 pixels but the icons are slightly smaller; only 168x168 pixels.

slots

Having looked at everything else, the slot_ directories look like the real meat of a save. With up to six of them present, each corresponds to the slots available to a game from where one is always the last manual save (made by selecting the Save option in menus) whereas the other five are autosaves that the game makes periodically.

A sample of six save slots in the menu to load a game. Note that all but one are marked as autosaves.

The assignment of saves to slots doesn’t follow any obvious pattern, but it seems like the slots are probably allocated round-robin when autosaving (cycling from 0 to 5 and back to 0) while skipping the slot that has a manual save in it. Marc Robledo suggests examining caption.sav to get the meta-information for each slot which should correspond to the information shown in the loading menu.

Saving some work for me, Marc’s savegame editor understands a lot of structure of the game’s save data and its source code is available so it can act as a form of documentation. That tool only supports loading caption.sav (the aforementioned metadata) and progress.sav which seems like it contains all of the core gameplay state like the player’s location and owned items.

There is no indication that either of these files contains the Hero’s Path data, so to continue the investigation that I wanted to do, footprint.sav was the clear choice; certainly it’s reasonable to assume that the game developers would have referred to it as a collection of footprints indicating where the player has been. direct_file_save_related.sav is left with an unknown purpose, but its small size probably indicates that its contents are uninteresting to me.

footprint.sav

The first thing I looked for regarding footprint.sav was whether anybody else had already documented the format. As expected (because I had also looked to see if a tool along the lines of what I wanted to create existed before I started), I didn’t find any useful documentation on its format. I did find a question relating to the TOTK save editor where the author claimed that footprint.sav contained the Hero’s Path data but further stated that nobody had documented its format.

Recognizing that TOTK is built in very similar ways to BOTW, I also looked to see if anybody had documented the data format used for the Hero’s Path in Breath of the Wild- there’s a chance they would use exactly the same format if the developers didn’t feel any need to change it between the two games. In this respect I found that Kevin Jensen had shared an informal specification, though it wasn’t immediately useful. It turns out that BOTW creates multiple trackblock.sav files each of which covers around 8 hours of gameplay. Since TOTK clearly doesn’t split the footprint files in this way, it’s unlikely to have the same overall structure.

Diffing

With nowhere else to start, I chose to begin by seeing how each of the save slots differed in their contents. Starting with the idea that it would be useful to explore how a tool like xdelta3 would express the differences between two of the slots which probably represented a short time difference. Using an optimized tool turned out to be difficult because I wanted something that I could interact with from Python (which I was planning to do any required programming work in) and I failed to get any of the libraries I found for binary diffing in Python to work.

Fortunately, a Stack Overflow commentor noted that Python’s built-in difflib can also be used for binary diffs. Arbitrarily choosing to start with slots 0 and 1, I printed out how difflib thinks the footprint files differed:

1
2
3
4
5
6


import difflib
matcher = difflib.SequenceMatcher(
 a=open('slot_00/footprint.sav', 'rb').read(),
 b=open('slot_01/footprint.sav', 'rb').read(),
)
matcher.get_opcodes()

get_opcodes took a long time to run (several minutes or more), but emitted only four opcodes:

[('equal', 0, 76, 0, 76),
('replace', 76, 77, 76, 77),
('equal', 77, 272796, 77, 272796),
('replace', 272796, 614760, 272796, 614760)]

To make sense of this output, I the get_opcodes documentation says that these constitute a set of instructions to convert the a input into b. They’re the same (equal) for the first 76 bytes, and from bytes 77 through 272796, then seemingly differ from there to the end of the file.

Since I didn’t know exactly how these two slots correlated with each other (which one is newer, in particular), I did the same thing to compare slots 1 and 2 rather than 0 and 1:

[('equal', 0, 76, 0, 76),
('replace', 76, 77, 76, 77),
('equal', 77, 272868, 77, 272868),
('replace', 272868, 614760, 272868, 614760)]

Based on the lengths of the equal segments, it looks like slot 1 has 272868 - 272796 = 72 bytes more in it than slot 0.

With a little bit more of an idea of what happens to a save over time (it looks like data simply gets appended), I then visually inspected the first 80 bytes of each slot’s footprint file to see both what the overall structure looks like and investigate what changes at offsets 76 and 77 which changed in each pair of slots that I compared:

>>> for slot in range(0, 6):
... name = f'slot_{slot:02}'
... with open(f'{name}/footprint.sav', 'rb') as f:
... print(name, f.read(80).hex(sep=' '))
slot_00 04 03 02 01 f4 e0 47 00 58 01 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 59 68 a0 c5 01 00 00 00
6b 6f dc 37 00 00 00 00 00 00 00 00 01 00 00 00
00 00 00 00 02 00 00 00 43 c2 e8 08 0c 0a 01 00
slot_01 ...
00 00 00 00 02 00 00 00 43 c2 e8 08 1e 0a 01 00
slot_02 ...
00 00 00 00 02 00 00 00 43 c2 e8 08 69 0a 01 00
slot_03 ...
00 00 00 00 02 00 00 00 43 c2 e8 08 c7 0a 01 00
slot_04 ...
00 00 00 00 02 00 00 00 43 c2 e8 08 e5 09 01 00
slot_05 ...
00 00 00 00 02 00 00 00 43 c2 e8 08 90 0a 01 00

Not paying much attention to the other bytes, it looked like the bytes at offsets 76 and 77 were part of some kind of count that increases over time. I guessed that it could be a multibyte integer and might be a number indicating how many data points are stored, and found that these values look like an incrementing counter if treated as little-endian:

0x09e5
0x0a90
0x0a0c
0x0a1e
0x0a69
0x0ac7

Since I didn’t think this would be a 16-bit integer, I also guessed that it would be 32 bits. That would include the following bytes with value 1 and 0, yielding values like 0x000109e5.

Further noting that it looked like slot 1 had 72 bytes more of data than slot 0, the difference of the two slots’ counter values was 0x010a1e - 0x010a0c = 18. 72 bytes divided by 18 is 4, which could indicate that this counter measures a quantity of 32-bit data points. Recalling the documentation for BOTW’s trackblock data, it stored a single 32-bit datum for each player location so this seemed like a solid guess.

Trying to also understand what changes where data gets added to a slot, I printed out the data from slots 0 and 1 beginning shortly before they differ and going to where slot 1 appeared to end. This turned out to be unremarkable: the longer slot had nonzero values past offset 272796 (where the shorter one ended) while the shorter slot’s data was all zero. This supported the idea that new data simply gets appended to the end of a file and a counter increments to track where the data ends.

slot_00 b0 39 78 25 30 3d 98 25 30 3f 78 25 f0 3e 48 25
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
slot_01 b0 39 78 25 30 3d 98 25 30 3f 78 25 f0 3e 48 25
30 41 48 27 70 41 58 27 30 44 28 27 f0 44 48 27
70 44 08 27 f0 44 48 27 70 3f 88 26 30 38 d8 26
70 35 d8 26 b0 35 e8 26 f0 34 38 27 30 3c 28 27
b0 3e c8 25 30 45 58 27 70 47 b8 26 70 45 48 27
f0 43 f8 26 b0 3e 38 28

With this initial look, I had enough to start trying to make sense of individual points: the 32-bit little-endian value at file offset 76 indicates how many 32-bit values are present, beginning at offset 364. Presumably the remaining 360 bytes of header are useful in some way, but are uninteresting at this time.

Point data

With an idea that the points were simply a list of 32-bit values, I then needed to start looking for patterns in each value to understand their meaning. Glancing at the values from slot 1 I had printed out to compare with slot 0, I noticed a few values that looked like they differed only slightly and were in sequence. The bytes and a possible little-endian interpretation of each:

Bytes	Little-endian value
`f0 44 48 27`	0x274844f0
`70 44 08 27`	0x27084470
`f0 44 48 27`	0x274844f0
`70 3f 88 26`	0x26883f70

The f0 44 value appears twice here, possibly indicating two points in the track at the same location or that each value is actually 64 bits wide and the intervening values capture some difference. The second value (70 44 ..) also has a very short Hamming distance from the first and third, differing in only two bits: f0 to 70 clears bit 7 and 48 to 08 clears bit 6 of the corresponding byte. The small Hamming distance suggests to me that each point probably is 32 bits of data, and any two points will tend to have small differences because the player won’t move very far between each point.

Being pretty confident that each point is stored in 4 bytes, I then needed to look for actual map coordinates in each point. Based on the data length for each slot and being able to tell that older saves will have shorter data length, I determined that in this instance my slot_02 save was the second-oldest entry in the game loading menu. I loaded that game and had a look at the map:

The map state as seen when loading my save in slot 1. Note that the current X, Y and Z coordinates are displayed in the lower right.

Also looking at the Hero’s Path in that save slot, the most recent movements were all in a fairly small area near the current location (-248, 648, -1225).

I know that at the time of this save I had been moving downward in a small area, so the somewhat circular pattern immediately south of the current location here is the most recent movements. Note that the marker for the most recent location in the path (the green person icon) is not at exactly the same location as the yellow triangle marking the current player location, though they are very close together.

Knowing that the recent footprint points stored in this save slot should be near to the position (-248, 648, -1225), my next step was to search for bit patterns in each 32-bit value that are similar to each component of those coordinates. I simplified this search somewhat by assuming that the game would store only a layer indicator (Sky, Surface, or Depths) rather than a full Z-coordinate for each point for two reasons:

Jensen’s documentation says BOTW used 13-bit sign-and-magnitude representation for X and Y coordinates, which wouldn’t fit in 32 bits if it were extended to store a similar Z coordinate.
Only an indication of which layer is relevant would be needed to display the path in the game, since TOTK only needs to dim parts of the track which are on layers not currently being viewed. In the image above, the dimmer portions of the track are from player movement on the surface or in the sky because the map is currently showing the depths and there is no way to visualize historical elevation changes.

Pattern-matching

Taking the last 32-bit value stored in this slot’s point data, the bytes are f0 3e 48 25. Assuming TOTK still used a 12-bit sign-and-magnitude format for X and Y coordinates like BOTW did, I started looking for bit patterns similar to the X coordinate; 249 (000011111001), and the Y coordinate; 601 (001001011001).

Not knowing whether each point should be interpreted as a little- or big-endian value, I started with big-endian and laid the bits out in a graphical way that should make it easier to see patterns and keeping in mind that the value in this point is probably not exactly the same as the coordinates that I know.

Y (601)?
Big-endian	0xf0								0x3e								0x48								0x25
	1	1	1	1	0	0	0	0	0	0	1	1	1	1	1	0	0	1	0	0	1	0	0	0	0	0	1	0	0	1	0	1
X (249)							0	0	0	0	1	1	1	1	1	0	0	1

In the big-endian interpretation we see an exact match for the X coordinate, but none for the Y.

Little-endian	0x25								0x48								0x3e								0xf0
	0	0	1	0	0	1	0	1	0	1	0	0	1	0	0	0	0	0	1	1	1	1	1	0	1	1	1	1	0	0	0	0
X (249)															0	0	0	0	1	1	1	1	1	0	0	1
Y (601)	0	0	1	0	0	1	0	1	1	0	0	1

Little-endian seems better; these 12-bit fields from bits 20-32 and 6-18 have values 251 and 596 which are very close to the known X and Y coordinates (offset by 2 and -5 units, respectively).

I next attempted the same interpretation an another slot that had player coordinates (-248, 648) and bytes b0 3e 38 28 to verify that it looked reasonable:

Little-endian	0x28								0x38								0x3e								0xb0
	0	0	1	0	1	0	0	0	0	0	1	1	1	0	0	0	0	0	1	1	1	1	1	0	1	0	1	1	0	0	0	0
X (248)															0	0	0	0	1	1	1	1	1	0	0	0
Y (648)	0	0	1	0	1	0	0	0	1	0	0	0

The Hamming distances in this instance are somewhat larger, but the differences in the supposed numbers are similar; it represents a shift of only (2, 5) units from the known player position.

With reasonable confidence in this basic layout of bits, I generated some more samples for myself by loading up a game and taking known movements in order to try to identify the sign of each coordinate and validate that they seem to be 12-bit values:

Start at position (-155, 1155)
Teleport to (222, 1085)
Teleport to (4632, -3712) and stay there for a while

By inspecting the values written into the Hero’s Path after doing this, I found the following 4-byte values:

0x483826ec
0x43d83788
0x43d8378c
0xe8048608, repeated multiple times

I put these four values into the same visual form to inspect them:

Bit number	31											20	19	18	17											6	5	3	2
0x483826ec	0	1	0	0	1	0	0	0	0	0	1	1	1	0	0	0	0	0	1	0	0	1	1	0	1	1	1	1	1
0x483826ec	Y = 1155												1	0	X = -155												1	1	1
0x43d83788	0	1	0	0	0	0	1	1	1	1	0	1	1	0	0	0	0	0	1	1	0	1	1	1	1	0	0	1	0
0x43d83788	Y = 1085												1	0	X = 222												0	1	0
0x43d8378c	0	1	0	0	0	0	1	1	1	1	0	1	1	0	0	0	0	0	1	1	0	1	1	1	1	0	0	1	1
0x43d8378c	Y = 1085												1	0	X = 222												0	1	1
0xe8048608	1	1	1	0	1	0	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	1	1	0	0	0	0	1	0
0xe8048608	Y = -3712												0	1	X = 4632												0	1	0

For the second and all subsequent values, I notice that bit 5 is cleared. Since only the first point has a negative X coordinate, I assume that bit 5 is set to negate the X coordinate; it is the sign bit for that value. Similarly, bit 19 changes from 1 to 0 when the Y coordinate becomes negative so that’s probably the sign of the Y coordinate although it’s odd that its meaning seems reversed from the X coordinate sign (being set to indicate a positive coordinate).

Only the first and third values have bit 2 set, but it’s also interesting that the point (222, 1085) appears twice (the second and third values) where the second appearance differs only in setting bit 2. The version of this data from BOTW seems to use one bit to indicate when the player teleports away from a location, which is probably what bit 2 is indicating; it’s a discontinuity in the track, where the next point shouldn’t be connected to this one with a line.

In the last point, the value of the 12-bit field for X coordinate is wrong, but making it 13 bits by including bit 18 as its most-significant bit yields the correct value: the X coordinate is actually 13 bits (plus sign), not 12!

In summary, the current theorized structure is:

Bits 20 through 31 are the Y coordinate.
Bit 19 is cleared if the Y coordinate is negative, or set if positive.
Bits 6 though 18 are the X coordinate.
Bit 5 is set if the X coordinate is negative, or clear if positive.
Bit s it set at the beginning of a path discontinuity such as when the player teleports.

There are only 4 bits left with unknown meaning, assuming this is all correct.

Layer indication

As noted earlier, I believed these values would store only an indication of which layer of the world the player was on rather than a whole Z coordinate. Since after looking at the X and Y coordinates in more detail there are only four unknown bits which aren’t even all contiguous, this seemed like a correct assumption.

Given there are three layers, I guessed a two-bit field would be used to store the layer and the fourth value for that field would either be unused or have special meaning. To investigate where that might be and whether such an assumption was correct, I looked at the same six points again (the two depths points from initial pattern matching and four from extended experiments) while noting what layer they corresponded to:

Layer	Value	Y coordinate	X coordinate	Unknown bits		Discontinuity
Depths	0x25483ef0	601	-249	1	0	No
Depths	0x28383eb0	633	-250	1	0	No
Surface	0x483826ec	1155	-155	0	1	Yes
Surface	0x43d83788	1085	222	0	1	No
Surface	0x43d8378c	1085	222	0	1	Yes
Surface	0xe8048608	-3712	4632	0	1	No

Since the value of bits 3 and 4 changes from 2 to 1 when moving from the Depths to Surface, it’s a safe guess that those two bits indicate which layer a given point is on.

To determine what value corresponds to the Sky, I then loaded a save and travelled into the sky, walked around some and saved the game. Inspecting the resulting data however, I didn’t see any of the expected points! There were only two more recorded locations, and both were near the last position before I went into the sky. By walking around for a bit longer and saving again, I was able to get the expected points to appear in the footprints file: it seems the game buffers footprints for a time before writing them to footprint.sav, so by walking around for a bit more time I caused it to buffer enough that the points I expected to see eventually got saved.

Looking at the points once I eventually got them, it seems the value 0 for bits 3-4 indicates the player is in the sky:

Layer	Value	Y coordinate													X coordinate														Layer value		Discontinuity	Unknown bits
Sky	0x67405dc0	-1652													375														0	0	No	0	0
Sky	0x67105ac0	-1649													363														0	0	No	0	0

This leaves it unknown what layer 3 might refer to. In BOTW one of the bits is described as indicating whether a point is “MainField or Dungeon / AocField”, but it doesn’t seem like that’s a meaningful distinction in TOTK. For one, the dungeons in the newer game are properly within the game world rather than behaving more like pocket universes as they do in Breath of the Wild, where each of the dungeons fills a space in the world but are larger inside than the space they fill and the player is prevented from getting too near to them on the overworld. Second, AocField appears to be an internal term referring to the completely separate world containing the Trial of the Sword in BOTW- TOTK has no equivalent; again, everything occurs in real space on the overworld in the newer game. For lack of any sensible options, I’ve assumed that the value 3 is unused for this field.

Before going on, I’ll summarize my understanding of the fields again:

Bit number

…

Meaning

Y-coordinate magnitude

Y-coordinate sign (negative when clear)

X-coordinate magnitude

X coordinate sign (negative when set)

0	Sky
1	Surface
2	Depths
3	Unused?

Set for discontinuity (warping away from this location)

Unknown

The final mystery bits

With a good understanding of nearly every bit of data, to decode the remaining two unknown data bits I chose to mine a save for any points that set either of them (because every sample I’ve looked at so far has both bits clear).

I wrote a little bit of code to iterate through a footprint.sav file and print out every point (binary and hex values) that set bit 0 or 1, alongside its location in the file and the coordinates represented, yielding the following list:

01001100010000000111010111000010 0x4c4075c2 47 ( 471,-1220)
00110111111100000110011011000010 0x37f066c2 86 ( 411, -895)
01011101111100000000011011000010 0x5df006c2 299 ( 27,-1503)
01101011000000001000101100000010 0x6b008b02 661 ( 556,-1712)
01010110101100001100000111000010 0x56b0c1c2 737 ( 775,-1387)
01001010011000001011100000000010 0x4a60b802 918 ( 736,-1190)
01001101111000001011101111000010 0x4de0bbc2 934 ( 751,-1246)
01001110000000001011001100000010 0x4e00b302 956 ( 716,-1248)
01001101110000001011110111000010 0x4dc0bdc2 982 ( 759,-1244)
01101001100100000110101110000010 0x69906b82 1040 ( 430,-1689)
01100011110000000110001001000010 0x63c06242 1069 ( 393,-1596)
01100100111100000100100010000010 0x64f04882 1178 ( 290,-1615)
01100001001000000111001000000010 0x61207202 1188 ( 456,-1554)
00110000111110000101010110101010 0x30f855aa 1575 ( -342, 783)
00110001001110000101010011101010 0x313854ea 1579 ( -339, 787)
01000001011000001011001001101010 0x4160b26a 2598 ( -713,-1046)
01100000000100100110000011101010 0x601260ea 2723 (-2435,-1537)
01001010011100011111011001101010 0x4a71f66a 2729 (-2009,-1191)
01001010101000011111010010101010 0x4aa1f4aa 2734 (-2002,-1194)
01000101101000011111100001101010 0x45a1f86a 2776 (-2017,-1114)
10011010100000110000100001101010 0x9a83086a 2964 (-3105,-2472)
10100101111100110101111111101010 0xa5f35fea 3071 (-3455,-2655)
11000100100000111111001111101010 0xc483f3ea 3163 (-4047,-3144)
11001111001101000101100001101010 0xcf34586a 3224 (-4449,-3315)
11000001011000111110111100101010 0xc163ef2a 3566 (-4028,-3094)
11010010010101000011101011101010 0xd2543aea 3692 (-4331,-3365)
11011111101001000111011000101010 0xdfa4762a 3741 (-4568,-3578)
11100110000001000111101101101010 0xe6047b6a 3764 (-4589,-3680)
11100110001001000111110111101010 0xe6247dea 3771 (-4599,-3682)
11011111110001000110011010101010 0xdfc466aa 4240 (-4506,-3580)
11100001101101000110100101101010 0xe1b4696a 4367 (-4517,-3611)
11100001101101000110011000101010 0xe1b4662a 4413 (-4504,-3611)
11011111001001000110101110101010 0xdf246baa 4442 (-4526,-3570)
11100010111001000110111001101010 0xe2e46e6a 4457 (-4537,-3630)
01100010010110101101110010101010 0x625adcaa 4850 (-2930, 1573)
00111000101110100011111101100010 0x38ba3f62 5987 (-2301, 907)
01000100101010101001010001110010 0x44aa9472 6084 (-2641, 1098)
00000100110100101010100011110010 0x04d2a8f2 6279 (-2723, -77)
00000001100100101010100010110010 0x0192a8b2 6283 (-2722, -25)
00000110011100101010100011110010 0x0672a8f2 6297 (-2723, -103)
00000011010100101010000110110010 0x0352a1b2 6316 (-2694, -53)
01110000100100110000011111101010 0x709307ea 6632 (-3103,-1801)
01111011010010000010100011000010 0x7b4828c2 11255 ( 163, 1972)
11001101001000010101010101000010 0xcd215542 15872 ( 1365,-3282)
01110001111110101111110101001010 0x71fafd4a 18074 ( 3061, 1823)
01101111011000100100101110000010 0x6f624b82 27029 ( 2350,-1782)

At a glance it’s easy to see that in this save bit 0 is never set so it might be unused. Bit 1 is set somewhat unpredictably, and not very often. I found the values between offset 3741 and 4457 interesting because they had fairly high density (with bit 1 being set with fairly high frequency given the number of points) and are all fairly close together, so I pulled up the map and went looking for what was in that area of the map.

The area around (-4500, -3500) on the Surface is in and around the Lightning Temple (indicated by the larger blue icon) at the far Southwest corner of the world inside the Gerudo Desert.

I thought this bit could indicate when the player is inside a dungeon, but that seemed unlikely because the Hero’s Path is recorded normally when inside dungeons and there are still gaps between the points with it set. Perhaps it instead indicates where the player died, as one of the flags in BOTW did?

The game conveniently displays an icon on the map to mark the last place the player died, and in this case I found it was near (2350, -1782) in the Sky: exactly the coordinates of the last point I found that has bit 1 set! These coordinates are the entrance to a shrine, and I found that a number of the points surrounding this one in the file (around point 27029) are exactly the same except for bit 1. I suppose that this period of the track represents time spent inside this shrine (since shrines do behave like pocket universes by being much larger within than without), so the point at index 27029 is a time when the player died inside this shrine.

Complete data format

Having located no points that set bit 0, I was confident that it was either unused or mostly unimportant so the following table summarizes the meanings of each field. The values are 32-bit little-endian integers beginning at byte offset 364 in the footprint.sav file, with the 32-bit little-endian integer at byte offset 76 indicating how many points are present.

Bit number

…

Meaning

Y-coordinate magnitude

Y-coordinate sign (negative when clear)

X-coordinate magnitude

X coordinate sign (negative when set)

0	Sky
1	Surface
2	Depths
3	Unused?

Set for discontinuity (warping away from this location)

Set for player death at this location (a different kind of discontinuity)

Unused?

To experiment with the tracks in my save game, I also developed some Python code that can load footprint.sav and collect the points, leaving them in a convenient format to inspect.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103


from dataclasses import dataclass
from enum import Enum


class Layer(Enum):
 SKY = 0
 SURFACE = 1
 DEPTHS = 2
 UNKNOWN = 3


@dataclass(frozen=True)
class Footprint:
 """A single point of the Hero's Path."""
 layer: Layer
 x: int
 y: int
 warp: bool
 death: bool
 unknown_flag: bool

 _Y_COORD_SHIFT = 20
 _Y_COORD_MASK = ((1 << 12) - 1) << _Y_COORD_SHIFT
 _Y_COORD_POSITIVE_MASK = 1 << 19

 _X_COORD_SHIFT = 6
 _X_COORD_MASK = ((1 << 13) - 1) << _X_COORD_SHIFT
 _X_COORD_NEGATIVE_MASK = 1 << 5

 _LAYER_SHIFT = 3
 _LAYER_MASK = 0b11 << _LAYER_SHIFT

 _WARP_FLAG = 1 << 2
 _DEATH_FLAG = 1 << 1
 _UNKNOWN_FLAG = 1 << 0

 @classmethod
 def from_word(cls, value: int) -> 'Footprint':
 y = (value & cls._Y_COORD_MASK) >> cls._Y_COORD_SHIFT
 if (value & cls._Y_COORD_POSITIVE_MASK) == 0:
 y = -y

 x = (value & cls._X_COORD_MASK) >> cls._X_COORD_SHIFT
 if (value & cls._X_COORD_NEGATIVE_MASK) != 0:
 x = -x

 layer = Layer((value & cls._LAYER_MASK) >> cls._LAYER_SHIFT)
 warp = (value & cls._WARP_FLAG) != 0
 death = (value & cls._DEATH_FLAG) != 0
 unknown_flag = (value & cls._UNKNOWN_FLAG) != 0

 return cls(layer, x, y, warp, death, unknown_flag)

 @staticmethod
 def _format_flag(value: bool, name: str) -> str:
 if value:
 return name
 else:
 return ' ' * len(name)

 def __str__(self):
 flags = (
 self._format_flag(self.warp, 'W')
 + self._format_flag(self.death, 'D')
 + self._format_flag(self.unknown_flag, 'U')
 )
 layer_tag = {
 Layer.SKY: 'Sky',
 Layer.SURFACE: 'Sur',
 Layer.DEPTHS: 'Dep',
 }.get(self.layer, 'Unk')
 return f'{layer_tag:3}({self.x:5},{self.y:5}){flags}'


@dataclass(frozen=True)
class FootprintSav:
 footprints: tuple[int]

 @classmethod
 def from_file(cls, path):
 with open(path, 'rb') as f:
 f.seek(76)
 count = int.from_bytes(f.read(4), byteorder='little')

 f.seek(364)
 points = []
 for _ in range(count):
 points.append(int.from_bytes(f.read(4), byteorder='little'))
 return cls(points)

 def __repr__(self):
 return f'<{type(self).__name__}; {len(self)} points>'

 def __len__(self):
 return len(self.footprints)

 def __getitem__(self, idx):
 if isinstance(idx, slice):
 return tuple(Footprint.from_word(w) for w in self.footprints[idx])
 return Footprint.from_word(self.footprints[idx])

 def __iter__(self):
 return (Footprint.from_word(p) for p in self.footprints)

This can be used for quick inspection of data or as a building block for additional data analysis, for instance to show the locations of every death in a save:

1
2
3
4
5


MY_NICE_SAVE = FootprintSav.from_file('slot_05/footprint.sav')
MY_NICE_DEATHS = tuple(p for p in MY_NICE_SAVE if p.death)
print('Found', len(MY_NICE_DEATHS), 'deaths')
for p in MY_NICE_DEATHS:
 print(p)

Running this on my save reveals that I died 67 times before completing the game, and shows the locations:

Found 67 deaths
Sky( 285,-1622) D
Sky( 473,-1598) D
Sky( 690,-1429) D
Sky( 748,-1416) D
Sky( 824,-1457) D
Sky( 833,-1451) D
Sky( 820,-1364) D
Sur( 392,-1087) D
Sur( 649, 873) D
Sur( 651, 873) D
...

Mapping the path

Having figured out the data format to my satisfaction, I needed to be able to visualize the data to really be satisfied with the results. Fortunately, others had already done much of the work in making map data available in a way that I can reuse it easily. A person going by the handle Slluxx published a browser-based game map several days before the official release date of Tears of the Kingdom, which was later superceded by a very similar map on the Zelda Dungeon web site.

The Zelda Dungeon (ZD) map is more complete at this point, so I wanted to use it as a reference- fortunately its source code is available on GitHub even though I had to guess that its source was available and where rather than having it easily-discovered from the map on its own. I found by inspecting the sources that the ZD map uses the Leaflet library to display maps in the browser and that the tiles³ making up the map exist in their repository on GitHub.

Since I had been doing my investigation in a Jupyter notebook, it was convenient that the ipyleaflet library allows Leaflet maps to be embedded in a notebook. I was able to write a little bit of code to display a zoomable map of Hyrule with only a little bit of reading documentation and looking at the tile images in the ZD repository (which I found only provide zoom levels 0 through 6 and use an unusual tile size of 564 pixels square):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


from ipyleaflet import leaflet, LayersControl, Map

def zeldadungeon_layer(name):
 url = f'https://raw.githubusercontent.com/zeldadungeon/maps/develop/public/totk/tiles/{name}/{{z}}/{{x}}_{{y}}.jpg'
 return leaflet.TileLayer(
 url=url,
 max_zoom=6,
 tile_size=564,
 no_wrap=True,
 name=name.title(),
 base=True,
)

def totk_map():
 map = Map(
 layers=[
 zeldadungeon_layer('sky'),
 zeldadungeon_layer('depths'),
 zeldadungeon_layer('surface'),
 ],
 crs=leaflet.projections.Simple,
 zoom=2,
 center=(-564/2, 564/2)
 )

 map.add_control(LayersControl())
 return map

totk_map()

Since this was intended only as a prototype, I chose to directly access the tile images from the ZD map repository on GitHub. I would make a copy and serve them myself for a real application, but it was very convenient to use somebody else’s GitHub repository as a tile source while prototyping.

The basic interactive map as displayed in my notebook, using tiles from Zelda Dungeon and capable of displaying any of the three layers.

Coordinate transformations

In order to plot points on the unadorned map that I was now able to display, I also needed to figure out how to translate from game coordinates (from around -5000 to 5000 on both axes) to map coordinates. Mapping tools like Leaflet are usually used to display maps of the Earth or at least other spherical bodies and consequently usually take coordinates as pairs of latitude and longitude, but the game world is much simpler because it’s a flat⁴ rectangle.

This kind of application is not unheard of however, so Leaflet provides a Simple coordinate reference system (CRS) that doesn’t do any of the clever spherical geometry required for handling maps of the Earth exemplified by CRSes like WGS 84 and instead one unit on either axis is mapped to one pixel at zoom level zero. This means that the coordinates on the map I’ve created are from 0 to 564 on both axes.

Although I could have located some points in the game with known coordinates and manually found their corresponding image coordinates on the map tiles, that would have been somewhat tedious and error-prone work. I instead looked at the Zelda Dungeon map application’s source code again and found that (after some computations) the transformation from game coordinates to tile coordinates (recalling that tiles are 564 pixels square) is best done by multiplying the game coordinate by 0.046875 and adding 282. Of if you like math notation, the following function $M(p)$ converts a game coordinate $p$ to a map coordinate $M(p)$:

$$ M(p) = \frac{564}{2} + \left( \frac{564}{12032} \times p\right) $$

The magic number 564 is the base tile size, and 12032 is a scale factor defined by the total size of the tiles in relation to the game coordinates. Any application using different tiles might have a different CRS, but it’s nice that I was able to reuse the ZD CRS alongside the map tiles.

I wrote a little bit more code to do this coordinate transformation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


from collections import namedtuple

class Transformation(namedtuple('Transformation', ('a', 'b', 'c', 'd'))):
 def transform(self, x: float, y: float, scale=1.0) -> tuple[float, float]:
 return (
 scale * (self.c * y + self.d),
 scale * (self.a * x + self.b),
 )

scale = 564 / 12032
offset = 564 / 2
MAP_TRANSFORM = Transformation(scale, offset, scale, -offset)

..and then to test the transformation, I plotted from the beginning of my save’s Hero’s Path until the first time I teleported.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


from ipyleaflet import AntPath

map = totk_map()

points = []
for p in MY_NICE_SAVE:
 points.append(MAP_TRANSFORM.transform(p.x, p.y))
 if p.warp:
 break

map.add_layer(AntPath(locations=points))
map

I knew that I had spent time circling the Great Sky Island during the tutorial sections of the game, so that this track follows logical borders in that area is an encouraging sign that the CRS is correct.

The first couple attempts at this didn’t have a correct CRS; it was either shifted from the correct location (in a few instances far outside the actual map bounds), flipped or rotated. I experimented by switching the x and y coordinate orders in a few places (Leaflet takes Y coordinates first by convention⁵) and changing the sign of some of the factors (scale, offset) until I ended up with the above code and image which looks correct.

Plotting deaths and life

That made for all the information and code I needed to make use of footprint data, so as another experiment I plotted every location I died on the map.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


from ipyleaflet import Marker

map = totk_map()

for p in MY_NICE_SAVE:
 if p.death:
 m = Marker(location=MAP_TRANSFORM.transform(p.x, p.y),
 title=f'{p} {MAP_TRANSFORM.transform(p.x, p.y)}', draggable=False)
 map.add(m)

map

It’s fairly clear for looking at this map that my deaths tended to be clustered, rather than scattered; I must have a tendency to treat a death in the game as a challenge and try again until I find a winning strategy, rather than avoiding the area in the name of safety.

This plot is somewhat misleading however because it always displays every point, regardless of which layer is shown behind them. It would certainly be possible to automatically show only the points corresponding to the visible layer, but it may not be possible with ipyleaflet.

I also had a go at drawing the same kind of line that the game displays for the Hero’s Path. As with deaths, this displays activity on every layer in the same way so it’s not possible to tell which layer of the world I was on at any given point; that would be a worthwhile improvement that I haven’t tried to make.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


from itertools import pairwise
from ipyleaflet import Polyline

map = totk_map()

chunks = []
chunk = []
for p1, p2 in pairwise(MY_NICE_SAVE):
 if p1.warp or p1.death:
 chunks.append(chunk)
 chunk = []
 continue

 chunk.append(MAP_TRANSFORM.transform(p1.x, p1.y))
 chunk.append(MAP_TRANSFORM.transform(p2.x, p2.y))

if chunk:
 chunks.append(chunk)

map.add(Polyline(locations=chunks, name="Hero's Path",
 color='rgba(0, 255, 255, 0.5)', weight=3, fill=False))
map

Even with no indication of which layer a given point is on, this track provides a decent sense of where I spent more or less time and illustrates how geography guides the way players move through the world.

This proved that I could do what I originally wanted to do, and it’s where I’m leaving this write-up.

Conclusions

I was somewhat surprised by how easy it was to learn the aspects of the footprint.sav file format that I cared about, though it was significantly simplified by the existence of documentation for BOTW’s similar data. I hope my notes on the process are useful even to people who aren’t interested in the Hero’s Path in particular, since it seems like when others do this kind of work they tend to only share the results and nothing of the process. The result in those cases tends to be that the process of reverse-engineering seems impossibly difficult to a beginner, but I hope that the description of my process (which I developed in an ad-hoc way, never having tried to do this kind of thing before) lifts the veil on at least one way to approach this kind of challenge.

In documenting this format I’ve filled in a knowledge gap that at least one other person wanted to fill, and hopefully have enabled others to do interesting things with the data in the future. Although I would like to create some tooling that allows others to visualize their own data with ease, I found that the appeal of doing so had been lost after I prototyped enough to show that it was possible. Perhaps I’ll revisit that tooling in the future, but right now I’m more interested in doing other things.

Since it might be interesting to view, I’m sharing the Jupyter notebook that I was working in when doing the work described in this post. It’s formatted differently from the narrative version here and is probably harder to read, but does offer interactive maps and complete code: TOTK Save Investigation.ipynb.

Extended ideas

It would be neat to visualize where real players have died most often by collecting a bunch of saves and generating a heatmap- I recall the time somebody with access to player data for Just Cause 2 mapped 11 million player deaths with interesting results, and although the Hero’s Path doesn’t permit quite the same level of detail I believe the results could be interesting.

It seems like Nintendo’s development teams have found that it’s nice to capture moments automatically for the player, since this year’s Super Mario Bros. Wonder automatically captures screenshots of gameplay that seem like they have similar attraction. ↩︎
“Hatago” is a Japanese word referring to inns located along national highways in the Edo period, so it makes sense that this word would be used by the developers to refer to the game’s stables that function as a kind of inn. ↩︎
For readers unfamiliar with how map-viewing applications are often implemented, they tend to provide “tiles” (pictures) of map imagery on demand, each of which is fairly small (often 256 pixels square) and has an associated “zoom level”; at any given zoom level a tile can be retrieved that covers any chosen point on the map. This approach is in part a concession to efficiency because a large map represented as a single huge image would often be too large for anybody to view with acceptable performance, and it allows lower zoom levels (more zoomed out) to omit small details that might otherwise make a map difficult to read while still making them visible at higher zoom levels. ↩︎
“Flat” meaning it’s a rectangle that exists in a purely two-dimensional space rather than being projected onto the surface of a sphere (or a shape that approximates a sphere), as real-world maps typically are. ↩︎
Mapping software generally doesn’t agree on whether (x,y) or (latitude,longitude) pairs are the more correct way to express coordinates. Humans looking at maps usually talk about positions with latitude and longitude in that order, but computer graphics usually uses Cartesian (x,y) coordinates instead (and even then with no particular consistency about whether Y=0 is at the top or bottom of the screen). There’s no sensible way to split that difference, so different libraries often make different choices about the order in which X and Y coordinates need to be given. ↩︎

Moving a Linux system's root without rebooting

Fri, 08 Sep 2023 01:00:00 +0000

On a long-lived Linux server I operate, it has periodically been desirable to move its root filesystem (containing core OS files and the like) to a new drive for various reasons. Over the years it’s migrated from a conveniently inexpensive mechanical hard drive to a series of increasingly capable and larger solid-state drives (SSDs), first connected over SATA and later in the form of M.2 NVMe cards. I recently sought to do another upgrade, swapping the current NVMe drive for a larger one. Making things more interesting, I’ve developed strategies to do this without rebooting except to physically plug or unplug devices!

Traditionally, migrating a system to a new root disk like this either involves reinstalling completely and restoring data afterwards, or booting from some kind of rescue media (a “Live CD”) and manually copying everything from the old to the new disk. Both of these require some system downtime and the former makes it easy to unintentionally leave behind important aspects of system configuration, so I’ve usually done the latter.

During an earlier migration of this same system, I realized that Linux’s Logical Volume Manager (LVM) could make these kinds of migration easier. Primarily LVM makes it easier to resize disk partitions after moving them to a new disk, allowing me to take advantage of progressively larger disks over time; but it also turns out that LVM’s layers of abstraction make it possible to move filesystems (including the root) without even turning the computer off, doing a kind of “switcheroo” that minimizes downtime.

Depending on the hardware configuration, this approach allows disk migration without any downtime, provided the storage devices involved can be hotplugged or will both end up being connected! Since it’s been a while since I last did this switcheroo and there have been cases since the last time where these techniques were relevant in conversation with other professional computer people, this time I’ve chosen to write up the process so it can both serve as notes for me the next time I want to do it, and to demonstrate it to others.

Setup

Before diving into the details, I should recap what the system configuration looks like. This is a fairly recent consumer-grade PC based around an Intel processor, with the OS installed on a M.2 NVMe SSD installed in a slot on the mainboard. The new drive I want to move to is another M.2 SSD but with greater capacity.

Because the mainboard does not have any spare M.2 slots where I could install the new SSD, I’ve used a USB enclosure to connect the new drive over USB, with expectation that after the migration is complete I can incur small downtime to take out the old SSD and install the new one into the slot on the mainboard. Use of a USB enclosure for the new drive probably causes some performance degradation since running everything over USB will tend to be somewhat slower, but that temporary loss of performance is unimportant to me.

Discovering the current state

Although I know this sytem uses LVM, it’s been a while since I configured it so many of the details are fuzzy. Before beginning, I needed to investigate what the current configuration was in order to understand what I should do to perform the switcheroo.

First, I used lsblk to understand what storage devices were connected and in use (with irrelevant ones omitted here):

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdf 8:80 0 953.9G 0 disk
nvme0n1 259:1 0 238.5G 0 disk
├─nvme0n1p1 259:2 0 512M 0 part /boot
└─nvme0n1p2 259:3 0 238G 0 part
├─myhost-root 254:0 0 40G 0 lvm /
├─myhost-home 254:1 0 60G 0 lvm /home
└─myhost-var 254:2 0 60G 0 lvm /var

Or reformatted as a table, omitting columns that aren’t interesting and translating a few items to more understandable versions:

Device name	Size	Type	Mountpoint
`sdf`	953.9G	disk
`nvme0n1`	238.5G	disk
`nvme0n1p1`	512M	partition	`/boot`
`nvme0n1p2`	238G	partition
`myhost-root`	40G	LVM LV	`/`
`myhost-home`	60G	LVM LV	`/home`
`myhost-var`	60G	LVM LV	`/var`

The disk sdf is the new drive in the USB enclosure, and nvme0n1 is the current boot disk containing the root filesystem as well as /home and /var which need to be moved. Most of the data is backed by LVM (the type column reads lvm, meaning it’s on a LVM LV), which enables this whole procedure.

There’s also a small (512MB) partition at nvme0n1p1 mounted at /boot. This will be important to preserve. Looking at the disk’s partition table in detail provides more useful information (with some things that are not relevant to this discussion removed):

# gdisk -l /dev/nvme0n1
Found valid GPT with protective MBR; using GPT.
Disk /dev/nvme0n1: 500118192 sectors, 238.5 GiB
Sector size (logical/physical): 512/512 bytes
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 500117503 238.0 GiB 8E00 Linux LVM

Knowing the exact sector size of the boot partition will be useful later (to confirm the new one is exactly the same), and this confirms that there are only two partitions on the old disk.

EFI configuration

The EFI system partition type for the boot partition indicates that it is in fact an EFI system partition (ESP), telling us a few things:

This machine uses UEFI to boot, not legacy BIOS.
It’s impossible to put this data on LVM, because the system firmware needs to be able to read the files stored therein. It’s formatted with a FAT32 filesystem, which is common for EFI systems; some bootloaders are capable of accessing data on LVM, but I doubt any UEFI firmwares out there can.
The boot sector and any data stored outside a partition does not need to be preserved, because only the data on the ESP is needed to boot. Compare to legacy BIOS, where the boot sector always contains bootloader code that must be preserved and sometimes bootloader code is stored in unpartitioned areas of the disk which must also be preserved.
The firmware may have references to partition UUIDs that tell it what to boot from, which would need to either be preserved or updated later.

To understand what the firmware’s boot configuration is, efibootmgr works well:¹

# efibootmgr
BootCurrent: 0001
Timeout: 5 seconds
BootOrder: 0001,0000
Boot0000* Arch Linux VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)72006f006f0074003d002f006400650076002f006d00610070007000650072002f006b006900720069007300680069006d0061005f006e00650077002d0072006f006f007400200072007700200069006e0069007400720064003d002f0069006e00740065006c002d00750063006f00640065002e0069006d006700200069006e0069007400720064003d002f0069006e0069007400720061006d00660073002d006c0069006e00750078002e0069006d006700
Boot0001* GRUB HD(1,GPT,abf0aa0a-b6ed-45a9-9c16-77727fef1538,0x800,0x100000)/File(\EFI\GRUB\grubx64.efi)

efibootmgr tells us here that the firmware is configured to prefer to boot GRUB (entry Boot0001, which comes first in BootOrder), and it finds the GRUB executable on HD number 1, in a partition with UUID abf0aa0a…. The other boot entry contains a lot of text, which in this output is displayed as hexadecimal bytes but I know that it’s actually meant to be text encoded with UCS-2 (similar to UTF-16: two bytes per character, with the second zero for most Latin characters).

Seeing this slight mess of text summons memories from when I first configured EFI boot on this system: I remember being unable to make the firmware directly boot the kernel, then giving up and using GRUB instead. While it seemed that the system’s firmware was willing to boot a Linux kernel image directly, it would not pass the required kernel command line options so the kernel was unable to do anything useful like mount the root filesystem. Using GRUB instead means the configuration is loaded from a file rather than any parameters passed from the firmware, sidestepping that issue.

It looks like the disk number (HD(1,… for the Boot0001 GRUB entry that is of interest here) is chosen by the firmware and perhaps correlates to a physical port on the mainboard, so there should be no special considerations in making the firmware select the correct disk to boot from. To select the correct partition to load GRUB from however, either the new partition’s UUID must be set to be the same as the old or the boot entry must be updated with a new UUID.

To confirm that this UUID refers to the UEFI system partition, I looked at /dev/disk/by-partuuid and verified that the file abf0aa0a… is a link to nvme0n1p1: the ESP. It seems plausible that there might be a way to tell the firmware to load something from the ESP directly without specifying the filesystem UUID, but I’m not sufficiently motivated to discover if that’s true right now.

Moving /boot

To move the ESP, I’ve chosen to reuse the original partition UUID so I don’t need to change firmware configuration at all. This should make it easier to switch back to the old disk if I need to.

First I used gdisk to interactively create a partition on the new disk with the same size, partition type, and UUID as the original:

# gdisk /dev/sdf
GPT fdisk (gdisk) version 1.0.9.1
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): p
Disk /dev/sdf: 2000409264 sectors, 953.9 GiB
Model: Generic
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): CE6A291B-12C9-4669-BF19-8696D81FBF4B
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 2000409230
Partitions will be aligned on 2048-sector boundaries
Total free space is 2000409197 sectors (953.9 GiB)
Number Start (sector) End (sector) Size Code Name
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-2000409230, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-2000409230, default = 2000408575) or {+-}size{KMGTP}: +512M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): EF00
Changed type of partition to 'EFI system partition'
Command (? for help): x
Expert command (? for help): c
Using 1
Enter the partition's new unique GUID ('R' to randomize): abf0aa0a-b6ed-45a9-9c16-77727fef1538
New GUID is ABF0AA0A-B6ED-45A9-9C16-77727FEF1538

After saving the new partition table and exiting gdisk, gdisk -l /dev/sdf verifies that the new partition has the same size (start and end sectors) as the old one, so I’m happy with this result.

Copying the filesystem

To copy the filesystem, I’d like to take a bit-exact copy (rather than copying the files and replicating the directory structure) to ensure there won’t be any unexpected differences between the original and copy. Since I only rarely need to write to this filesystem and certainly don’t need to right now (aside from writing the copy to the new disk), I can remount it read-only then copy the block device directly:

# mount -o remount,ro /boot
# dd if=/dev/nvme0n1p1 of=/dev/sdf1 bs=1M
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 14.6401 s, 36.7 MB/s

Making it read-only before copying ensures that its state on disk is consistent and the system won’t make any changes that could leave it in an inconsistent state while I’m making the copy. To ensure the new copy remains the same as the old one, this filesystem should remain read-only until I replace it with the new one. Otherwise changes wouldn’t be propagated to the copy, and I’d need to repeat this step later.

LVM switcheroo

With the non-LVM data moved, it’s on to the rest. To recall how LVM works (or provide a quick introduction to readers unfamiliar with it), the Linux Logical Volume Manager (LVM) offers a variety of features to abstract away the details of persistent storage devices.

Physical volumes (PVs) are created on physical storage (actual hard disks).
Multiple PVs can be combined into a volume group (VG), which acts like a storage pool made up of all of the PVs in the VG.
Logical volumes (LVs) are created on VGs and act like regular disk partitions, but given superpowers by the layers of VG and PV underneath.

By sitting between actual storage and the “partition” that typically has a filesystem on it (a LV), it’s possible to configure the LV to behave entirely unlike a real disk. A few of the interesting features include:

A “thin” LV can be larger than the underlying PV, which might be useful in a system that is actively managed and the administrator ensures there is always enough storage available: they can create a very large filesystem that doesn’t need to be resized as storage is added, without having enough storage for the entire filesystem from the outset.
“Snapshot” LVs capture changes to another LV, exactly capturing its contents at the time the snapshot was created without blocking writes to the original. This works a lot like the snapshot features of ZFS and btrfs, but isn’t tied to use of any particular filesystem.
Allocation policy of a LV with regard to the PVs in its VG can be controlled, often using the raid LV type to use any of the common RAID allocation policies to spread data across disks to improve performance, provide redundancy against disk failure, or both.

VGs and LVs are all given names, which is where the names like myhost-root seen earlier come from. myhost is the name of the VG I’m using, and root is the name of the LV holding the filesystem that gets mounted at / (the system root).

Desired configuration

After doing some reading of the LVM-related documentation (in particular lvmraid(7), I concluded that temporarily configuring my LVs as RAID1 across the two devices is a sensible approach. By also setting the activation mode to degraded, the system will allow the relevant LVs to be used as long as either PV it’s on (assuming RAID1 with two devices) is present. By doing this, I can reconfigure the LVs then wait for them to be synced, and reboot to swap in the new storage. If something goes wrong with the new one, I can connect the old one and they will still have the same contents so I can easily roll back and try again.

First, create a PV on the new disk in a new partition that fills the rest of the device (sdf2) and add it to the VG (which is named myhost):

# pvcreate /dev/sdf2
Physical volume "/dev/sdf2" successfully created.
# vgextend myhost /dev/sdf2

At this point vgdisplay shows there are two active PVs in the VG, and its size is much larger than it was previously. Now convert each of the LVs I care about to RAID1, mirrored across both PVs:

# for lv in root var home; do
for> lvconvert --type raid1 --mirrors 1 myhost/$lv
for> done
Are you sure you want to convert linear LV myhost/root to raid1 with 2 images enhancing resilience? [y/n]: y
Logical volume myhost/root successfully converted.
Are you sure you want to convert linear LV myhost/var to raid1 with 2 images enhancing resilience? [y/n]: y
Logical volume myhost/var successfully converted.
Are you sure you want to convert linear LV myhost/home to raid1 with 2 images enhancing resilience? [y/n]: y
Logical volume myhost/home successfully converted.

After conversion, the LVs need to sync: make a copy of all their data on the new PV. The lvmraid manpage says to use lvs -a -o name,sync_percent to monitor this process:

# lvs -a -o name,sync_percent
LV Cpy%Sync
root 3.71
[root_rimage_0]
[root_rimage_1]
[root_rmeta_0]
[root_rmeta_1]

Before continuing, I waited until Cpy%Sync was 100% for each of the volumes. To ensure that the system will still try to boot with only one replica present, I also verified that the current LVM configuration allowed degraded RAID LVs to be activated:

# lvmconfig --type current activation/activation_mode
activation_mode="degraded"

Since this was already set as I wanted, I didn’t need to make any further changes, and proceeded to shut down the computer, swap out the old disk and install the new one.

Unexpected misbehavior

After swapping the two disks and disconnecting the one in the USB enclosure (leaving only the new one installed in the machine), the firmware booted the Linux kernel correctly, but the system didn’t boot successfully: it complained that it couldn’t find the root filesystem. Using a rescue shell I was able to determine that all my LVs were present, but inactive (unusable until activated) and in the partial state, seemingly indicating that some important metadata was stored only on the original PV and LVM refused to allow the LVs to activate because it didn’t have all of the data needed to manage them correctly.

I suspect there may have been a way to repair this RAID issue, but I opted to fall back to the approach I had used previously rather than embark into the unknown while being stuck with a computer that’s offline until it’s fixed.

Without redundancy

Since I had already managed to break the system more by attempting to use RAID than the old approach would have, I fell back at this point to the workflow I had used previously that still allows everything to be moved without downtime other than for physically moving hardware around but makes rollback to the original state more difficult. Unfortunately because I had already broken the system, I had to do this from a rescue shell after crawling around to connect the requisite cables so I could use the rescue shell.

The plan for this was:

Convert the VG back to linear allocation, rather than raid1; keep a single copy of each LV’s data, having no particular opinion where it’s stored.
Move all data off the old PV and onto the new one.
Remove the old PV from the VG, guaranteeing that nothing will be stored on it.
(optional) Wipe the old PV so it can’t be accidentally re-added to the VG.

Each of these steps conveniently maps to a single command to run:

# lvconvert --type linear myhost/root myhost/var myhost/home
# pvmove /dev/sdf2
# vgreduce myhost /dev/sdf2
# pvremove /dev/sdf2

pvmove takes a while to complete, because it moves all data on sdf2 to other PVs in the VG. Since I moved the new drive to the mainboard slot when I was ready to try the RAID, sdf is now the old drive. Then vgreduce removes the PV from the named VG (failing if there is any data still stored on the nominated PV), and pvremove wipes the PV signature off the old disk.

After doing that, I rebooted normally and the system came up as expected this time, running entirely on the new disk. Meanwhile, the old disk has no data left on it and can be reused for other purposes.

Useful tools, if you don’t mess it up

I hope this writeup is useful to others, and expect it will be of use to myself in the future as well. LVM provides some very useful capabilities to system administrators that I believe many users are unaware of, and greater awareness is the first step towards making use of these tools.

My failed experiment with LVM RAID reinforces that when doing low-level computer maintenance it’s often best to go with a proven approach to avoid nasty surprises: this nasty surprise wasn’t too bad, but it did delay my dinner while I sorted it out. Had the RAID worked how I’d expected then it would have become another tool in my arsenal, and because this system isn’t very critical to anybody, on balance the learning experience seems to have been worthwhile. For the future however, I’d want to either do more experimentation before trying that approach again, or simply follow the well-trod path (using pvmove) for safety.

It’s usually impossible to interrogate the firmware to learn about boot configuration when using legacy BIOS; being able to do this is a useful feature of UEFI. ↩︎

Efficiently Capturing Time-Lapse video with a Raspberry Pi

Fri, 02 Sep 2022 06:42:55 +0000

Earlier this year, I had a desire to capture time-lapse video of some construction that would take an unknown amount of time and occurred mostly during short periods of activity separated by intervals of inactivity with varying time. Because the overall recording time was unknown, this represented an interesting set of challenges that I wrote some software to address, using a Raspberry Pi and a USB webcam. (See the end of this article for the complete source.)

Existing options

There is no shortage of articles around the web describing how to capture time lapse videos with a Raspberry Pi, such as Caroline Dunn’s article in Tom’s Hardware. I find that most of these involve some kind of script to take still images at intervals, and another component (usually using ffmpeg) to combine those images into a video.

Although this approach works fine, it suffers from several obvious shortcomings:

Frames stored as individual image files tend to require much more space to store them than an equivalent video file, since each frame will usually be very similar to the preceding ones: video codecs are designed to take advantage of this redundancy.
Captured images and generated videos must be manually retrieved (or deleted) on capture completion. For long captures or when generating multiple sequences, this may require regular human intervention to ensure the system is still running and storage is still available.

Since the events I wished to capture would take an unknown amount of time (possibly months), it was important to me that the system should be reliable and not require regular (manual) maintenance as well as not use more storage space than necessary. To that end, I chose to write my own set of scripts that is described by the rest of this article.

Available hardware

The hardware I had available amounted to a Raspberry Pi 2 (and a USB WiFi adapter) with 32GB microSD card as boot volume, a Logitech C925e USB webcam, and extension cords and power supply suitable to place the system wherever I needed. The Pi 2 is a rather dated and slow machine by now, but ought to be sufficient for this application.

Video capture

The Logitech C925e advertises support for 1080p video at 30 frames per second, which is a much higher framerate than I need for time-lapse video capture (around one frame per second seems fine) but a respectable resolution. Investigating the camera’s actual capabilities shows it’s actually capable of higher resolution than that:

$ ffmpeg -f v4l2 -list_formats all -i /dev/video0
[video4linux2,v4l2 @ 0x1083cd0] Raw : yuyv422 : YUYV 4:2:2 : 640x480 160x90 160x120 176x144 320x180 320x240 352x288 432x240 640x360 800x448 800x600 864x480 960x720 1024x576 1280x720 1600x896 1920x1080 2304x1296 2304x1536
[video4linux2,v4l2 @ 0x1083cd0] Compressed: mjpeg : Motion-JPEG : 640x480 160x90 160x120 176x144 320x180 320x240 352x288 432x240 640x360 800x448 800x600 864x480 960x720 1024x576 1280x720 1600x896 1920x1080

This output indicates that the camera supports output as either raw video or Motion JPEG in a variety of resolutions up to 1920x1080 pixels. Raw video can also be output at higher resolutions up to 2304x1536, but at a lower framerate (around 12 fps, it turns out). Since I didn’t care about any framerate higher than about 1 fps, it was a logical choice to run at that higher resolution.

The basic use of ffmpeg for this application uses a V4L2 device as input (the webcam) and outputs at a very low framerate to timelapse.mp4:

ffmpeg -i v4l2 -video_size 2304x1536 -i /dev/video0 \
-vf fps=0.1 -t 60 timelapse.mp4

In this instance I’ve captured video at one frame per 10 seconds (-vf fps=0.1) and chosen to capture one minute of video (-t 60). In the final script these are configurable, but this illustrates the concept nicely.

Although the required framerate for video encoding is low in this application, video encode performance remains a concern because the Raspberry Pi 2 is not a very powerful computer. I wanted to use a video codec that achieves good compression, and it needed to do so with enough speed that frames can be encoded at least as frequently as they are captured.

Some versions of the Raspberry Pi support hardware-accelerated H.264 encoding (in a quick search, it’s unclear if that includes the Pi 2), but I didn’t try to make that work: there doesn’t seem to be support for the relevant hardware in ffmpeg so I would have needed to use some other software to do the encoding, and it’s unclear what the hardware encoder’s limitations are (such as maximum resolution). I instead did some manual experimentation by doing a live video capture with assorted codecs:

H.264 (x264)
VP8 (libvpx)
H.265 (x265)
VP9 (libvpx-vp9)

VP9 and H.265 are attractive choices because they are very good at efficiently compressing video, but I found performance to be too bad to be usable for this application (achieving less than 0.1 frames per second at the target resolution). Both x264 and VP8 perform acceptably and achieve similar compression, so I opted to use VP8 since it’s not legally encumbered by any patent licensing requirements (unlike H.264).

I somewhat arbitrarily chose a maximum bitrate of 20 megabits per second and CRF of 4 to achieve a high-quality encode, and ended up with this ffmpeg invocation:

ffmpeg -i v4l2 -video_size 2304x1536 -i /dev/video0 \
-an -c:v libvpx -b:v 20M -crf 4 \
-vf fps=0.1 -t 60 timelapse.mp4

With a way to use ffmpeg to capture time-lapse video directly (no intermediate image files!), we can move to thinking about where video will be stored.

Storage considerations

As mentioned earlier, I only have a 32GB SD card at hand for this Raspberry Pi, and I don’t trust it not to corrupt data without warning (both the Pi itself and the card; I don’t really trust either with my data) so it seemed important to ensure that the Pi’s local storage would not fill up with video and prevent further recording, as well as to copy data off the Pi shortly after its creation. I chose to address this by having the system upload video to Google Cloud Storage (GCS). There’s nothing particularly special about GCS over one of the many other object storage systems available from many different service providers; it was just convenient for me to use GCS.

I didn’t want to completely clean up video periodically (by deleting old files) in case of an upload failure, and it’s useful to get more frequent feedback on how video capture is going in the form of segments that can be viewed immediately so I also wanted to have the system incrementally upload video as it is captured rather than uploading larger chunks at long intervals (say, every day). Incremental upload also helps reduce the bandwidth needs of the system, since the total data transfer is spread over a longer interval.

As discussed above, by capturing video at a low frame rate rather than individual images at the same rate we can save storage space, improve picture quality, or possibly both. Incrementally uploading video files is somewhat more challenging than handling a similar collection of still images however, since still images have a convenient 1:1 relation to the captured frames (so it’s easy to assume that a file’s existence implies a complete frame) whereas video files become larger over time as frames are added.

Incremental upload

To incrementally upload video, we need to choose a container that remains valid when more data is appended to it, then design a method to efficiently append new data to what’s already present in remote storage.

Container choice

ISO MPEG-4 containers (.mp4 files) are a common choice for video files and are supported by most software. However this container is not very well-suited to this application because by default some metadata (the MOOV atom) gets placed at the end of the file. ffmpeg can put that at the beginning of a file to make a file “streamable” by using the -movflags faststart option, but that doesn’t really solve the live capture problem because the metadata stored in the MOOV atom that the faststart option moves around needs to be derived from the entire encoded file: ffmpeg implements faststart simply by outputting a file, then moving the MOOV block to the beginning and copying the rest of the file to follow it. Since this requires the entire input be available first, it is not appropriate for a pseudo-live stream.

The Matroska container (.mkv, and also conventionally used for webm) on the other hand turns out to work well for this application: I found that ffmpeg does update some headers at the beginning of a Matroska file when it stops encoding (similar to what it does for mp4 fast start), but the fields that get populated are not required to decode the video: they appear to only contain things like the total video length, which decoders do not require. In some experiments, I found that other programs were happy to play back a Matroska video that I had copied while ffmpeg was encoding it, even when capturing live video without a defined duration; they simply stopped playback on reaching the end of the data. In some situations players failed to report the overall video length or showed a wrong duration when asked to decode such a truncated file, but they were still able to play back everything that was present.

gcs-incremental

Recalling that I chose to use Google Cloud Storage to store captured video, gsutil is a convenient way to interface with the object storage system from shell scripts. Since ffmpeg is also easily driven from a shell script, the default choice for implementing the entire system was also shell, rather than some other (perhaps less quirky) programming language.

To implement incremental upload of files, the general algorithm for copying a ‘source’ file on the local system to a ‘destination’ file on remote storage can be expressed as follows (assuming, as we established in the previous section, that files are only appended to):

Check whether destination file exists
- If no, copy entire source file to destination file and exit
Get size of destination file
- If same size as source file, do nothing and exit
Append bytes from source file starting at offset remote size to destination file

Somewhat problematically, in most object storage systems like Google Cloud Storage, “objects” (files in our abstraction) are immutable: it is not possible to modify an object in place. Making changes to an existing object will then usually involve making a copy of the object with the changes applied, and doing so is most obviously implemented by downloading the original and making changes, then uploading the changed version (possibly replacing the original object).

It should be obvious that appending to a file stored on GCS by downloading it and re-uploading doesn’t achieve the goal of incremental upload, since in that case we could simply upload the entire local file. Fortunately, it’s possible to “compose” an object from multiple pieces: given two objects, gsutil compose can be used to concatenate them into a single object without making a copy of either. With that operation, appending to a file for incremental upload is simply a matter of uploading the new data as a new object, then performing a compose operation to add it to the original object.

The following shell script implements this incremental upload; I call it gcs-incremental. When passed the path to a local file and a location on Cloud Storage, it implements the algorithm described above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


#!/bin/bash -e

SOURCE_FILE="$1"
GS_PATH="$2"
DEST_FILE="${GS_PATH}/$(basename "${SOURCE_FILE}")"

info() {
 echo "$@" >&2
}

if ! gsutil -q stat "${DEST_FILE}"; then
 info "${DEST_FILE} does not exist; uploading entire file"
 gsutil cp "${SOURCE_FILE}" "${DEST_FILE}"
else
 SOURCE_SIZE=$(stat --format=%s "$SOURCE_FILE")
 DEST_SIZE=$(gsutil stat "${DEST_FILE}" | awk '$1 == "Content-Length:" { print $2 }' || echo 0)
 TO_UPLOAD_SIZE=$(("${SOURCE_SIZE}" - "${DEST_SIZE}"))
 PART_FILE="${DEST_FILE}.part.${SOURCE_SIZE}"

 if [ "${TO_UPLOAD_SIZE}" = 0 ]; then
 info "Nothing to upload; stopping"
 exit 0
 fi
 info "Uploading ${TO_UPLOAD_SIZE} bytes.."

 dd if="${SOURCE_FILE}" skip="${DEST_SIZE}" count="${TO_UPLOAD_SIZE}" iflag=skip_bytes,count_bytes \
 | gsutil -q cp - "${PART_FILE}"
 gsutil compose "${DEST_FILE}" "${PART_FILE}" "${DEST_FILE}"
 gsutil rm "${PART_FILE}"
 fallocate --punch-hole --offset 0 --length "${SOURCE_SIZE}" "${SOURCE_FILE}"
fi

There are several aspects of this implementation worth noting:

Getting the size of a file on GCS is slightly tricky because gsutil stat prints file properties in a format similar to HTTP headers. It’s not too hard to extract a number with awk.
gsutil doesn’t provide a way to select only part of a file to upload, so we use dd to read part of the file and pipe the data to gsutil. Use of a pipe prevents parallelization of the upload so performance is limited somewhat.
In order to compose the old and new file parts, we need to write a temporary file. This script assumes that a suffix of .part and an integer is sufficiently unique to avoid potential conflicts, but it would probably misbehave if multiple uploaders were trying to update the same file.
After a chunk of a file is uploaded, that data is erased from the local disk by using fallocate to punch a hole in the file, replacing all of the data that’s been uploaded with zeroes and freeing any space on disk that it used.

The hole-punching in the source file is what allows the overall time-lapse capture system to assume that the amount of storage available is not a concern. Files that have been uploaded will remain on disk but consume essentially no space while retaining their original size, making it easy to tell whether a file has been uploaded in its entirety even after the fact. Failed uploads may cause increased disk usage (because holes will not be punched), but no loss of data so they can be retried later.

Having implemented mechanisms to capture video (easily done with ffmpeg) and to incrementally upload those videos to GCS (gcs-incremental), what remains is to combine them into something that will capture video and incrementally upload it.

`timelapser`

Putting everything together into one script, the intended function can be summarized as follows:

Capture video with ffmpeg for a chosen duration
- In a loop, run ffmpeg and emit to a file with a chosen name with capture duration set to total duration - elapsed duration
- After the specified time is elapsed, ensure videos are uploaded and exit
Periodically do an incremental upload of captured video files, stopping once video capture ends

Capture task

The video capture itself is fairly easy to write. Assuming a few variables specifying things like how long to capture for, what video device to use as input, and what framerate to output, I ended up with these shell functions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


now() {
 date +%s
}

run_capture() {
 local framerate=$(echo "scale=4;1/${SECONDS_PER_FRAME}" | bc)
 local i=0
 local current_time=$(now)
 local end_time=$((${current_time} + (${RUNTIME_MINUTES} * 60)))

 # Run ffmpeg in a loop in case of premature exit/failure, targeting
 # one segment that goes until the end time.
 echo "Starting capture for ${RUNTIME_MINUTES} minutes, until timestamp ${end_time}"
 while [ ${current_time} -lt ${end_time} ]
 do
 ffmpeg -n -loglevel error \
 -f v4l2 -video_size "${VIDEO_RESOLUTION}" -i "${VIDEO_DEVICE}" \
 -t $((${end_time} - ${current_time})) \
 -vf fps="${framerate}" -an -c:v libvpx -b:v 20M -crf 4 "${i}.mkv"
 # Increment file serial number to avoid overwriting old data
 let i=i+1
 current_time=$(now)
 done
}

echo "Running for $RUNTIME_MINUTES minutes"

run_capture & capture_pid=$!

now provides the current UNIX time, which is convenient for computing the total amount of time capture has been running. run_capture assumes its working directory is appropriate for storing video and captures a sequence of files 0.mkv, 1.mkv and so forth. Capturing a sequence of files ensures that if some transient error occurs (perhaps if the camera is accidentally unplugged) capture will resume without overwriting any older data.

Because the script needs to also run periodic uploads, run_capture is started in the background and its PID is saved so its status can be polled later, in particular for checking whether it has completed and exited.

Upload task

Periodically doing an incremental upload is slightly more interesting, because I wanted to upload video parts at a regular cadence without any particular dependence on how long the upload takes. A naive version might implement a simple algorithm:

Wait for interval
Do incremental upload
If capture is still running, goto 1

If it takes any meaningful amount of time to perform the upload however, the uploads will be separated by the chosen interval and occur less frequently than intended. This is probably actually fine, but I chose to get clever with it to achieve a regular cadence.

Queueing

If uploads should occur at intervals without regard for how long they take to complete, this implies there must be two concurrent processes: one that waits for intervals to expire (the “timer” task) and another that actually does the upload (the “uploader” task). This becomes difficult when we recall that gcs-incremental cannot be expected to work correctly if invoked in parallel, since this implies there must be some mechanism to synchronize uploads both between incremental uploads and the final upload that runs once capture completes.

A reasonably simple approach to this problem in more capable (than shell scripting) programming languages is to use a multi-producer queue: the uploader task can pull upload jobs out of a queue and execute them serially, while the timer and capture tasks place new jobs into the queue as appropriate (at intervals and once capture completes, respectively).

In a shell script, I realized it’s possible to implement a queue with a pipe: if the receiver reads lines from a pipe in a loop until closed, other tasks can write lines to the pipe which will be processed in sequence. I ended up with this code for the receiver:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


segment_uploader() {
 while read n; do
 echo Segment trigger "$n"
 for f in *.mkv; do
 info "Do incremental upload of file $f"
 gcs-incremental "$f" "${GS_PATH}"
 done
 info "Segment upload completed (for now)"
 done
}

# Run another task to scan and upload segments, which we signal by sending
# a message through a named pipe.
msgpipe=$(mktemp -u)
mkfifo -m 600 "${msgpipe}"
# Run a task that does nothing but holds the pipe open; killing this
# pipe holder will terminate the uploader when it reaches EOF (having processed
# anything that was already put into the FIFO).
sleep infinity >"${msgpipe}" & pipe_holder_pid=$!
segment_uploader <"${msgpipe}" & uploader_pid=$!

trigger_upload() {
 now > "${msgpipe}"
}

We create a named pipe msgpipe and direct data from that pipe to the input of segment_uploader. The while read n loop will read lines from the input and stop once the input is closed, running gcs-incremental over all of the video files in its working directory.¹ Invoking the trigger_upload function will queue uploads of all current files.

The pipe_holder is an unusual component that is required only as a result of POSIX named pipe semantics: the read end of a pipe is closed once all writers disconnect, which in this application would be after the first segment upload is triggered because trigger_upload opens the pipe, writes to it and closes it again. The presence of the pipe_holder prevents the uploader task from exiting until the pipe_holder itself exits.

Because it may be possible for concurrent writes to the pipe to be accidentally interleaved, I forced invocations of trigger_upload to be serialized through use of signals:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


interruptible_sleep() {
 # Sleep in the background to make the sleep interruptible; waiting on a
 # foreground process isn't interruptible, but the wait builtin is.
 sleep $@ &
 wait $!
}

trap trigger_upload USR1
{
 while :; do
 interruptible_sleep ${SEGMENTS_INTERVAL}
 kill -USR1 $$ >/dev/null
 done
} & waker_pid=$!

The loop represented by waker_pid simply waits at intervals and sends SIGUSR1 to the main script. We trap that signal and in response execute trigger_upload that writes to the pipe. Reception of this signal can interrupt a wait as embodied in the interruptible_sleep function and asynchronously trigger an upload, but it is guaranteed by the system (through the general semantics of signals) that only one handler will execute at a time.

Capture completion

The final piece is to wait for capture to complete and trigger one final upload. Since the PID of the capture task was stored, this is as simple as waiting on it in a loop:

1
2
3
4
5
6
7
8
9


# Wait for capture to finish
while true; do
 wait ${capture_pid}
 wait_status=$?
 if [ $wait_status -lt 128 ]; then
 echo "capture task exited with status $wait_status"
 break
 fi
done

The dance with wait_status here is required because wait can be interrupted by a signal, and in fact we expect it to be periodically interrupted by a SIGUSR1 when we want to trigger an upload (as described in the previous section). In this situation the return code of wait is documented to be 128 or greater, so only when the return code is less than 128 is the capture task known to have exited.

Once the capture task exits, all that remains is to clean up and ensure all data has been uploaded:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Terminate the waker
kill $waker_pid

# Run a final segment upload
trigger_upload

# Terminate the pipe holder to close the write end of msgpipe; wait for the
# uploader to complete then exit.
kill $pipe_holder_pid
wait $pipe_holder_pid
echo "Waiting up to 1 hour for uploader to finish processing.."
(sleep 1h; kill -HUP $$) & wait $uploader_pid
echo "Done!"

We first terminate the waker task to prevent any more uploads from being triggered, and trigger a final upload. Since the capture task has exited by this point, this upload is guaranteed to see all the data that will ever exist.

After triggering the upload we kill the pipe_holder, closing the msgpipe which will make the uploader exit once it processes everything remaining in the pipe. To avoid waiting forever if there’s a problem while uploading, I chose to wait only up to an hour for it to complete before exiting.

Configuration

As discussed earlier, there are a few variables set at the top of the script guiding script operation. These mostly configure how video should be captured, but also specify the location in GCS for data storage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# The following variables may be overridden at runtime
RUNTIME_MINUTES=1
SECONDS_PER_FRAME=5
VIDEO_DEVICE=/dev/video0
# Supported resolutions can be found interactively:
# ffmpeg -f v4l2 -list_formats all -i ${VIDEO_DEVICE}
VIDEO_RESOLUTION="1280x720"
SEGMENTS_INTERVAL=1m

GS_PATH="gs://test-videos-01/$(date --rfc-3339=date)"

Because it’s useful to be able to change these without modifying the script, I opted to make it take paths on the command line which indicate files that timelapser will execute during startup.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


info() {
 echo "$@" >&2
}

while [ "$#" -gt 0 ]
do
 info "Loading configuration from $1"
 source "$1"
 shift
done

sourceing configuration files permits arbitrary configuration to be easily written and doesn’t require any special parsing, which is convenient. The configuration I ended up using for actual video capture looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


RUNTIME_MINUTES=$((60 * 9))
VIDEO_DEVICE=/dev/video0
VIDEO_RESOLUTION=2304x1536
SECONDS_PER_FRAME=10
SEGMENTS_INTERVAL=10m

today="$(date --rfc-3339=date)"
GS_PATH="gs://my-timelapse-video/${today}"

mkdir -p "${HOME}/${today}"
cd "${HOME}/${today}"

This runs captures for 9 hours at 0.1 fps, uploading video segments every 10 minutes. Because video is captured to the working directory, the configuration ensures that a directory named for the current time is created to store video, and the same directory name is used in the remote storage. Using a directory name based on the start time allows video segments to be easily correlated with when they were actually captured.

Automation

To run the system automatically, I set up some systemd units that will capture a video every day during working hours. A timer that triggers on weekdays:

1
2
3
4
5
6


[Timer]
OnCalendar=Mon..Fri 09:00:00
Persistent=true

[Install]
WantedBy=multi-user.target

And the matching service that is started by the timer:

1
2
3
4
5
6
7


[Unit]
Description=Capture time-lapse videos

[Service]
User=timelapser
Type=simple
ExecStart=/usr/bin/timelapser /etc/timelapser.conf

The service runs the script as its own user which is not really required, but is convenient for confining the effects of video capture to a well-defined space (mostly that user’s home directory).

Packaging

I wanted to make it easy to deploy and manage the scripts, in particular to be able to more easily handle changes and deploy the scripts to a fresh Pi in the future should I desire. The easy approach to deployment is to simply copy the scripts to the machine to run them, but it’s somewhat easier at deployment-time to use the OS package manager. Since Debian derivatives are usually used on Raspberry Pis, I spent some time learning how to create Debian packages and constructed a timelapser package containing the scripts and configuration needed to run this system.

The package does the following:

Install timelapser and gcs-incremental scripts to /usr/bin
Install the systemd units to /usr/lib/systemd/system
Install the sample configuration to /etc/timelapser.conf

It does not currently create the user or attempt to configure gsutil, so post-install operations should probably include:

Create the user and allow access to video devices: adduser timelapser --ingroup video (optionally choosing a non-default location for the home directory and so forth)
Log in to a Google cloud account for storage access: sudo -u timelapser gcloud auth login
Edit /etc/timelapser.conf to configure video capture options and storage location
- If a bucket does not already exist, create a bucket to store uploaded video

To change the time at which video is captured, systemctl edit timelapser.timer can be used to override the provided OnCalendar clause.

Finally, the usual systemd commands can be used to start automatically running capture on a schedule:

systemctl enable --now timelapser.timer

Or to run it once then stop (useful for testing configuration):

systemctl start timelapser.service

Downloads

Congratulations on making it to the end of this article. Here is the result all in one place, much easier to use as-is than trying to reassemble it from the code fragments in the article.

timelapser_1.0.tar.xz: complete code and packaging information, buildable with debuild.

gitlab.com/taricorp/timelapser: at time of this writing, the same as the above source tarball hosted on Gitlab. Easier to browse and may receive some updates.

Discussion

Although I’m happy with how this system works, some additional work is called for once a complete time-lapse has been captured.

Because videos are captured such that they play back in real-time (with video duration being equal to the original amount of time over which it was captured), I first combine all the video segments for each day and add a time readout:

1
2
3
4
5
6
7
8


for f in *.mkv
do
 echo "file $f" >> concat.txt
done

ffmpeg -f concat -i concat.txt \
 -vf 'drawtext=text=%{pts\\:hms}:fontsize=32:fontcolor=white:borderw=2:x=(w-tw)/2:y=lh,setpts=PTS/600.0' \
 -r 60 -f matroska dayfast.mkv

Using the concat input format allows days where video capture was interrupted and resumed later (writing to another file) to still result in a single video for the entire day.

The drawtext filter applied via the -vf option takes the timestamp of each frame (starting at 0 at the beginning of the video) and formats it as hours, minutes and seconds then overlays the text on the video at the top-middle. setpts then takes the same timestamp and divides by 600 (setting the output frame’s timestamp to that new value), so the video now plays back 600 times faster than the input.

Since sometimes there are long stretches of “nothing”, it’s useful later to do some filtering of each day’s video to drop frames where there’s very little change, combining the videos for each day into a single longer video; again using the concat input format to ffmpeg again and a different set of filters:

1
2
3


ffmpeg -f concat -i ... \
 -vf 'select=gt(scene\,0.02),setpts=N/(30*TB)' \
 out.mkv

The select filter will use or discard frames, and gt(scene,0.02) will only select those frames that differ from the previous frame by more two percent (according to some unspecified scene metric).² To speed the resultant video up further, setpts=N/(30*TB) increases speed by a further 30 times.

Compared to use of PTS in the earlier example, TB is used here because input frames are dropped: the PTS is based on the input frame’s time, so if PTS were used then the time filled by dropped frames would still exist in the output video but the frame itself would not exist (the previous one would continue to be shown). Since the goal of dropping similar frames is to reduce the runtime of the final video while preserving interesting activity, TB is the better choice.

As I write this conclusion, I’ve used this set of scripts to capture two different sets of videos to good effect, each covering more than a month of real time. The results have been satisfactory, and the system has been entirely hands-off aside from initial configuration and disabling it again when I was done, which nicely fulfills the goal of having a system that requires minimal attention.

I notice while writing this that the uploader task could be made more efficient by receiving the name of a file to upload rather than a string that is otherwise ignored, which would allow it to inspect only the relevant file rather than all those that exist. ↩︎
I arrived at the 2% difference in scene metric by experiment, finding that value wasn’t too sensitive (changes in the display time didn’t cause frames to be retained, for instance) but also that it didn’t seem to drop interesting periods of action. ↩︎

Marking token boundaries in TI-BASIC with Unicode magic

Thu, 26 May 2022 03:57:39 +0000

Users who are accustomed to writing TI-BASIC on computers in plain text like any other programming language are probably familiar with sometimes needing to explicitly mark where token boundaries occur. I’ve been doing some thinking about this lately, and have arrived at a proposal for a way to improve the situation for some uses.

TI-BASIC Background

What enthusiasts often refer to as the TI-8x series of calculators is the TI-83+ and its variants, including the improved TI-84+ and color-screen CE (and happily abandoned CSE) versions. These are calculators sold by Texas Instruments and commonly used in middle- and high-school math instruction. The internal architecture of these calculators is based on a Zilog Z80 processor with 512 kB or so of Flash memory and 32 kB of RAM (with larger memories in the newer versions).

Most of the TI graphing calculators support dialects of BASIC, which are usually referred to as TI-BASIC. The details differ between calculator families, but here I am concerned with the TI-BASIC dialect used on the 8x calculators.

TI-BASIC is stored on calculators as a sequence of tokens, each of which is one or two bytes long. Each token has a string that it is translated to for display: the token consisting of the bytes 0xbb,0x6d is displayed as AsmComp, for instance.

When programming TI-BASIC on a calculator, code is entered directly as tokens either through select keystrokes or by selecting tokens from menus. While pressing the 1 key enters a 1 token, pressing the key sequence VARS,7,1 enters the token Str1. Each syntatic element of the language is unambiguously specified to the calculator as a sequence of keystrokes, even if the representation of a set of tokens as plain text could be interpreted in multiple ways: although the string “sin(” could be displayed by pressing either the sin( key or the sequence s,i,n,(, the program as stored on the calculator is unambiguous because it captures which sequence of keys was pressed (in the form of tokens).

In more recent history, people have written tools allowing programmers to write TI-BASIC programs on general-purpose computers and translate them to program files that can be executed by calculators: the major examples of these tools are Token IDE and SourceCoder.¹

These programming tools support both tokenizing: translating plain text (which is easy to write on most computers) into tokens and packaging those into program files that can be placed on calculators, as well as the reverse detokenization operation where a collection of tokens is converted back into text. Typically a program will be tokenized in order to run it, and detokenized if changes are to be made to it (provided a plain-text copy does not already exist). The programs or portions of programs that perform tokenization and detokenization are usually called the tokenizer and detokenizer.

The need for breaks

Sometimes a marker needs to be inserted in the plain-text source code that a person writes in order to indicate where a string that might be interpreted as a single token should instead be interpreted as multiple tokens. As discussed above, there is more than one way to write tokens that display as “sin(” for example: if a programmer is writing text on a computer rather than tokens on a calculator, how can a tool determine which sequence of tokens is desired?

Perhaps the most common example of needing to do this is when the string “pi” appears in a program and it must be written as “p\i” to prevent the tokenizer from converting it to “π”. Although a calculator has a button for the π symbol, most computer keyboards do not: the tools allow a programmer to write “pi” instead as a convenience. This approach of offering easier-to-type aliases for tokens that contain unusual characters and inserting a backslash to indicate strings that should be broken into multiple tokens was first used by TokenIDE and later also supported by SourceCoder.

It’s surprisingly tricky to detect when a break like this needs to be inserted when converting tokens back into plain text, as discussed previously on on Cemetech: TI-BASIC (at least the 8x variant) was designed to only ever be written in tokens, so some way to mark token boundaries is required where a suffix of the concatenation of two or more valid tokens is also valid as another token.

Since that formalism is a little confusing when written in words, an example: if we have tokens “a”, “ab”, “bc”, and “c” then without any token break indicator it is unclear how to tokenize the plaintext string “abc”: it could be [a, bc] or [ab, c]. Inserting a break (\) disambiguates “a\bc” as [a, bc] and “ab\c” as [ab, c]. If detokenizing [ab,c], the output suffix “bc” when encountering the c token is also a valid token so we know a break must be inserted to disambiguate.

That’s all well and good, but backslashes are kind of ugly and break the flow when you’re reading code (even if they are necessary due to the language design). It occurs to me that Unicode has thousands of interesting characters, at least some of which could be used as explicit token breaks like we typically use backslash for while making code somewhat easier for humans to read.

Exploring Unicode alternatives

Unicode TR14 describes the blessed Unicode line breaking algorithm that guides when it is permitted to split text across multiple lines inside a block of text. While not strictly useful for the desired application of splitting tokens with no or little effect on how the code looks to humans, it does provide some pointers to interesting characters, including:

Glue characters like Zero Width Joiner (ZWJ), U+200D: an invisible character that prevents breaking the text on either side of it (of particular use in emoji sequences as described by TR51!). The opposite of what we want.
Next Line (NEL), U+0085: forces the following text to appear on a new line. This behaves the same as most programmers would expect a Carriage Return or Line Feed character (or the combination of the two) to behave,² and it turns out that section 5.8 of the Unicode standard spends many paragraphs on suggesting how applications treat each of the possible newline characters. The variety of options exist largely due to historical differences in how computer systems record line breaks, where NEL in particular is probably unfamiliar to most because it was used by EBCDIC computers which were mostly IBM machines and are very uncommon these days. Interesting, but not useful for this application.
Assorted punctuation, such as ‘!’, ‘}’ and ‘[’. These forbid line breaks before or after them depending on the orientation, such as ‘)’ forbidding a break before it because a close parentheses logically binds to the text that precedes it. Getting closer, but line breaks aren’t really relevant to wanting to mark the boundaries between tokens.

Adjacent to ZWJ in the code space we find an interesting character: U+200D ZERO WIDTH NON-JOINER (ZWNJ). Unicode Section 23.2 says this (and ZWJ for the opposite) is designed to mark where connections between characters are forbidden, as in cursive scripts or if a ligature might be used. Inserting a ZWNJ between two characters that might otherwise be joined forces them to be disconnected.

ZWNJ

If one figuratively squints at the intent of ZWNJ, it seems similar to the needs outlined for TI-BASIC: we want to prevent characters from running together in some situations. Where normally a tokenizer will eagerly join characters into tokens, a ZWNJ could be inserted as a break character that is otherwise invisible to human readers.

With this in mind, we could label the use of a backslash to escape tokens as the traditional method and compare it to use of ZWNJ (call it “invisible” breaks) or doing nothing:

Split mode	Plaintext	Tokenized
None	`Disp "I like to eat pie`	`Disp "I like to eat πe`
Traditional	`Disp "I like to eat p\ie`	`Disp "I like to eat pie`
Invisible	`Disp "I like to eat p‌ie`	`Disp "I like to eat pie`

As already established, not inserting a break is ambiguous and because tokenizers must take the longest prefix of a given input as a token,³ the “pi” ends up incorrectly transformed to the Greek letter pi. In the traditional break style, we insert a backslash to force “pi” to be interpreted as two Latin letters rather than being translated to the Greek pi token.

In the invisible mode, there is still a break present but it is not visible in the written text: I have inserted a ZWNJ character (HTML &zwnj;) which can be interpreted by a tokenizer in the same way as a backslash (which is to say, ignored other than forcing a token split).

Discussion

Although I’m pleased with the idea to insert ZWNJs into human-readable BASIC programs, doing so in general seems limited by readers’ needs to only some contexts. It does also offer some interesting possibilities, though.

If a person might visually read out the plaintext source code and convert it to tokens (such as by typing it into a physical calculator), the loss of visual breaks means that the human must attempt to resolve any ambiguities that appear. While an experienced TI-BASIC programmer can probably discern intent from the program’s context in order to disambiguate, depending on a reader’s skill seems like a suboptimal solution. As a counterpoint however, a novice programmer may not even be familiar with the backslash-as-break “traditional” convention either: in that case invisible breaks could be superior.

If a user is expected to be able to copy and paste Unicode text to convert it into tokens (such as in source code published to the web, like this article), invisible splits are convenient and easy to read as long as tokenizers can be assumed to understand them. If a user might do visual transcription (such as manually typing code from a book into a calculator), traditional (visible) splits may be preferred.

Increasing break frequency

Invisible breaks also present an interesting opportunity to mark more token boundaries than only those required to disambiguate textual source code: what if a ZWNJ were inserted on every token boundary? Doing so would make a given program’s source code forward-compatible with alternate token sets that have more tokens! Because breaks must be inserted where ambiguity is known, adding more tokens can introduce new ambiguities that might not be handled by an older detokenizer (which has a smaller set of known tokens).

If it can be assumed that every pair of tokens has break character between them, then a tokenizer can simply split its input text on break characters and emit tokens matching exactly the strings that are separated by breaks.

Unfortunately this would require a somewhat different mode of operation for tokenizers when compared with the traditional longest-prefix matching. It may be possible to support both modes concurrently however, if a tokenizer first attempted an exact match of the input up to the next break character and fell back to longest-prefix matching in case of no exact match in order to handle breaks between every token but also support minimal-break inputs.

This break-every-token approach is possible with traditional breaks as well as invisible, which is easier to illustrate. Consider a program fragment written for a monochrome-display TI-83+: “Red cat”. The string “Red” is a token in its own right on the CSE and CE 8x calculators (because they have color screens), so interpreting this as a program for color calculators would tokenize it differently. This becomes clear if we insert breaks around every token:

Monochrome: R\e\d\ \c\a\t
Color: Red\ \c\a\t

A program written as the monochrome version with breaks around every token as illustrated here cannot be mistaken for the color one, and the color one cannot be mistaken for the monochrome! While doing so with visible break characters makes it much more difficult to read the code, invisible breaks would not affect readability the same way visible ones do.

Concluding

I think it would be pretty cool if existing tools added support for these proposed invisible break characters. They’re not appropriate for all use cases (so perhaps shouldn’t be the default), but can be useful and would be even more useful with the proposed change to the usual tokenization algorithm.

The TI-83 uses essentially the same dialect of BASIC as the 8x series and was first sold in 1996. Some versions of TI’s GraphLink software included a program editor that I’ve never used, and SourceCoder first appeared in 2005. TokenIDE appeared around 2011 and its use of XML files to describe tokens was adopted by later versions of SourceCoder. I imagine TokenIDE’s open source nature contributed to its success in influencing later work on SourceCoder. ↩︎
Typewriter users might also recognize these terms and intuit their meaning, since they were included in the ASCII character set in 1967 due to their importance to teletypes, where there is a physical distinction between simply advancing the feed by one line and moving the carriage back to the start of the line. ↩︎
This requirement may not be obvious: if there are two tokens “Y” and “Yellow” for instance, given input “Yellow” a tokenizer must choose the longer of the tokens matching the input (namely, “Yellow”). If it did not, it would be impossible to reliably recognize the token “Yellow” because “Y” might be treated as a token instead, leaving “ellow” to be tokenized separately. ↩︎

Managing Google Photos duplicates with Python

Sat, 30 Apr 2022 04:05:40 +0000

I recently had a bit of a problem with the files that had ended up in Google Photos on my account: the Google Drive desktop synchronization app seemed to have noticed the many (reasonably-high-resolution) thumbnails that my local photo management application (Lightroom) creates, and had uploaded many near-duplicate images.

It seems this wasn’t a problem with the old “Backup and Sync” application because it supported excluding some files from backup (so I could have it ignore the directory that thumbnails get put into), but the new Drive application lacks such a feature.

While Google’s sync tool does know how to avoid uploading exact duplicates of photos, it doesn’t do any similarity matching on image content, so thumbnails (with the same content but lower resolution) and alternate formats (sidecar JPEGs that go along with camera raw files) end up duplicated in the Google Photos library. I have previously ignored the duplicates in alternate formats because they weren’t too annoying, but when the sync tool uploaded a few thousand duplicate thumbnails I felt the need to do something about it.

Existing programs

There aren’t many existing options for managing duplicate items like this. Doing a web search for “Google Photos remove duplicates” mostly gives back content farm-style articles that talk about how Photos provides some duplicate-finding features (yes, but they’re limited) and usually go on to suggest some nonfree application for finding duplicates on your local computer. These are all unhelpful because:

I don’t have duplicates on my local computer, and want to clean up the duplicates in the Google library so it’s easier to browse (and to save some space)
The official duplicate finding tool doesn’t appear to look at visual similarity, so misses the thumbnails I want to remove
They’re trying to sell me something that doesn’t do what I want

I did discover Rémi Mikel’s duplicate finding tool which seemed like a step in the right direction, but seemed to perform too badly to be very useful (probably because there are thousands of images that I want to get rid of, and it tries to display all of them in your browser). It did suggest an approach I could take to do this myself though, by using the public Google Photos API.

Rolling an API client

With some ideas in mind, I wrote a little Python application to gather data. Using the public API is fairly straightforward; the most difficult part is probably in simply setting something up to be able to authenticate with my own credentials and access my photos.

Configuring authentication

In the Google Cloud console, I created a project and enabled the Photos Library API for it, then created an OAuth client ID that I could use in my application. Since the steps are easier to illustrate than describe, here are screenshots of the process after having created a project.

Enable the Photos Library API in the API library.
Create an OAuth consent screen for the application. This gets shown to users when providing the application access to the data in their Google account. It’s fine to make this an “External” application.
- This requires you to fill in an app name and some email addresses, but what exactly those are is up to you; they don’t really matter if this project is only going to be for personal use, as it was in my case.
- When asked to select scopes for the app, select the relevant ones for the Photos Library API. photoslibrary and photoslibrary.sharing are sufficient for this use case. Any scopes that aren’t enabled here won’t be available to the application (permission will be denied) when run.
- The third step adds users to the allowlist for a testing app, which is the default configuration. Non-testing apps are available to the public but may require review by Google before they can be published, but since this is for personal use on my own computer it’s fine to let it stay as a test app. I added my own email address to the list of test users and continued on.
Having configured the consent screen, we can now create a new OAuth client ID, in the API credentials page
- Choose to create new credentials, and select “OAuth client ID”.
- Set the Application type to Desktop and fill in a name. This program runs locally and is not a web application, so the type must be Desktop in order for the local authentication flow to be permitted.
- Download the JSON for the created client details after the client is created. The program will need to provide this to Google when logging in.

I saved the downloaded file as client_secret.json and then got to actually writing my program.

Downloading data

Rather than walk through the process of writing the program I did, I’ll simply present the code after describing what I chose to have it do.

Since I was taking inspiration from Mikel’s tool, I knew I only wanted to gather photo metadata to compare, which is largely returned when simply listing a user’s library. To play with that data interactively and figure out what I wanted to filter on to delete, I decided to store it in a SQLite database, since that would be easy to query later. With that in mind, I simply made the program call mediaItems.list in a loop, taking a page (100 items) at a time until there were no more pages. For each item, it extracts the metadata I care about and stores it in the database.

To work with the database I opted to use SQLAlchemy simply because it was a convenient way to manage types that the database might not directly understand and run queries in a slightly less cumbersome way. In total, these libraries are needed to run the below code (all installable from PyPI):

google-api-python-client
google-auth-oauthlib
SQLAlchemy
python-dateutil

With that out of the way, here’s the actual code I wrote:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86


import datetime as dt

import dateutil.parser
import googleapiclient.discovery
from google_auth_oauthlib.flow import InstalledAppFlow
from sqlalchemy import Column, String, create_engine, DateTime, Integer, Float, Interval
from sqlalchemy.orm import declarative_base, Session

Base = declarative_base()


class Photo(Base):
 __tablename__ = 'photos'

 id = Column(String(256), primary_key=True)
 filename = Column(String(128), nullable=False)
 mime_type = Column(String(64), nullable=False)
 creation_time = Column(DateTime(timezone=True), index=True)
 width = Column(Integer())
 height = Column(Integer())
 camera_make = Column(String(64))
 camera_model = Column(String(64))
 focal_length = Column(Float())
 f_stop = Column(Float())
 iso_equivalent = Column(Integer())
 exposure_time = Column(Interval())


engine = create_engine("sqlite:///photos.sqlite", future=True)
Base.metadata.create_all(engine)

# No idea how to restore saved credentials, so don't try. Many examples use
# oauth2client.file.Storage but that doesn't work, at least in recent versions:
# https://github.com/googleapis/google-api-python-client/issues/491
credentials = None
if credentials is None or credentials.invalid:
 flow = InstalledAppFlow.from_client_secrets_file(
 'client_secret.json',
 scopes=['https://www.googleapis.com/auth/photoslibrary'],
 )
 credentials = flow.run_local_server(
 host='localhost',
 port=8000,
 open_browser=True,
 )
print('Authenticated OK')

service = googleapiclient.discovery.build('photoslibrary', 'v1', credentials=credentials, static_discovery=False)

with Session(engine) as session:
 page_token = None
 while True:
 session.begin()
 response = service.mediaItems().list(pageSize=100, pageToken=page_token).execute()
 for item in response.get('mediaItems', []):

 photo = Photo(
 id=item['id'],
 filename=item['filename'],
 mime_type=item['mimeType'],
 )
 metadata = item.get('mediaMetadata')
 if metadata is not None:
 if 'creationTime' in metadata:
 photo.creation_time = dateutil.parser.isoparse(metadata['creationTime'])
 if 'width' in metadata:
 photo.width = int(metadata['width'])
 if 'height' in metadata:
 photo.height = int(metadata['height'])
 if 'photo' in metadata:
 photo_meta = metadata['photo']
 photo.camera_make = photo_meta.get('cameraMake')
 photo.camera_model = photo_meta.get('cameraModel')
 photo.focal_length = photo_meta.get('focalLength')
 photo.f_stop = photo_meta.get('apertureFNumber')
 photo.iso_equivalent = photo_meta.get('isoEquivalent')
 if 'exposureTime' in photo_meta and photo_meta['exposureTime'].endswith('s'):
 photo.exposure_time = dt.timedelta(seconds=float(photo_meta['exposureTime'][:-1]))

 print(photo.id)
 session.merge(photo)

 session.commit()
 page_token = response.get('nextPageToken')
 if page_token is None:
 break

Running this opens a web browser to grant access to a Google account, and when authentication succeeds it then grabs metadata for every item in the Google Photos library, printing the ID of each item while it goes.

Finding unwanted thumbnails

To find the thumbnails I don’t want, I can start by manually running queries against the generated SQLite database. It turns out the unwanted thumbnail files have a common filename format that appears to be a UUID, and they’re always .dng files. It wasn’t too difficult to search for the files that have matching filenames and (to avoid deleting those that don’t seem to correspond to any other image) avoid matching those that don’t appear to have an original file (taken by the same camera at the same time) already known:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


SELECT *
FROM photos AS main
WHERE filename LIKE '________-____-____-____-____________.dng'
AND EXISTS(
 SELECT id from photos
 WHERE creation_time = main.creation_time
 AND camera_make = main.camera_make
 AND camera_model = main.camera_model
 AND id != main.id
)

This returns 3651 rows from my library, so then the problem is how to take those images (probably by ID) and remove them from Google Photos.

Removing unwanted images

Unfortunately, the Google Photos API doesn’t provide any way to delete photos. This seems to be largely because a malicious application could use it to delete all of a user’s photos (a reasonable concern!), but it makes what I want to do difficult. One user on the above bug suggests using the API to add photos to a new album, from which the user can delete photos with relative ease, which I think is a reasonable choice.

I wrote another script (copying the authentication code from before into a new auth module for reuse), taking the ID of an album on the command line, as well as the name of a file containing item IDs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


import sys
import googleapiclient.discovery

album_id = sys.argv[1]
item_ids_file = sys.argv[2]

from .auth import credentials

service = googleapiclient.discovery.build('photoslibrary', 'v1', credentials=credentials, static_discovery=False)

with open(item_ids_file, 'r') as f:
 item_ids = f.readlines()


def chunks():
 # batchAddMediaItems takes no more than 50 items per call
 for i in range(0, len(item_ids), 50):
 yield item_ids[i:i+50]

for chunk in chunks():
 service.albums().batchAddMediaItems(albumId=album_id, body=dict(
 mediaItemIds=chunk
 )).execute()

I created an album manually via the API, and created a file with one ID per line by exporting the results of the above SQL query to a CSV file with only one column (the ID of the item). Unfortunately, this fails with an HTTP 400, complaining that Request contains an invalid media item id. After some experimentation (manually using the API to get a single ID and add it to an album), I realized the problem was an unfortunate but intentional API limitation:

Note that you can only add media items that have been uploaded by your application to albums that your application has created.

..so it turns out it’s impossible to do anything useful via the API alone. It might be possible to do this via browser automation like a “delete all photos” tool I happened across, but that’s a bigger hack than I’m interested in right now, so I’m forced to give up on the goal for now.

Other uses of the data

I had a few other ideas of interesting things to do with the database I collected in addition to the desired “remove unwanted thumbnails”:

Locate other duplicates by finding images with similar metadata (camera, capture time, ISO equivalent, shutter speed, …). This would probably match raw files that have matching sidecar JPEGs, where I would likely prefer to remove the raws since the JPEGs tend to be better suited to browsing and I don’t use Google Photos as a repository for high-quality originals.
Find files that exist in Google Photos but not on my local storage. It’s possible that some files would have been accidentally deleted that I could recover a version of (not the originals, probably) from the web. It’s also possible there would be a lot of false positives of photos that I intentionally deleted and would want to remove from Google, though.

In general, there are some interesting possibilities for ways in which photos data in this form could be swizzled into a helpful outcome, but it’s unfortunate that the APIs provided for Google Photos mean there aren’t many ways the data can be used to any meaningful end.

Monitoring (and preventing) excessive hard drive head parking on Linux

Sat, 14 Aug 2021 05:15:10 +0000

It is fairly well-known among techies that hard drives used in server-like workloads can suffer from poor configuration by default such that they frequently load and unload their heads, which can cause disks to fail much faster than they otherwise would. While I have been aware of this in my home server as well, it is easy to forget to ensure that disks are not silently killing themselves by cycling the heads. Since I use Prometheus to capture information on the server’s operation however, I can use that to monitor that my hard drives are doing well.

Collecting SMART metrics

The Prometheus Node Exporter is the canonical tool for capturing machine metrics like utilization and hardware information with Prometheus, but it alone does not support probing SMART data from storage drives. It does support reading arbitrary metrics from text files written by other programs with its textfile collector however, which is fairly easy to integrate with arbitrary other tools.

Smartmontools is the typical package of tools for reading SMART information from drives on Linux, and conveniently the community-maintained example scripts for collecting system information with Prometheus includes both a Python and shell script (smartmon.sh and smartmon.py) that generate metrics for all devices on a system that support SMART reporting. I run the shell script every 5 minutes with a systemd service triggered by a timer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[Unit]
Description=Export smartctl metrics to Prometheus Node Exporter

[Service]
Nice=-10
ExecStart=/bin/sh -c 'exec /usr/local/bin/node_exporter_text_smartmon > /var/state/prometheus/smartctl.prom'
# Write nothing except the output file
ProtectSystem=strict
ReadWritePaths=/var/state/prometheus
# Shell needs a temp directory
PrivateTmp=true
ProtectHome=tmpfs

My node exporter is then configured to read text files from /var/state/prometheus, and Prometheus itself gathers metrics from the node exporter.

Monitoring metrics

With the SMART metrics captured by Prometheus, it’s fairly easy to write a query that will show how often a given disk is parking its heads. The smartmon_load_cycle_count_value metric seems like it would be the right one to query, but that actually expresses a percentage value (0-100) representing how many load cycles remain in the specified lifetime- on reaching 0 the disk has done a very large number of load cycles.

Somewhat more useful for monitoring is the smartmon_load_cycle_count_raw_value, which provides the actual number of load cycles that have been done. By taking the rate of those it becomes pretty easy to see which disks are loading and unloading most, so I choose to query with sum(rate(smartmon_load_cycle_count_raw_value[6h])) by (disk) to summarize by disk and get a 6-hour rolling average rate in load cycles per second, pictured here:

In this case, there are at least two disks that I probably need to configure, since /dev/sde seems to be parking as often as about every 4 minutes (0.004 Hz) and /dev/sdc is only parking slightly less often. /dev/sdb also seems worth inspecting.

Preventing excessive parking

To prevent parking more often that is useful (for a server, usually that choice would be “very rarely”), there are a couple ways to do it and which apply will depend on what the hard drive vendor’s firmware supports.

Of the three disks that I decided need some attention, I have one Western Digital disk and two Seagate ones. From the SMART data again, they are specifically these models:

/dev/sdc: Seagate Archive HDD (SMR) ST8000AS0002-1NA17Z, firmware version RT17
/dev/sde: Seagate IronWolf ST8000VN004-2M2101, firmware version SC60
/dev/sdb: Western Digital Red WDC WD40EFRX-68N32N0, firmware version 82.00A82

APM

The APM specification dating from 1992 includes some controls for hard drives, allowing a host system to specify the desired performance level of a disk and whether standby is permitted by sending commands to a disk. On Linux, the hdparm tool is most typically used to manage APM- in particular hdparm -B allows the user to specify a value to specify to the drive, corresponding to the table of Advanced Power Management levels in the ATA/ATAPI specification (Table 30 in T13/1321D revision 3), which I also reproduce here:

Maximum performance: FEh
Intermediate power management levels without Standby: 81h-FDh
Minimum power consumption without Standby: 80h
Intermediate power management levels with Standby: 02h-7Fh
Minimum power consumption with Standby: 01h
Reserved: FFh
Reserved: 00h

The specification text further elaborates on the meanings of these values:

Device performance may increase with increasing power management levels. Device power consumption may increase with increasing power management levels. The power management levels may contain discrete bands. For example, a device may implement one power management method from 80h to A0h and a higher performance, higher power consumption method from level A1h to FEh. Advanced power management levels 80h and higher do not permit the device to spin down to save power.

If we wanted to allow the disk to still park its heads but at minimum frequency, setting the APM value to 7Fh (hdparm -B 127) seems to be the correct choice. To prevent parking the heads at all a value greater than 128 may do the job (254 is a common choice, as the highest-power setting available), but it’s possible that some disks won’t behave this way because the ATA specification refers only to spinning down the disk and does not specify anything about parking heads.

Unfortunately, APM settings don’t persist between power cycles so if we wanted to change disk settings with APM they would need to be reapplied on every boot. On a Linux system this could be done with a udev rule matching a chosen drive, for instance matching the chosen disk’s serial number:

ACTION=="add", SUBSYSTEM=="block", KERNEL=="sd[a-z]", \
ENV{ID_SERIAL_SHORT}=="ABCDEFGH", \
RUN+="/usr/bin/hdparm -B 127 -S 0 /dev/%k"

(The properties like ID_SERIAL_SHORT can be queried on a running system using udevadm info, such as udevadm info /dev/sdd to get the properties of the disk currently assigned ID sdd.)

Disk vendors typically provide their own vendor-specific ways to do persistent configuration of power management settings, so it’s worth trying to use those instead so the desired configuration doesn’t depend on the host system applying it, instead being configured in the drive (but in some cases it might be desirable to have the host configure that!).

Western Digital `idle3`

For drives made by Western Digital, the inactivity timer for parking the heads is called the idle3 timer. Of particular note, WD Green drives ship configured to park the heads after only 8 seconds of inactivity which could notionally wear out the disk in a matter of months if the heads are cycling more-or-less continuously!

The idle3-tools package allows configuring the timer on Linux, though timer values are rather unintuitive- the tool sets a “raw” idle3 timer value, so a value like 232 (0xe8) actually means 3120 seconds according to idle3ctl -g105, and it’s unclear which interpretation of a given value actually applies to a given drive- if important that would need to be determined by experiment.

idle3ctl -d /dev/sdb -s 254

The other slight annoyance when setting the idle3 timer on WD drives is that changes only take effect when the drive is powered on, usually meaning the host computer must be fully shut down and started back up for any changes to be seen- this makes experimentation to determine how raw timer values are interpreted a slower and more tedious process.

Seagate EPC

Most Seagate disks have configurable Extended Power Conditions (EPC) settings that include timers for how long the disk needs to stay idle before entering various low-power modes.

According to at least one manual, the four low-power states are:

idle_a: power down some electronics
idle_b: park the heads (unloading them)
idle_c: reduce spindle speed, heads unloaded
standby_z: stop spindle completely

Seagate provide a “Seachest” collection of tools for manipulating their drives, but rather more usefully to users of non-Windows operating systems like Linux they also offer an open-source openSeaChest. The tool to use there is openSeaChest_PowerControl which allows each of the EPC timers to be configured, in an invocation like:

openSeaChest_PowerControl -d /dev/sdc --idle_a 6000 --idle_b 1800000 --idle_c 2400000

The timer values specified are in milliseconds, so this example will park the disk heads after 30 minutes of inactivity. The current settings for a disk can be queried with the --showEPCSettings flag.

My Seagate Archive SMR disk (which began life as an external hard drive and was retired from that role when it became too small to hold as much as I wanted to back up to it) apparently doesn’t support reporting EPC settings (since asking for them says so), and initially didn’t accept new values for the idle timers either. After using the --EPCfeature enable option however, it seems to have accepted custom idle timer values: I’ll have to watch the park counts on that to ensure it actually worked.

Verifying settings

At a glance, changing idle3 and EPC settings seems to have done the job nicely; here is the same graph of head park rates per disk as before, but on a smaller timescale that makes individual head parks visible. The parking rate basically drops to zero at the time I updated the settings for the Seagate drives, and the Western Digital one hasn’t changed because it needs to be powered off to change that setting and I haven’t done so yet.

Monitoring SSDs

This same setup can be used to monitor wear on SSDs, which is rather convenient. While SSDs don’t have any heads to park, most do report a media_wearout_indicator that represents the amount of data written to the device in relation to the amount that it’s specified to accept before the Flash storage medium wears out.

For the system I’m monitoring here, the SSD that it boots from has a wearout indicator sitting on 95 of 100 (only 5% of the rated life consumed), visibly unchanged for a long time so it’s not very interesting as an example.

Cross-compiling CMake projects for Windows

Wed, 20 Jan 2021 22:55:37 +0000

I sometimes find myself wanting to cross-compile programs with CMake on a Linux machine such that I get a standalone .exe that can be given to mostly non-technical Windows users to run. This isn’t hard, but finding the right options is a little bit of a challenge every time, so now I’m recording the procedure here; both as a reminder to myself, and to provide a quick recipe that future searchers can use.

Cross-compiling for Windows will of course need an appropriate toolchain, which these days tends to be mingw-w64. Many Linux distributions provide packages for it: mingw-w64 on Debian (including Ubuntu and variants), mingw-w64-gcc on Arch and similar for other distributions.

CMake documents how to specify cross-compilation options, but it’s not terribly clear which settings are mandatory. For simple needs, only three variables must be set:

CMAKE_SYSTEM_NAME being set implies that you’re cross-compiling, which will prevent cmake from trying to do things like run the binaries it builds (unless they’re tool binaries being built for the host, rather than the target).
CMAKE_C_COMPILER is the name of the compiler to use
CMAKE_CXX_COMPILER is the name of the compiler to use for C++ sources

Targeting mingw-64 then, the cmake invocation looks like this:

cmake -DCMAKE_SYSTEM_NAME=Windows \
-DCMAKE_C_COMPILER=i686-w64-mingw32-gcc \
-DCMAKE_CXX_COMPILER=i686-w64-mingw32-g++ \
path_to_sources

The i686- prefixed compiler builds 32-bit binaries, which I usually prefer to build because they’ll work both on 32- and 64-bit Windows. If no support for 32-bit Windows is required, the x86_64--prefixed tools will build 64-bit binaries instead (eg, x86_64-w64-mingw32-gcc).

When building binaries to share it’s probably helpful to do a non-debug build by also setting CMAKE_BUILD_TYPE, perhaps to MinSizeRel or RelWithDebInfo.

Libraries

Libraries that you might depend on can be built in the same way, and installed to a chosen directory by setting CMAKE_INSTALL_PREFIX then building the install target. For instance building a libpng static library (which itself provides options to enable a shared library and tests which we turn off):

cmake -DCMAKE_SYSTEM_NAME=Windows \
-DCMAKE_C_COMPILER=i686-w64-mingw32-gcc \
-DCMAKE_CXX_COMPILER=i686-w64-mingw32-g++ \
-DCMAKE_INSTALL_PREFIX=$HOME/windows_binaries \
-DPNG_SHARED=OFF -DPNG_TESTS=OFF \
path_to_sources
cmake --build . --target install

As I’ve observed previously, you can then link against these libraries by setting CMAKE_PREFIX_PATH. So if we want to build against the libpng that was just built and generate a “release” binary:

cmake -DCMAKE_SYSTEM_NAME=Windows \
-DCMAKE_C_COMPILER=i686-w64-mingw32-gcc \
-DCMAKE_CXX_COMPILER=i686-w64-mingw32-g++ \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_PREFIX_PATH=$HOME/windows_binaries \
path_to_sources

libpng notes

While libpng (used as an example above) provides a nice CMake configuration that cross-compiles cleanly, it depends on zlib which does not. However zlib does distribute a Makefile intended for targeting GNU tools on Windows. So before building libpng I first compile zlib with a particular incantation:

make -f win32/Makefile.gcc \
BINARY_PATH=${INSTALL_DIR}/bin \
INCLUDE_PATH=${INSTALL_DIR}/include \
LIBRARY_PATH=${INSTALL_DIR}/lib \
SHARED_MODE=0 \
PREFIX=i686-w64-mingw32- \
install

Because this is just a Makefile, sadly the appropriate options need to be discovered by reading the source: projects that natively use CMake are much easier to support!

Windows MIME type detection pitfalls

Thu, 24 Sep 2020 23:12:12 +0000

I’ve been doing some Django development lately, and was mystified why it seemed the debug toolbar on my local development instance wasn’t showing up, though it had been in the past. It turns out to have been a surprising interaction between browsers sometimes enforcing that resources be served with correct MIME types and the way Windows provides system-wide MIME type configuration (which seems to have major flaws)!

Investigating further, I found that the markup for the debug toolbar was in fact being generated and placed in the pages served by my development server, but not being shown. I guess that there may have been a recent update to django-debug-toolbar that may have broken it, and found that version 3.0 (quickly followed by 3.1) had recently been released. Downgrading to version 2.2 fixed the issue, so I was confident I hadn’t somehow misconfigured the debug toolbar.

Looking more closely at the browser console, I noticed an important error message:

Loading module from “http://localhost:8000/static/debug_toolbar/js/toolbar.js” was blocked because of a disallowed MIME type (“text/plain”).

It seems that django-debug-toolbar 3.0 started loading the scripts that make it work as Javascript modules rather than plain scripts, because in the past I’ve noticed (and learned to ignore) similar warnings when using the Django development server:

The script from “http://localhost:8000/static/myscript.js” was loaded even though its MIME type (“text/plain”) is not a valid JavaScript MIME type.

This change has thus broken the debug toolbar because the scripts that make it work are no longer being executed since my local server is serving javascript files with the wrong Content-Type. Why is that?

MIME type guessing

Static files (including the Javascript files causing me problems) are served by Django when using the development server, using the django.view.static.serve view. Of particular note here, it uses the mimetypes module to guess the MIME type of files based on their name:

1
2


 content_type, encoding = mimetypes.guess_type(str(fullpath))
 content_type = content_type or 'application/octet-stream'

Digging into what mimetypes does, it’s documented to read mappings from the registry on Windows, and from a set of known paths on all other operating systems. The relevant code for loading mappings from the registry looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


with _winreg.OpenKey(_winreg.HKEY_CLASSES_ROOT, '') as hkcr:
 for subkeyname in enum_types(hkcr):
 try:
 with _winreg.OpenKey(hkcr, subkeyname) as subkey:
 # Only check file extensions
 if not subkeyname.startswith("."):
 continue
 mimetype, datatype = _winreg.QueryValueEx(subkey, 'Content Type')
 if datatype != _winreg.REG_SZ:
 continue
 self.add_type(mimetype, subkeyname, strict)

In short: it enumerates keys in HKEY_CLASSES_ROOT, ignoring any that don’t have a name starting with “.” and uses the value of a “Content Type” subkey (if present) as the MIME type for files with that extension.

As far as this goes, it seems quite reasonable. Microsoft document HKEY_CLASSES_ROOT as a combined view of system-wide and per-user filetype associations, and the “Content Type” subkey may be set to a file type’s MIME type.

Opening up the registry editor on my system, I do see that the value of HKEY_CLASSES_ROOT\.js\Content Type is text/plain. So this is where Python got the incorrect MIME type from, and I can fix it by changing the value of Content Type. For instance, this snippet in a .reg file (maybe js.reg) can be imported to set a correct MIME time for .js files, changing a per-user setting if set or the global one otherwise:

Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\.js]
"Content Type"="text/javascript"

Problems with this system

While it seems abstractly reasonable for programs to use the registry-based mechanism for getting file MIME types, the other ways it is used in practice seems inappropriate for these purposes.

As a Django-based example, I found a Stack Overflow question with similar incorrect MIME type while investigating my initial problem. While the original question asker arrived at a workaround of manually configuring a correct MIME type in Python, after investigation it seems clear that the problem is better considered a system misconfiguration and I feel correcting the data in the registry is more appropriate.¹

Python isn’t even the only platform that suffers from certain prevalence of incorrect MIME types in the Windows registry. A fairly recent bug for the Go language indicates that the standard library for Go uses the same approach for guessing MIME types, with the same tendency to get wrong data.

On a fresh install of Windows, it even seems that many MIME types are unconfigured. When I checked what content type was set in the registry for .js files on a pristine copy of Windows (using a Windows Sandbox to have a quick and easy look), I found that there was none set. Being unset by default, it seems that in many cases (if nothing has set a mapping in the registry) Django will serve javascript as application/octet-stream because Python won’t find any configuration, which seems incorrect.

Fixing the problem once and for all

While it would be relatively easy to ship a reasonable MIME type mapping with Python or Django (indeed, Go seems to do this and use values from the registry instead if available), doing so would only fix the default case where no value is set.

The root problem with MIME type guessing on Windows seems to be that incorrect types often get written to the registry. It’s not obvious where these incorrect values come from, but I suppose the bad data tends to be written by some programs when they are configured to open those files. It seems like these programs might be badly-behaved when they do this since associating a file type with a program doesn’t necessarily say anything about the file’s MIME type, but it seems widespread enough that fixing that is intractable.

While it is possible to provide “correct” MIME type mappings with software (and this will often work around the problem with Windows), doing so is not correct in all cases because users may sometimes want non-standard mappings and it’s not possible to tell the difference between a mapping that has intentionally been set to an unusual value and one that was accidentally set that way.

Aspirationally, changes to the mimetypes API in Python (or similar ones in Go, I suppose) might improve the situation for many users. An option could be added that would tell the system to either ignore or prefer user-configured mappings, allowing applications to choose their preferred mode. However, doing so pushes complexity onto application authors in ways that probably cannot be predicted in all situations, which would mean the option would need to be exposed to application users to be set as required.

As best I can tell, the least-wrong solution to the situation of users often ending up with incorrect MIME type mappings on Windows is to make the problem more widely known so new software doesn’t misbehave in the same ways around writing bad mappings, and users have an easier time of correcting the problem if it does occur.²

Now you know!

I filed a bug against Django after doing this investigation myself, and found a proposal from 2008 that Django ship default MIME types for these purposes. It remains to be seen what the Django developers think of the situation. ↩︎
If APIs were changed to allow specifying the preferred source of mappings then better documentation would be needed anyway, so end users could be aware of the option if it were needed. ↩︎

tihle: a unique TI calculator emulator

Wed, 29 Jul 2020 09:00:00 +1000

Today I’m publishing tihle, a new emulator targeting TI graphing calculators (currently only the 83+, but maybe others later). There’s rather a lot to say about it, but here I will discuss the motivation for a new emulator and the state of the art followed by technical notes on the design and initial development process.

Yes, it runs Phoenix!

Read on for that discussion, or jump straight to the project homepage on GitLab which has a live demo that runs in your web browser and other resources including source code.

Motivation

I’ve long been involved in the community of people who program for the TI-83+ series of graphing calculators; it was on that platform that I got started programming in the first place. These days I play more of an advisory than active role in writing programs, by running much of Cemetech behind the scenes and providing the occasional input to others. I think calculators are an excellent way to introduce people to programming, since they are readily available devices and not too complex to get started on, while still having enough capability to support experienced developers in doing interesting things (because they are readily accessible embedded systems). I fully credit getting started with programming calculators for having ended up programming embedded systems profesionally.

Tim's workspace for CalcDCC looks like it could contain any embedded engineer's prototype.

Unfortunately, the future of programming for TI calculators seems to be in peril. About a month ago, the news came out that new versions of the OS for the TI-84+ CE (the most recent variant of the 83+ featuring a color screen and improved eZ80 processor) will remove support for running native code.

Death of a platform

While this doesn’t necessarily affect older calculators or programs, in the long term it seems to spell doom for calculators as an inroad to developing embedded software in particular. Programs written in the calculators’ dialect of BASIC continue to be accessible and a new Python implementation fills the void somewhat, but they lack in depth- where a user could spend time and effort developing native programs constrained only by the hardware they run on before, in the future users will be limited to only those capabilities provided by TI’s software.

In much the same way that I believe Scratch is a decent introduction to programming but completely hides interesting details and is not seriously used by anybody but those using it a learning tool, I also believe removal of support for running native code will ultimately mean people will no longer have calculators (which are often required school equipment!) as a useful entrypoint to serious programming.

Historical interest

In addition to the loss of a way to introduce people to programming, removing support for native code also effectively throws away a large existing library of programs that stretches back more than 20 years (nicely embodied in ticalc.org, which first came online in 1996). While the 84+ CE is a young platform relative to the TI-83+ series as a whole (and is incompatible with earlier software), it already has a rich library of programs created by users that will effectively be lost when they can no longer be run on the hardware they are designed for.

A small selection of the games that are available for the TI-84+ CE today: Calcuzap, Colossal Cave Adventure and Tetric A.

The precedent of effectively destroying the work of community members is troubling, and I am motivated to look for ways to preserve it. In much the same way that there still exist thriving communities around long-obsolete home computers like the Apple II and Commodore 64 today, I think it’s worthwhile to try to provide a similar opportunity by preserving the platform into a hostile future by working to give the systems and software ongoing life beyond what their creators envisioned (or perhaps more pointedly, beyond what they decided they could make money from).

As a newer platform than those early home computers that still have active communities, information and resources may be rather easier to come by for these calculators because much of the information was born digital and has always been online. However this is also hazardous to preservation, because if items are readily available online there may not be any replacements available if the original goes away. For instance, TI used to freely provide an SDK for the 83+, but have more recently made it much more difficult to access. Perhaps even more concerningly, TI’s web site no longer seems to provide any information about the TI-84+ CSE, seeming to deny that it ever existed (though manuals and software are still available if you know where to look).

A calling

Recognizing these concerns about the loss of a valuable resource for beginning programmers and loss of interesting history to the grind of the education-industrial complex,² what are we to do?

There exist a number of web sites that document the calculators and offer resources related to them- I’m already involved in that, which is valuable and generally well-preserved on the Internet Archive. But if the existing resources are useless on current hardware as they largely become when large classes of programs are not runnable, new tools become required: I believe this situation calls for emulation of the calculators, to make the platform accessible to everybody.

The state of the art

There already exist a number of emulators for TI calculators, including:

WabbitEmu, which is open-source and generally good quality.
jsTIfied, an emulator available as a web application.
Virtual TI, one of the earliest calculator emulators to be created.
PindurTI, which is now defunct but offered some useful capabilities in its day.
..and others

While most of these are accurate enough to run most programs, this is not the whole story: all of them require a ROM image to work. The ROM includes the calculator’s operating system and boot code, which implement huge swaths of functionality from keyboard and display handling through implementing floating-point arithmetic and the TI-BASIC interpreter.

TI jealously protect their OS and boot code because from the point of view of their business, the entire value of the calculator is in its software. If an emulator can run the same software on a more general machine, what reason is there for a user to purchase a $100 device from TI?³ But if we want to preserve community software for emulation, we don’t really care about the same functions that TI are interested in protecting.

Though truly accurate emulation still requires all OS functions be available in the same way, many applications only use a small subset of them. So what options are available for emulating the system software?

OS implementations

TI’s EOS (TI-OS) is provided with the calculator, and can be downloaded from TI in order to update a calculator. However, they added a clickwrap agreement to the OS in 2013 that attempts to forbid use of the OS for emulation. While that clause is questionably enforceable, nobody (myself included) is very interested in challenging it- mostly because TI has historically tended to turn a blind eye to the distribution of ROMs. Any such distribution in a larger, more public forum increases the risk of calling in the lawyers, so for these and the reasons discussed above, I do not believe it is tenable to use TI’s OS for general-availability emulation.
Brandon Wilson’s OS2 seems most relevant, in that it wants to reimplement TI-OS “but better.” However, it hasn’t been updated in 11 years at this point and seems dead. It’s also dependent on the assembler and tools provided by Zilog, which only run on Windows and are closed-source.
GlassOS is implemented in C and actually seems useable, but lacks compatibility with any existing software.
KnightOS is UNIX-like and seems to have a selection of useful software, but again is not compatible with any existing software beyond a small library of ports targeting it.
Many other people (myself included) have attempted to write OSes, but they don’t merit mention here because they’re of even less practical use.

Boot code

While the calculator OS implements most functionality, the boot code is also relevant- TI’s OS uses some support code from the boot code, and of course the boot code is responsible for handling the system power-up sequence and allowing recovery from a no-OS situation.

The boot code on these calculators is meant to be read-only,⁴ and is programmed into the device at the factory. Thus, there is no official source for boot code short of reading it from the memory of a physical calculator and the community consensus tends to be that sharing copies of the boot code would infringe on TI’s copyright.

Ben “FloppusMaximus” Moody at one time published a reimplementation of the calculator boot code called “BootFree,” which seems to have disappeared from the Internet except for the version integrated with WabbitEmu. While it seems BootFree disappeared for reasons related to the addition of the clickwrap agreement to the EOS downloads provided by TI,⁵ ⁶ I don’t think the same reasons apply to emulation of non-EOS software- thus, BootFree seems like a fine resource in pursuit of making emulation available to all.

Design

If the goal is to freely emulate calculator programs, we recognize that while the boot code situation seems compatible with accurate emulation, the current available libre OSes are insufficient. While it is technically feasible to emulate TI’s EOS (the only reasonable choice for emulating the current software library), legal forces make it untenable to use as a base for publicly-available emulation with the goal of allowing anybody to run programs for the platform.

It’s also worth noting that there are two major divisions in kinds of programs that run on these calculators: they can be implemented in the provided BASIC dialect (TI-BASIC), or distributed as machine code for the calculator. TI-OS implements the interpreter for TI-BASIC so both of these can be run on the existing emulators, but because using TI-OS is not an option for this project we need to choose how to approach things. The possibility space of emulation projects forks:

Build a TI-BASIC interpreter, not necessarily one that runs on an emulated calculator.
Create a whole-machine emulator that works without TI-OS.

While I’ve explored the feasibility of creating a from-scratch TI-BASIC interpreter in the past, it’s never made much progress and I got rather bogged down in accurate emulation of the calculator’s rather unique decimal floating-point format.⁷ I have instead chosen to work on a whole-machine emulator that doesn’t require TI-OS.

A novel emulator: thinking like a gamer

It is interesting to compare emulation of calculator programs to that of game systems (which are also common targets for emulation). In game systems the interest in emulation is almost entirely on the programs (games) that are separate from the system. Similarly, I am currently interested in providing emulation of third-party programs that happen to be run on a calculator.

Game systems tend to have small amounts of platform code that game software might use, which emulators tend to hide the existence of: a tiny minority of emulator users would have the capability to extract the firmware from a game system they own, and though most users probably engage in copyright infringement to obtain copies of game software to run on emulators, the emulators owe their existence to the fact that they don’t contain any copyright-infringing code themselves.

On calculators, emulation of interesting programs with existing emulators requires a copy of the system software which notionally requires access to a physical calculator to obtain. This means the emulators can reliably be considered not to fall afoul of any laws, though the barrier to entry is raised. However, the programs themselves are almost entirely made available freely by their authors- among the hobbyists who write calculator software, the platform is largely open. If an emulator can do without a ROM image of a calculator, it is feasible to freely provide the existing library of software to all comers.

Game system emulators tend to solve their inability to distribute the system firmware through high-level emulation: various operations that may be taken by software can be recognized, and the emulator can implement those operations itself. This approach both avoids any dependency on system firmware for its implementation and can often achieve better performance because the emulator need not faithfully emulate the underlying process- only its side effects are in scope. Though high-level emulation tends to sacrifice accuracy in ways that can cause some emulated software to behave incorrectly, this is often because the emulator must take shortcuts to achieve acceptable performance. When emulating a calculator with a 40-year-old CPU architecture that runs at a few Megahertz, acceptable performance should be much easier to achieve without noticeable shortcuts.⁸

With the precedent of existing emulators that are very faithful to the known hardware behavior and decision that no existing calculator OS is appropriate for the application, tihle is born. The name indicates how important that feature is to its existence: it is a high-level-emulator for TI calculators.⁹

Implementation, or: the development log

With a plan in mind, I had to start building something. I could have started with the core of one of the existing emulators, but had a few reasons to instead start from (mostly) scratch:

Code quality is highly variable, and I’m not familiar with their code. I have no idea what kinds of bugs and pitfalls may exist in them.
Support for high-level emulation would require an unknown amount of modification.
Portability is questionable, since they’re mostly implemented in various flavors of C or C++ and typically have Windows as the primary target system.

I’m a big fan of Rust and know that portability of Rust programs is pretty good because the language implementation’s standard library does a pretty good job of abstracting away platform details. In addition, I know the compiler has good support for WebAssembly as a platform, which should make it reasonably easy to make a port that runs in a web browser. Running in a web browser is desirable because it is the most accessible way to allow people to emulate games- see for instance the Emularity, which powers emulation of thousands of programs on the Internet Archive.

Cores

Though I chose to start from scratch with an emulator written in Rust, I still didn’t want to spend the effort to implement CPU emulation from scratch. Since the Z80 is such an old CPU and so widely used, there are a number of existing emulators that I might be able to use the core from. Rust is easy to link against C code, so there are many options to consider. Among the most notable:

YAZE-AG is very mature and emulates a whole CP/M system.
Fuse is also very mature and emulates the ZX Spectrum. It also has a reputation for high accuracy thanks in part to its comprehensive test suite.

The Visual 6502 project has a gate-level Z80 simulator that could yield ultimate accuracy at a "blistering" 5 Hz simulated clock speed! It's not feasible to use.

I eventually settled on a core that’s not part of any major emulator: Manuel Sainz de Baranda y Goñi’s “redcode” core. It’s written in very portable C and designed to be used as a library so it takes very little setup or modification to use in my application. It does depend on a large external header-only library as provided, but I was able to adapt it to be more standalone with some work.

I tend to prefer to use permissive licenses on my software, but the redcode Z80 core is GPL-licensed. Since linking it into my binaries would require my code to also be GPL I would prefer to use a core with a more permissive license, but accepting GPL in exchange for not needing to implement the core myself seems like a fair and expedient trade on balance. In the future I might choose to implement my own core and change the license for tihle, but for the time being it will be copyleft.

Taking emulation up a level

Building the CPU core and getting it to run some code wasn’t terribly hard, which left the core issue of how to implement high-level emulation. The emulator needs to be able to recognize when the CPU needs to “trap” into operations provided by the emulator.

I initially defined traps as simply taking some action on instruction fetch. When the CPU would attempt to read memory at chosen addresses, the emulator would execute a trap handler based on the memory address being read, then return data equivalent to instructions appropriate to the trap- usually just the value C9 for a ret instruction.

This approach quickly turned out to be problematic as I was testing the emulator. I chose Phoenix as the target program for initial development because it’s a sort of classic calculator game and as a bonus shouldn’t depend too tightly on emulation accuracy- Phoenix was designed to be portable across a number of calculators, so hopefully doesn’t contain many assumptions about the system.

A few screenshots of Phoenix.

Unfortunately, I quickly ran into problems with the way Phoenix generates random numbers. Its FAST_RANDOM routine uses some self-modifying code to read values from semi-random memory addresses, mixing the read values through a few shifts and xors. The initial address it reads is the CPU reset vector; 0x0000, which I had also defined as the trap that terminates emulation. To work around this I modified reset detection: instead of simply trapping reads from that address, I made the “terminate” flag instead be controlled by any write to port 255, which is unused on the calculator. By putting code at the reset vector to write to port 255, we can avoid spurious termination.

It quickly becomes obvious that treating termination separately from other traps doesn’t work well. If the program decides to read a value from 0x0028 for instance (where control flow jumps to execute a system routine), it would cause incorrect actions to be taken which would almost certainly cause the program to begin behaving incorrectly. What’s required is a way to ensure traps are only taken when executing the target code, not if the system happens to read a given memory address as data. It wasn’t an immediate problem, but this will be revisited later.

MirageOS

The initial implementation of the memory subsystem only emulated RAM, not the Flash memory in the calculator that holds the OS. Regular programs only execute out of RAM, so it was easy enough to trap on any access to Flash. This became problematic once emulation of Phoenix reached a certain point, because it turns out I had chosen to use the MirageOS version of Phoenix, meaning there was another dependency on the contents of Flash that I had not expected.¹⁰ MirageOS¹¹ is a shell implemented as a Flash application; it runs directly from Flash, and provides useful support routines to programs designed to take advantage of it.

This version of Phoenix uses the setupint routine provided by Mirage to implement its timer interrupt that controls game speed. Since Mirage isn’t open-source, I would have had to either reverse-engineer the details of how that works or simply map Mirage into memory. I opted to do the latter, since Mirage is freely redistributable and it will always be easier to emulate it than faithfully implement its functionality in terms of emulator traps.

Hoisted by my own bugs: a debugging adventure

After adding Mirage to the system, things still weren’t working. I spend some time reverse-engineering setupint and by painstakingly comparing that with execution traces from the emulator, the flow that routine was taking seemed reasonable. It looked like it was jumping somewhere incorrect on servicing the first interrupt after setting it up! Checking the value of the I register that specifies the location of the interrupt vector table to the core, it didn’t seem correct- meaning there was a problem with the core. Let’s dive into that adventure (or skip ahead to the next section if you prefer not to learn some about my debugging process):

Mirage installs its interrupt vector table to the block of memory at 0x8B00, filling it with the value 0x8A. Because custom interrupts on the 83+ must always use the Z80’s interrupt mode 2 but the calculator is not designed to use this mode, this ensures that all interrupts will vector to 0x8A8A. Breaking execution at 0x71D2 where it loads the I register:

1
2
3
4
5
6
7
8


ROM:71CE ld hl, 8B00h ; Interrupt vectors go at 8B00
ROM:71D1 ld a, h
ROM:71D2 ld i, a
ROM:71D4 dec a ; A = 8A
ROM:71D5 ld bc, 101h
ROM:71D8 rst 28h ; Fill 256 bytes of 8A at 8B00
ROM:71D8 ; ---------------------------------------------------------------------------
ROM:71D9 dw 4C33h ; MemSet

As expected, HL was 0x8B00 and A should have been 8B, but instead it had value 0x84 and after the ld i, a instruction the value of I hadn’t changed!

Digging deeper, I had to debug the emulator- not just inspect the emulated CPU state. This was actually more difficult than expected, because I had been developing with CLion on Windows, using a Rust toolchain targeting the MSVC tools. CLion only supports debugging with GNU tools however, so I couldn’t use the integrated debugger. I do have a copy of Visual Studio handy however, which works just fine to debug MSVC binaries.

I ended up adding a pause to the emulator on startup so I could launch it through CLion, then switch to Visual Studio to attach the debugger and set breakpoints before unpausing the emulator. This worked quite tolerably, since Visual Studio was still able to make sense of all the debug symbols in the binary so the only tedious part was in getting the debugger attached.

With everything hooked up, I saw the following state in my debugger:

By inspecting the assembly and comparing with the debugger’s view of the locals, I can tell that rcx points to the emulator object, and offset 0x40 from that refers to the state of the CPU core. PC of the core has already been incremented to the next instruction, and the intent seems to be that the value of the emulated A register will be tansferred through ecx into I. However, the value it’s reading is 0, not 0x84 as it should be based on the debugger’s view of the AF register pair.

The redcode emulator uses some utility types that I reimplemented when I didn’t want to include the whole “Z” library that it wants to use, which include the Z16Bit union used to represent the 16-bit register pairs. It looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


typedef union {
 zuint16 value_uint16;

#ifdef IS_BIG_ENDIAN
 // 0 is the low-order byte of these aggregates
 struct {
 zuint8 index1;
 zuint8 index0;
 } values_uint8;
#else
 struct {
 zuint8 index0;
 zuint8 index1;
 } values_uint8;
#endif
} Z16Bit;

The intent of this type appears to be in allowing easy access to each byte of a 16-bit value; this could be done just with shifting and masking that presumably the compiler could optimize, but I suppose it was designed to not depend on compilers being efficient. In this case, it seems the program is actually reading the wrong byte of the value; when it contains 0x8400 and we want to read the value of A, it should take the high byte of the value but is instead taking the low. The macro A in the source code for the core expands to an access to object.state.af.values_uint8.index0 which should be the high byte of the AF register pair.

So it turns out I made a mistake in adapting that definition to my needs, and index0 is meant to be the high-order byte of the aggregate value, not the low! Simply swapping the locations of index1 and index0 then rebuilding, the state was then correct:

Not only does A now have the value we expect from reading the Mirage disassembly, ecx now does contains 0x8B and the value is correctly written to I. Problem solved!

Rethinking traps

With improved confidence in the core’s correctness now, emulation can progress to the point where it’s relevant to resume thinking about how to correctly implement traps such that non-instruction reads from memory won’t spuriously trigger them. With the change in reset handling to trap on writes to a particular port, it makes sense to explore the ways in which traps could better be handled.

It’s likely that there will need to be a number of different traps; certainly for a handful of system routines. This means any trap method must be able to provide a way to identify which trap to execute, but it must only be triggered when executed by the CPU. Some ideas come to mind:

Write a value to a selected port that’s not otherwise used on the calculator.
Provide a flag from the core to the emulator for handling memory reads to indicate an instruction fetch rather than any other data read.
Implement a custom instruction that causes a trap.

Modifying the core to handle memory reads differently has unknown complexity so I didn’t want to do that, and writing a value to a port didn’t seem like a very good choice because it can only accomodate 256 values per chosen port. While I don’t know how many traps will eventually be needed, 256 seems like it may not be enough so I opted to look into creating a custom instruction.

Looking at what instructions are defined in both the Z80 and eZ80 documentation (leaving the door open for eZ80 emulation later), there’s a lot of unused space in the ED-prefixed instructions that isn’t used by either CPU and isn’t known to have useful undocumented effects.

According to Jacco Bot, ED00-ED3F and EDC0-EDFF all have no discernable effect so are good candidates for custom instructions. Cross referencing with the eZ80 CPU manual (Zilog UM0077, table 109), much of the first block is filled in while the second remains sparsely used.

The `ED`-prefixed instruction space for the eZ80 CPU.

Choosing semi-arbitrarily, ED25 seems like a nice TRAP instruction, since 25 is an ASCII ‘%’ which seems like it might “pop” out of a text representation of memory a little bit.

To indicate which trap should be taken, I chose to have the instruction include a 16-bit value which indicates to the emulator which trap is desired. This provides a very large number of possible traps that ought to be sufficient for any future needs.

Implementing this new instruction turned out to be very easy, and I learned a little more about how the core is implemented. I simply needed to add a hook for traps (to call from the C core into my Rust emulator) and add a new function that calls that hook, then insert a pointer to that function in the table of ED-prefixed instruction handlers.

1
2
3
4
5
6


INSTRUCTION(ED_tihle_trap) {
 PC += 4;
 uint16_t trap_no = (READ_8(PC - 2) | (READ_8(PC - 1) << 8));
 CYCLES += object->trap(object->context, trap_no);
 return 0;
}

The need for an OS

Now that traps are actually a special instruction, it turns out the system needs some kind of OS image! Without one, there’s no way to trigger traps on calls into OS code. As a quick solution, I implemented a tiny OS image that traps on the major OS entrypoints:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


; Trap instruction: ED25 + 16 bit parameter
.addinstr TRAP * 25ED 4 NOP

#define TRAP_RESET 0
#define TRAP_BCALL 1
#define TRAP_OS_INTERRUPT 2

.seek $0000
 trap TRAP_RESET
 rst 00h

.seek $0028
 trap TRAP_BCALL
 ret

.seek $0038
 trap TRAP_OS_INTERRUPT
 reti

The important traps here are for calls to OS routines at 0x0028, and the default interrupt handler at 0x0038. The handler for TRAP_BCALL inspects the CPU state to choose what system routine to emulate, so it provides most of the OS functionality that any program needs.

Debugging in pictures

With this work, the general structure of the emulator seems sound. Although I didn’t want to implement an OS to begin with, one is now present; however though its complexity is strictly controlled according to whether I find it easier to implement routines in Z80 or as traps, so this seems reasonable. It was then a matter of implementing and debugging the core functions that Phoenix needs.

For some parts of debugging it was easier to write small programs that exercised only the function(s) that needed debugging; these might be good to promote to unit tests in the future.

Text rendering seemed partially correct; it turned out I was computing an 8-bit index into the font bitmap rather than pointer-sized, so getting the character to display was wrong for most characters.

Displaying text is not particularly hard, though getting a copy of the bitmap font that the calculator uses was a bit of work. I ended up getting each character as an image, then writing some Python scripts to combine them into a binary blob that can be embedded in the emulator. It nearly worked with only a little work, but I did have to debug some bad computations stemming from getting an 8-bit character value and integer rollover.

The title screen was a little too enthusiastic but tamed with some changes to correctly emulate undocumented behaviors.

My initial traps for the _PutS system routine failed to emulate the undocumented behavior that it updates HL to point past the string that is displayed. Text output in the small font also had some problems with updating the screen coordinates to draw at. The vertical lines in the small-font text were apparently an error in my font bitmap data that was easily fixed, though the black line at the top of the screen was a different problem.

Fixing some of the LCD bugs shows Phoenix wants to display white on black, and shows there remain a few bugs to be solved.

My LCD emulation¹² had some issues to fix around not being in the correct mode initially and failing to update its addresses correctly in some situations. It turns out Phoenix is somewhat unusual among games for the 83+ in that it directly accesses the LCD for all its display operations, rather than delegating to a library function provided by the OS (_GrBufCopy) or a shell (fastcopy) which makes it a good test of basic LCD driver emulation. The display warping was a simple problem of having the wrong display width parameter in the driver emulation.

With the display looking largely correct, the final major hurdle to something that looks like a game was in generating timer interrupts, which Phoenix uses to control the game speed. This required implementing a few control ports which the CPU uses to enable or disable interrupts and ensuring that interrupts would be triggered at the correct times accurately, but wasn’t too bad although I spent some time vexed by a bug where IRQ flags never got reset so interrupts fired continuously.

With working interrupts, the menu looks as it should!

Handling input

In order to make the game playable, the last piece of the puzzle is in allowing it to receive input. On a physical calculator this is through a simple matrix keyboard, interfaced directly to the CPU via I/O port 0. Where some complexity comes in is in how the calculator OS provides its input abstractions, which turns out to be important but not well documented.

TI-OS provides a routine for scanning the keyboard, called _GetCSC. It returns one value, a scan code corresponding to the key (if any) that is currently pressed. The documentation leaves it at that however, when the behavior around multiple keys being pressed or keys being held is also important.

It turns out that _GetCSC is implemented largely via interrupts; while servicing regular timer interrupts, the OS scans the keyboard for keys that are being pressed, and if any are then stores that value in RAM. _GetCSC reads that value and clears it. If everything were that simple it would be very easy, but the interrupt handler also debounces some keys; the directional arrows and del key can repeat if held, while others will not. The actual timing is that the first repeat of a held key comes after about 48 interrupts, and every 10 thereafter meaning the first repeat comes after about 250 milliseconds and subsequent ones occur at 20 Hz.

While OS2 contains an implementation of this logic (which seemed overcompilicated until I actually understood how this key repeat works), I opted to implement keyboard scanning as a trap instead. While the keyboard hardware is fully emulated because Phoenix only uses _GetCSC for a few things and directly interfaces with the keyboard for the rest, the OS timer interrupt calls into the emulator to do keyboard scanning because it’s easier to implement and probably more performant.

Without correct key repeat control the menus work but are too fast to control; afterwards, the game is perfectly playable!

With this, I’m rather surprised that Phoenix is playable! There’s certainly more work to be done to improve everything, but “Phoenix is playable” was the goal I set for myself that must be reached before I published the project; so.. here we are.

Current state and future

Currently, I’ve really only tested Phoenix. It’s possible that some other games will work in tihle as it stands, and others might work with only minor changes. There are notable holes that I’d like to fill:

Programs are loaded directly to the point in memory that they execute from. Programs and data should be loaded into memory as if they were managed normally by TI-OS, which will allow programs that want to be able to load external data (such as level packs) to do so, and also enable saving or loading other data, such as high scores or generated programs.
Rather than directly launching programs, launch a shell and allow users to interactively select a program to run. (Though a mode to directly launch programs may still be desirable.)
Improve LCD emulation to support greyscale graphics. Programmers sometimes implement rapid switching between black and white to approximate several shades of grey (4-16 depending on how ambitious they are and how much CPU time is worth dedicating to it for the application), which will require LCD emulation to have some “memory” in order to simulate the slow response time of the calculator LCD.
Support an on-screen keyboard, useful for touchscreen devices and easier to navigate than pressing keys on a computer keyboard.
Improved debugging tools, to set breakpoints and more interactively explore the system state. Very useful for debugging new programs that don’t yet run correctly, or for developing new programs.
Your suggestion here! (Suggestions and feature requests via Gitlab issues or other means are welcome.)

I’m quite happy with the ability to run the emulator in a web browser as it is right now. While I can’t claim much credit for it beyond choosing tools that would support that target,¹³ it’s still very gratifying to see everything come together to make emulation so accessible. In the longer term, I’d like to try to get emulation via tihle available on the Internet Archive, like many DOS games are today.

The “minimal” OS that’s included with the emulator is currently a pretty significant hack that I’d like to clean up some, and in the long term it would be cool to make it a much more complete TI-OS reimplementation- possibly taking in most of OS2, and building on top of Brandon’s work. In an ideal world the high-level parts of tihle might eventually become obsolete (or optional) if the OS implementation got good enough.

Links and further reading

tihle lives on GitLab: https://gitlab.com/taricorp/tihle

There you can read more, download binaries to run tihle yourself (or do so directly in your browser), read the source code as well as contribute: I welcome contributions, suggestions and discussion, especially if they come with code!

You can learn more about the technical details of calculators and programming them at WikiTI, though many of the details can be rather inscrutable without additional explanation or prior knowledge. The official TI-83 Plus SDK documentation can be found on the Internet Archive; much of its contents are omitted from WikiTI, so it’s a valuable source of basic information. Learn TI-83 Plus Assembly in 28 Days is a decent introduction to the platform, intended for readers with no experience programming in assembly but plenty usable by more advanced readers as well, with its combination of simplified official documentation and community-sourced information.

If you’re looking for discussion or programs, Cemetech is one forum where experienced calculator programmers (and me!) hang out, and are happy to discuss things. There’s also an area to find programs there, but ticalc.org is one of the oldest calculator community sites, and has a huge library of software to browse.

I thought I might give this a try after Gravis commented on doing something similar; it can benefit readers who don’t use assistive technology but are better able to consume the content in a different format. ↩︎
Most users of these calculators have them because their schools require them, and schools tend to require them because TI spend significant effort in selling the calculators to schools and positioning them as tools for standardized testing. ↩︎
Most users only need a physical calculator because more capable devices (phones and computers) are not permitted in standardized testing! ↩︎
I understand the write-protection of the boot code on many calculators is not as robust as expected from understanding the hardware’s capabilities such that it can actually be modified from software only, but the fact remains that it’s not intended to ever be updated. ↩︎
WabbitEmu added support for automatically downloading an EOS image from TI and combining it with a copy of BootFree to make a fully-functional ROM without ever touching a real calculator sometime in 2010. This seems to have flown under TI’s radar for a few years, possibly until the release of jsTIfied in 2012 brought emulation more into the mainstream. Where previously emulation had a somewhat higher barrier to entry, the availability of an emulator running entirely in a web browser combined with the possibility of automatically making a complete software image from freely-available resources seems to have motivated TI both to begin more aggressively marketing their own emulator (TI-SmartView) and place barriers in front of community emulation in the form of clickwrap in order to extract licensing fees from educators who might otherwise use free emulators. ↩︎
Anecdotally, Cemetech saw a number of new users following the release of jsTIfied who were clearly in education because for a time it required users to register accounts on the site to use it (a choice related to a technical limitation that some operations required data be sent to the server and bounced right back). I assume there were a number of teachers who might have instructed their students to use jsTIfied in order to avoid any extra costs in accessing the calculators required by their curriculum. ↩︎
I now note that accurate emulation of that particular aspect is likely unnecessary, but haven’t revisited that concept because I chose to think about the whole-machine emulation. ↩︎
Concretely, the TI-83+ CPU runs at 6 MHz and a Z80 cannot execute any instructions in less than 4 clock cycles (and most take more than 4, possibly as many as 23). A CPU in a general-purpose device today might run at 3 GHz and achieve average throughput of two instructions per cycle, meaning an emulating machine today is comfortably 1000 times faster than the Z80 it emulates (and maybe more like 5000 times faster depending on the workload!). ↩︎
Some readers might ask: “why not improve an existing libre OS like OS2?” My answer is that doing so is too hard: while the Z80 is a well-understood CPU to program for, its age means much of the tooling available to us today that makes programmers more productive is simply not available. While a truly free (libre) EOS replacement would be ideal, I do not consider it to be immediately feasible. ↩︎
On the 83+, the Z80’s 64 KiB memory space is split into 4 banks. Typically, page 0 of Flash (which contains OS code) is always mapped into the low 16 KiB, “bank A” from 0x4000-0x8000 can be swapped to contain different parts of Flash, and the top 32 KiB is RAM. Flash applications like Mirage execute of out bank A. ↩︎
Not to be confused with the library operating system of the same name, which I also have some experience with. ↩︎
On the 83+ the LCD driver is controlled through two ports from the CPU. Other calculators like the TI-85 have memory-mapped displays, but the 83+ series requires the CPU to push data to and from the display and emulators must understand most of the LCD driver’s command set in order to work correctly. ↩︎
I chose to implement the frontend to the emulator with SDL specifically because I knew emscripten has an SDL port that allows software using it to run in a web browser without major modification; other people have already done the heavy lifting to make Rust+SDL applications work in browsers, I just knew I’d be able to make use of it. ↩︎

fb-hitler Updated

Tue, 28 Jul 2020 11:14:33 +1000

I updated fb-hitler a few years ago and never wrote about it; today I spent some time to describe the new version, which is implemented more robustly: now it’s a bootable image that runs in real mode, not a Linux application.

QEMU looks like this when running the program.

Read more on the project page or GitLab.

A Wayback Machine Bookmarklet

Sat, 21 Mar 2020 09:37:39 +0000

Sometimes I find it useful to be able to quickly save a page to the Wayback Machine, often to be able to provide a stable link to a page that I don’t control- for instance if I’m pointing somebody to a document that describes something they’re asking about, then it’s nice to ensure that there will still be an archived copy if the original goes away.

While the landing page on web.archive.org has a “save now” form to quickly save a page given its URL, this is still more cumbersome than I’d like- it involves copying the desired URL, opening a new tab to web.archive.org, pasting the URL into the form and pressing the save button- in exactly that order.

The concept of bookmarklets comes to the rescue: little snippets of Javascript in a bookmark to do some function- it’s easy enough to open a new tab to a given URL with some javascript, so by inspecting how the “Save page” button works, I can automate it:

javascript:void(window.open('https://web.archive.org/save/'+location.href));

I’ve put that string into a bookmark that sits on my browser’s bookmarks bar, so I can just click on that to open a new tab which will save the currently-shown page to the Wayback Machine in a new tab. Much easier!

Of course this is still cumbersome to do with many URLs, such as if I want to archive all the links in a blog post. Fortunately AMBER automates that particular use case, and larger applications tend to be the realm of fetching WARCs with wget.

Fomu: a beginner's guide

Wed, 06 Mar 2019 00:00:00 +0000

FPGAs are pretty cool pieces of hardware for tinkering with, and have become remarkably easy to approach as a hobbyist in recent years. Boards like the TinyFPGA BX don’t require any special hardware to use and can provide a simple platform for modestly-scoped projects or just for learning.

While historically the software tools for programming FPGAs are proprietary and provided by the hardware manufacturer, Symbiflow (enabled and probably inspired by earlier work like Project IceStorm) provides completely free and open-source tooling and documentation for programming some FPGAs, significantly lowering the cost of entry (most vendors provide some free version of their design software but limited to lower-end devices; a license for the non-free version of the software is well into the realm of “if you have to ask, you can’t afford it”) and appearing to yield better results in many cases.¹

As somebody who finds it fun to learn new things and experiment with new kinds of creations, FPGAs are quite interesting to me- they’re quite complex devices that enable very powerful creations, with excellent depth for mastery. While I did some course lab work with Altera FPGAs in university (and a little bit of chip design/layout later), I’d call those mostly canned tasks with easily-understood requirements and problem-solving approaches; it was sufficient to familiarize myself with the systems, but not enough to be particularly useful.

The announcement of Fomu caught my interest because I was aware of the earlier Tomu but wasn’t sufficiently interested to try to acquire any hardware. With Fomu however, I’m rather more interested because it enables interesting capabilities for playing with hardware- others have already demonstrated small RISC-V CPUs running in that FPGA (despite its modest logic capacity), for instance.

Even more conveniently for being able to play with Fomu, I’ve been in contact with Mithro who is approximately half of the team behind Fomu and gotten access to a stockpile of “hacker edition” boards that have been hand-assembled but not programmed at all. With slightly early access to hardware, I’ve been able to do some exploration and re-familiarize myself with the world of digital logic design and figure out the hardware.

Hardware

In summary, Fomu is a small (9.4 by 13 by 0.6 millimeters) circuit board with a Lattice ICE40UP5K-UWG30 FPGA, a 16-megabit SPI Flash for configuration (and other data) storage, a single RGB LED for blinkiness and a 48 MHz MEMS oscillator to provide a clock.

Board component side. The other side is mostly just the USB pads.

The whole thing is built so it can fit inside a standard USB port. Production boards are meant to ship with a USB bootloader that allows new configurations to be uploaded to the board only via that USB connection, but hacker boards are provided completely unprogrammed (and untested).

Schematic

Before we can make the hardware do something, we’ll need to understand how everything is put together:

Unfortunately, this schematic leaves some things to be desired. While it does allow us to see what parts are actually on the board and generally how they’re connected, it fails to clearly mark the external connections- power and data lines on the USB connector, test points and utility I/O pads.

The USB connections are easy to figure out, however; it’s a standard pinout so we can easily identify which physical pads on the board correspond to VUSB (5V supply), ground and the two data lines (USBP, USBN). Rather trickier to work out is the function of each of the test points on the board, though there is a provided template for laser-cutting a programming jig which provides some hints:

Here red and green lines are cuts, while black is raster engraving for marking.

This jig is meant to be built up by stacking four layers of material, engraving a small pocket in the bottom to hold the board to program and inserting pogo pins in the small holes to contact with the test points on the board. This template helps us in that it has labels for the test points, though! All of the test points are clearly identified, except it’s unclear what voltage is expected on the power supply.

By inspecting the board myself, I eventually determined that the test point for supplying power (marked VCC on the programming jig template) is downstream of the 3.3V regulator (not connected to the USB power supply pad) so it expects 3.3 Volts for programming.

Convenient pinout diagram

By way of improving the schematic, here’s that same photo of the board with the signal names from the schematic pointed out on each of the pads, and the individual chips pointed out.

And the same in tabular form for easy searching:

Silkscreen	Schematic	Description
V	+3V3	3.3V rail
S	CS	SPI chip select (active low)
C	SCK	SPI clock
I	MISO	SPI MISO
O	MOSI	SPI MOSI
R	CRESET_B	FPGA reset (active low)
G	GND	Ground
1	PIN1	User I/O 1
2	PIN2	User I/O 2
3	PIN3	User I/O 3
4	PIN4	User I/O 4

Bootstrapping

My first task in attempting to bootstrap a board and load some configuration on it was building a programming jig. Given there was already a template for a laser-cut acrylic one and I have access to a benchtop laser cutter, this was easy:

It’s a little bit ugly because the pogo pins I had ready access too are too small to nicely fit in the laser-cut holes so I had to carefully glue them in place.

Actually programming a board can be done with the fomu-flash utility running on a Raspberry Pi. I conveniently had a Raspberry Pi 2 to hand, so a little wiring to the Pi’s GPIO header had a jig that should work. Unfortunately, it didn’t- all I got out when trying to make it identify the on-board flash chip was 1s:

1
2
3
4
5
6
7
8
9


$ fomu-flash -i
Manufacturer ID: unknown (ff)
Memory model: unknown (ff)
Memory size: unknown (ff)
Device ID: ff
Serial number: ff ff ff ff
Status 1: ff
Status 2: ff
Status 3: ff

I gave up on that hardware after spending a while experimenting with it, and decided to design a custom programming jig that might be a little easier to ensure pin alignment is good. This is a little bit tricky because the minimum pitch of the test points is just 1.8 mm, which is not large enough for the 0.1-inch (2.54 mm) DuPont connectors commonly used for prototyping and desirable in this case because they’re very easy to connect to the Raspberry Pi’s GPIO header.

A better jig

Fortunately, I also have access to rather sophisticated prototyping tools and had some nice parts handy from other projects. In particular, a Form 2 stereolithographic 3D printer and some good pogo pins, Preci-dip 90155-AS. I computer-modeled a jig to be 3d-printed that should be both compact and robust, pictured below (see the end of this post for downloadable resources):

Isometric view with all edges visible

Top view

Bottom view

Features

The design takes great advantage of the flexibility of 3d printing for fabrication: it is easy to install the pogo pins off-vertical by making the (press-fit) holes at an arbitrary angle; this would be very difficult with conventional fabrication, but it allows the spacing of the pins at the top of the jig to be large enough that 2.54mm connectors can be used, despite the pad spacing on the board being only 1.8mm.

On the bottom side, there are several narrow features that act as a shelf to support the board (which is 0.6mm thick) so its outside surface is flush with the bottom surface of the jig. A semicircular boss on one side mates with the cutout on the PCB to key the jig so it is obvious when the board is correctly oriented in the jig. A small cutout on one edge allows a tool to be inserted to pull the board out if needed, because the fit is close enough that it might stick sometimes (or a tool could be pushed through from the top).

As a manufacturability consideration, the top surface has a slant between opposite corners. This improves the print quality on a Form 2- because dimensions on that side are not critical the part is designed to be printed with that side “down” (actually up, once in the printer) and supports attached to it on that end. By allowing the printer to gradually build up a slope rather than immediately build a plane, it can better produce the intended shape- an earlier version of the design with a flat top had a very rough finish because large and thin layers of material tend to warp until enough material is built up to be self-supporting.

The choice of pogo pins in particular is key, since they’re made with a small shoulder and retaining barbs that allow them to be easily press-fit into a connector shell:

The one downside of these pins is the short tail, intended for mounting to a circuit board. While the aforementioned DuPont connectors can be mated to the tail, they are not very secure and come off at the slightest force. A revised design choosing parts for their function and not just immediate availability might prefer to use a part like 90101-AS, which is intended for wire termination rather than board mounting- then wires can be securely attached to the pin rather than tenuously placed on it. My workaround that didn’t involve buying more parts was carefully gluing the wires in place, which seems to work okay.

Programming

Having built a jig that I could be confident would work correctly, we now return to the problem of actually programming the board. Connecting the new jig to my Raspberry Pi in the same way I did the first one, it failed in the same way- reading all 1s.

At this point I was rather stumped, with a few possible explanations for the problems:

Both jigs are unreliable
I’m wiring the jigs up incorrectly
Software on my Pi is configured incorrectly
All of the Fomus I tried were faulty

To discount the first two possibilities, I was able to borrow Mithro’s professionally-built jig that already had a Raspberry Pi 3 connected to it. I didn’t have any credentials to log in to that Pi and use it interactively however, so I was limited to checking its wiring and carefully ensuring I connected my Pi to the jig in the same way, then try programming again. This also failed.

Mithro’s jig. It seems very cleverly built to me, clearly designed by somebody with a lot of experience designing these kinds of fixtures.

Having tried that I had to assume my Pi was somehow misconfigured, since it seemed increasingly unlikely that I was doing anything wrong and it seemed implausible that all of my boards were faulty. I eventually took the SD card out of the other jig’s Pi and inspected the software it would run by connecting it to another computer. This amounted to the same fomu-flash program I was using, so I inspected the system configuration in /boot/config.txt and found a variety of non-default options that seemed plausibly useful. Ultimately, I found some magic words:

1

dtparam=spi=on

This option makes the kernel on the Pi expose a hardware-assisted SPI peripheral, which seems like an obvious missing option until you realize that fomu-flash actually bit-bangs SPI because the hardware support is insufficient for this application. In any case, I did find that turning that option on makes everything work correctly:

1
2
3
4
5
6
7
8
9


$ fomu-flash -i
Manufacturer ID: Adesto (1f)
Memory model: AT25SF161 (86)
Memory size: 16 Mbit (01)
Device ID: 14
Serial number: ff ff ff ff
Status 1: 02
Status 2: 00
Status 3: ff

I reported the bug and made a note of this in the documentation so hopefully nobody else has to deal with that problem in the future, even if the root cause is mystifying.

Success!

With the ability to talk to the configuration flash, it’s then possible to write an actual bitstream. To avoid needing to write one myself, it’s easy to take the LED blinker example from the fomu-tests repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


fomu-tests/blink$ make FOMU_REV=hacker
...

Info: constrained 'rgb0' to bel 'X4/Y31/io0'
Info: constrained 'rgb1' to bel 'X5/Y31/io0'
Info: constrained 'rgb2' to bel 'X6/Y31/io0'
Info: constrained 'clki' to bel 'X6/Y0/io1'
Warning: unmatched constraint 'spi_mosi' (on line 5)
Warning: unmatched constraint 'spi_miso' (on line 6)
Warning: unmatched constraint 'spi_clk' (on line 7)
Warning: unmatched constraint 'spi_cs' (on line 8)
Info: constrained 'user_1' to bel 'X12/Y0/io1'
Info: constrained 'user_2' to bel 'X5/Y0/io0'
Info: constrained 'user_3' to bel 'X9/Y0/io1'
Info: constrained 'user_4' to bel 'X19/Y0/io1'
Warning: unmatched constraint 'usb_dn' (on line 13)
Warning: unmatched constraint 'usb_dp' (on line 14)

...

Info: Device utilisation:
Info: ICESTORM_LC: 33/ 5280 0%
Info: ICESTORM_RAM: 0/ 30 0%
Info: SB_IO: 5/ 96 5%
Info: SB_GB: 1/ 8 12%
Info: ICESTORM_PLL: 0/ 1 0%
Info: SB_WARMBOOT: 0/ 1 0%
Info: ICESTORM_DSP: 0/ 8 0%
Info: ICESTORM_HFOSC: 0/ 1 0%
Info: ICESTORM_LFOSC: 0/ 1 0%
Info: SB_I2C: 0/ 2 0%
Info: SB_SPI: 0/ 2 0%
Info: IO_I3C: 0/ 2 0%
Info: SB_LEDDA_IP: 0/ 1 0%
Info: SB_RGBA_DRV: 1/ 1 100%
Info: ICESTORM_SPRAM: 0/ 4 0%

...

Built 'blink' for Fomu hacker

$ fomu-flash -w blink.bin
Erasing @ 018000 / 01969a Done
Programming @ 01959a / 01969a Done
$ fomu-flash -v blink.bin
Reading @ 01969a / 01969a Done

Programming that to the board yields a blinking LED as expected, so I’ve achieved success in the basic form of this project by getting the FPGA to do something. Further exploration will involve writing gateware with Migen rather than straight Verilog (because I find Verilog to be rather tedious to write) and trying to build a system around a RISC-V CPU (because that sounds interesting).

Resources

If you want to make your own copy of the programming jig or just explore it, you’ve got several options:

View the model at OnShape. This will allow you to view and make changes to the parametric model, which is what you’ll need to make most useful changes to it.
View the STL online. A quick and dirty way to get an interactive view of the model.
Download the STL. If you just want to try to 3D print your own, this is all you need. It may also be useful if you want to make changes using a 3d modeling program (rather than a CAD program).
Download a Solidworks part file. This was just exported from OnShape, but you might prefer this if you want to use SolidWorks to edit the model.

All of the official documentation for Fomu is available on Github. For basic information (such as what I referred to when writing up this project), that’s a great starting point.

I designed the programming jig in OnShape which is a pretty good and very convenient CAD tool.

FPGA vendors don’t publish all the information required to build configuration bitstreams for their hardware, possibly because they wish to support their side business in selling design tool licenses- this despite the fact that (anecdotally, since I can’t recall where I saw it) many FPGA developers say that vendor tooling is one of the biggest annoyances in their work. The open-source tools require a fair amount of painstaking reverse-engineering of chips to create! ↩︎

Temperature Logging: Redux

Wed, 22 Aug 2018 00:00:00 +0000

Previously as I was experimenting with logging the temperature using a Raspberry Pi (to monitor the temperatures experienced by fermenting cider), I noted that the Pi was something of a terrible hack, and it should be possible to do more efficiently with some slightly less common hardware.

I decided that improved version would be interesting to build for use at home, since it’s both kind of fun to collect data like that, and actually knowing the temperature in the house is useful at times. The end result of this project is that I can generate graphs like the one below of conditions around the house:

Software requirements

My primary requirement for home monitoring of this sort is that it not depend on a proprietary hub (especially not one that depends on an external service that might go away without warning), and I’d also like something that can be integrated with my existing (but minimal) home automation setup that’s based around Home Assistant running on my home server.

Given my main software is open source it should be possible to integrate an arbitrary solution with it, with varying amount of reverse engineering and implementation necessary. Because reverse-engineering services like that is not my idea of fun, it’s much preferable to find something that’s already supported and take advantage of others’ work. While I don’t mind debugging, I don’t want to build an integration from scratch if I don’t need to.

Hardware selection

As observed last time, the “hub” model for connecting “internet of things” devices to a network seems to be the best choice from a security standpoint- the only externally-visible network device is the hub, which can apply arbitrary security policies to communications between devices and to public networks (in the simplest case, forbidding all communications with public networks). Indeed, recent scholarly work (PDF) suggests systems that work on this model but apply more sophisticated policies to communications passing through the hub.

With that in mind, I decided a Zigbee network for my sensors would be appropriate- the sensors themselves have no ability to talk to public networks because they don’t even run an Internet Protocol stack, and it’s already a fairly common standard for communication. Plus, I was able to get several of the previously-mentioned Xiaomi temperature, humidity and barometric pressure sensors for about $10 each; a quite reasonable cost, given they’re battery powered with very long life and good wireless range.

One of the Xiaomi temperature/humidity/pressure sensors.

Home assistant already has some support for Zigbee devices; most relevant here seems to be its implementation of the Zigbee Home Automation application standard. Though the documentation isn’t very clear, it supports (or, should support) any radio that communicates with a host processor over a UART interface and speaks either the XBee or EZSP serial protocol.

Since the documentation for Home Assistant specifically notes that the Elelabs Zigbee USB adapter is compatible, I bought one of those. Its documentation includes a description of how to configure Home Assistant with it and specifically mentions Xiaomi Aqara devices (which includes the sensors I had selected), so I was confident that radio would meet my needs, though unsure of exactly what protocol was actually used to communicate with the radio over USB at the time I ordered it.

Experimenting

Once I received the Zigbee radio-on-a-usb-stick, I immediately tried to manually drive it using whatever libraries I could use to set up a network and get one of my sensors connected to it. This ended up not working, but I did learn a lot about how the radio adapter is meant to work.

For working with it in Python, the Elelabs documentation points to bellows, a library providing EZSP protocol support for the zigpy Zigbee stack. It also includes a command-line interface exposing some basic commands, perfect for the sort of experimentation I wanted to do.

Getting connected was easy; I plugged the USB stick into my Linux workstation and it appeared right away as a PL2303 USB-to-serial converter. Between this and noting that bellows implements the EZSP protocol, I inferred that the Elelabs stick is a Silicon Labs EM35x microcontroller running the EmberZNet stack in a network coordinator mode, with a PL2303 exposing a UART over USB so the host can communicate with the microcontroller (and the rest of the network) by speaking EZSP.

SiLabs marketing does a pretty good job of selling their software stack.

Having worked that out and made sense of it, I printed out a label for the stick that says what it is (“Elelabs Zigbee USB adapter”) and how to communicate with it (EZSP at 57600 baud) since the stick is completely unmarked otherwise and being able to tell what it does just by looking at it is very helpful.

Trying to use the bellows CLI, the status output seemed okay and the NCP was running. In order to connect one of my sensors, I then needed to figure out how to make the sensor join the network after using bellows permit to let new devices join the network. The sensors each came with a little instruction booklet, but it was all in Chinese. With the help of Google Translate, I was able to take photos of it and find the important bit- holding the button on the sensor for about 5 seconds until the LED blinks three times will reset it, at which point it will attempt to join an open network.

On trying to run bellows permit prior to resetting a sensor to get it on the network, I encountered an annoying bug- it didn’t seem to do anything, and Python emitted a warning: RuntimeWarning: coroutine 'permit' was never awaited. I dug into that a little more and found the libraries make heavy use of PEP 492 coroutines, and the warning was fairly clear that a function was declared async when it shouldn’t have been (or its coroutine wasn’t then given to an event loop) so the function actually implementing permit never ran. I eventually tracked down the problem, patched it locally and filed a bug which has since been fixed.

Having fixed that bug, I continued to try to get a sensor on my toy network but was ultimately (apparently) unsuccessful. I could permit joins and reset the sensor and see debug output indicating something was happening on the network, but never any conclusive messages saying a new device had joined and rather a lot of messages along the lines of “unrecognized message.” I couldn’t tell if it was working or not, so moved on to hooking up Home Assistant.

Setting up Home Assistant

Getting set up with Home Assistant was mostly just a matter of following the guide provided with the USB stick, but using my own knowledge of how to set up the software (not using hassio). Configuring the zha component and pointing it at the right USB path is pretty easy. I did discover that specifying the database_path for the zha component alone is not enough to make it work; if the file doesn’t already exist setup just fails. Simply creating an empty file at the configured path is enough- apparently that file is an sqlite database that zigpy uses to track known devices.

Still following the Elelabs document, I spent a bit of time invoking zha.permit and trying to get a sensor online to no apparent success. After a little more searching, I found discussion on the Home Assistant forums and in particular one user suggesting that these particular sensors are somewhat finicky when joining a network. They suggested (and my findings agree) that holding the button on the sensor to reset it, then tapping the button approximately every second for a little while (another 5-10 seconds) will keep it awake long enough to successfully join the network.

The keep-awake tapping approach did eventually work, though I also found that Home Assistant sometimes didn’t show a new sensor (or parts of a new sensor, like it might show the temperature but not humidity or pressure) until I restarted it. This might be a bug or a misconfiguration on my part, but it’s minor enough not to worry about.

At this point I’ve verified that my software and hardware can all work, so it’s time to set up the permanent configuration.

Permanent configuration

As mentioned above, I run Home Assistant on my Linux home server. Since I was already experimenting on a Linux system, that configuration should be trivial to transfer over, but for one additional desire I had: I want more freedom in where I place the Zigbee radio, in particular not just plugged directly into a free USB port on the server. Putting it in a reasonably central location with other radios (say, near the WiFi router) would be nice.

A simple solution might be a USB extension cable, but I didn’t have any of those handy and strewing more wires about the place feels inelegant. My Internet router (a TP-Link Archer C7 running OpenWrt) does have an available USB port though, so I suspected it would be possible to connect the Zigbee radio to the router and make it appear as a serial port on the server. This turned out to be true!

Serial over network

To find the solution for running a serial port over the network, I first searched for existing protocols; it turns out there’s a standard one that’s somewhat commonly used in fancy networking equipment, specified by RFC 2217. RFC 2217 specifies a set of extensions to Telnet allowing serial port configuration (bit rate, data bits, parity, etc) and flow control over Telnet.

A diagram of RFC 2217 application from some IBM documentation.

Having identified a protocol that does what I want, it’s then a matter of finding software that works as a client (assuming I’ll be able to find or write a suitable server). Suitable clients are somewhat tricky however, since from an applicaton perspective UART use on Linux involves making specialized ioctls to the device to configure it, then reading and writing bytes as usual. Making an RFC2217 network serial device appear like a local device would seem to involve writing a kernel driver that exports a new class of RFC2217 device nodes supporting the relevant ioctls- none exists.¹

An alternate approach (not using RFC 2217) might be USB/IP, which is supported in mainline Linux and allows a server to bind USB devices physically connected to it to a virtual USB controller that can then be remotely attached to a different physical machine over a network. This seems like a more complex and potentially fragile solution though, so I put that aside after learning of it.

Since Linux doesn’t have any kernel-level support for remote serial ports, I needed to search for support at the application level. It turns out bellows uses pyserial to communicate with radios, and pySerial is a quite featureful library- while most users will only ever provide device names like COM1 or /dev/ttyUSB0, it supports a range of more exotic URLs specifying connections, including RFC 2217.

So given a suitable server running on a remote machine, I should be able to configure Home Assistant to use a URL like rfc2217://zigbee.local:25 to reach the Zigbee radio.

Serial server

The next step in setting up the Zigbee radio plugged into the router is finding an application that can expose a PL2303 over the network with the RFC 2217 protocol. That turned out to be a short search, where I quickly discovered ser2net which does the job and is already packaged for OpenWRT. Installing it on the router was trivial, though I also needed to be sure the kernel module(s) required to expose the USB-serial port were available:

1

# opkg install kmod-usb-serial-pl2303 ser2net

Having installed ser2net, I still had to figure out how to configure it. While the documentation describes its configuration format, I know from experience that configuring servers on OpenWRT is usually done differently (as something of a concession to targeting embedded systems without much storage). I quickly found that the package had installed a sample configuration file at /etc/config/ser2net:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


config ser2net global
 option enabled 1

config controlport
 option enabled 0
 option host localhost
 option port 2000

config default
 option speed 115200
 option databits 8
 option parity 'none'
 option stopbits 1
 option rtscts false
 option local false
 option remctl true

config proxy
 option enabled 0
 option port 5000
 option protocol telnet
 option timeout 0
 option device '/dev/ttyAPP0'
 option baudrate 115200
 option databits 8
 option parity 'none'
 option stopbits 1
# option led_tx 'tx'
# option led_rx 'rx'
 option rtscts false
 option local false
 option xonxoff false

Unfortunately, this configuration doesn’t include any comments so the reader is force to guess the meaning of each option. They mostly correspond to words that appear in the ser2net manual, but I didn’t trust guesses so went digging in the OpenWRT packages source code and found the script responsible for converting /etc/config/ser2net into an actual configuration file when starting ser2net.

My initial guess at the configuration I wanted looked something like this:

1
2
3
4
5
6
7
8


config proxy
 option enabled 1
 option port 5000
 option protocol telnet
 option timeout 0
 option device '/dev/ttyUSB0'
 option baudrate 57600
 option remctl true

The protocol is specified as telnet because RFC 2217 is a layer on top of telnet (my first guess was that I actually wanted raw until actually reading the RFC and seeing it was a set of telnet extensions), and the device is the device name that I found the Zigbee stick appeared as when plugged into the router.² Unfortunately, this configuration didn’t work and pyserial gave gack a somewhat perplexing error message: serial.serialutil.SerialException: Remote does not seem to support RFC2217 or BINARY mode [we-BINARY:False(INACTIVE), we-RFC2217:False(REQUESTED)].

Without much visibility into what the serial driver was trying to do, I opted to examine the network traffic with Wireshark. I first attempted to use the text-mode interface (tshark -d tcp.port==5000,telnet -f 'port 5000'), but quickly gave up and switched to the GUI instead. I captured the traffic passing between the server and router, but there was almost nothing! The client (pyserial) was sending some Telnet negotiation messages (DO ECHO, WILL suppress go ahead and COM port control), then nothing happened for a few seconds and the connection closed.

Since restarting Home Assistant for every one of these serial tests was quite cumbersome, at this point I checked if pyserial includes any programs suitable for testing connectivity. It happily does, provided in my distribution’s package as miniterm.py. Running miniterm.py rfc2217://c7:5000 failed in the same way, so I had a quicker debugging tool.

At this point the problem seems like it’s at the server side, so I stopped the ser2net server on the router and started one in the foreground, with a custom configuration specified on the command line:

1
2
3


$ /etc/init.d/ser2net stop
$ ser2net -n -d -C '5000:telnet:0:/dev/ttyUSB0:57600 remctl'
ser2net[14914]: Unable to create network socket(s) on line 0

While ser2net didn’t outright fail, it did print a concerning error message. Does it work if I change the port it’s listening on?

1

$ ser2net -n -d -C '1234:telnet:0:/dev/ttyUSB0:57600 remctl'

And then running miniterm.py succeeds, leaving me with a terminal I could type into (but didn’t, since I don’t know how to speak EZSP with my keyboard).

1
2
3
4
5


$ miniterm.py rfc2217://c7:1234 57600
--- Miniterm on rfc2217://c7:1234 57600,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

--- exit ---

I discovered after a little digging (netstat -lnp) that miniupnpd was already listening on port 5000 of the router, so changing the port fixes the confusing problem. A different sample port in the ser2net configuration would have prevented such an issue, as would ser2net giving up when it fails to bind to a requested port instead of printing a message and pretending nothing happened. But at least I didn’t have to patch anything to make it work.

With ser2net listening on port 2525 instead, Home Assistant can connect to it (hooray!). But it immediately throws a different error: NotImplementedError: write_timeout is currently not supported. I’ve found another bug in a rarely-exercised corner of this software stack, have I?

Well, kind of. Finding that error message in the pyserial source, something is trying to set the write timeout to zero and it’s simply not implemented in pyserial for RFC2217 connections. This is ultimately because Home Assistant (as alluded to earlier with bellows and zigpy) is all coroutine-based so it uses pyserial-asyncio to adapt the blocking APIs provided by pyserial to something that works nicely with coroutines running on an event loop. When pyserial-asyncio tries to set non-blocking mode by making the timeout zero, we find it’s not supported.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


def _reconfigure_port(self):
 """Set communication parameters on opened port."""
 if self._socket is None:
 raise SerialException("Can only operate on open ports")

 # if self._timeout != 0 and self._interCharTimeout is not None:
 # XXX

 if self._write_timeout is not None:
 raise NotImplementedError('write_timeout is currently not supported')
 # XXX

While I could probably implement non-blocking support for RFC 2217 in pyserial, that seemed rather difficult and not my idea of fun. So instead I looked for a workaround- if RFC 2217 won’t work, does pyserial support a protocol that will?

The answer is of course yes: I can use socket:// for a raw socket connection to the ser2net server. This sacrifices the ability to change UART parameters (format, baud rate, etc) on the fly, but since the USB stick doesn’t support changing parameters on the fly anyway (as far as I can tell), this is no problem.

Final configuration

The ser2net configuration that I’m now using looks like this:

1
2
3
4
5
6
7
8


config proxy
 option enabled 1
 option port 2525
 option protocol raw
 option timeout 0
 option device '/dev/ttyUSB0'
 option baudrate 57600
 option remctl 0

And the relevant stanza in Home Assistant configuration: (The baud rate needs to be specified, but pyserial ignores it for socket:// connections.)

1
2
3
4


zha:
 usb_path: 'socket://c7:2525'
 database_path: /srv/homeassistant/.homeassistant/zigbee.db
 baudrate: 57600

After ensuring the zigbee.db file exists and restarting Home Assistant to reload the configuration, I was able to pair all three sensors by following the procedure defined above: call the permit service in Home Assistant, then reset the sensor by holding the button until its LED blinks three times, then tap the button every second or so for a bit.

I did observe some strange behavior on pairing the sensors that made me think they weren’t pairing correctly, like error messages in the log (ERROR (MainThread) [homeassistant.components.sensor] Setup of platform zha is taking longer than 60 seconds. Startup will proceed without waiting any longer.) and some parts of each sensor not appearing (the temperature might be shown, but not humidity or pressure). Restarting Home Assistant after pairing the sensors made everything appear as expected though, so there may be a bug somewhere in there but I can’t be bothered to debug it since there was a very easy workaround.

Complaining about async I/O

It’s rather interesting to me that the major bugs I encountered in trying to set up this system in a slightly unusual configuration were related to asynchronous I/O running in event loops- this is an issue that’s become something of my pet problem, such that I will argue to just about anybody who will listen that asynchronous I/O is usually unnecessary and more difficult to program.

That I discovered two separate bugs in the tools that make this work relating to running asynchronous I/O in event loops seems to support that conclusion. If Home Assistant simply spawned threads for components I believe it would simplify individual parts (perhaps at the cost of some slightly more complex low-level communication primitives) and make the system easier to debug. Instead, it runs all of its dependencies in a way they are not well-exercised in, presumably in search of “maximum performance” that seems entirely irrelevant when considering the program’s main function is acting as a hub for a variety of external devices.

I have (slowly) been working on distilling all these complaints into a series of essays on the topic, but for now this is a fine opportunity to wave a finger at something that I think is strictly worse because it’s evented.

Conclusion

I’m pretty happy with the sensors and software configuration I have now- the sensors are tiny and unobtrusive, while the software does a fine job of logging data and presenting live readings for my edification.

I’d like to also configure a “real” database like InfluxDB to store my sensor readings over arbitrarily long time periods (since Home Assistant doesn’t remember data forever, reasonably so), which shouldn’t be too difficult (it’s supported as a module) but is somewhat unrelated to setting up Zigbee sensors in the first place. Until then, I’m pretty happy with these results despite the fact that I think the developers have made a terrible choice with evented I/O.

Live sensor readings from Home Assistant; nice at a glance.

I did find somebody asking for input on the implementation of exactly that, but it looks like nothing ever came of it. A reply suggesting an application at the master end of a pty (pseudoterminal) suggests an interesting alternate option, but it doesn’t appear to be possible to receive parameter change requests from a pty (though flow control is exposed when running in “packet mode”). ↩︎
I was concerned at the outset that the router might be completely unable to see the Zigbee stick, since apparently the Archer C7 doesn’t include a USB 1.1 OHCI or UHCI controller, so it’s incapable of communicating at all with low-speed devices like keyboards! I’ve heard (but not verified myself) that connecting a USB 2.0 hub will allow the router to communicate with low-speed devices downstream of the hub as a workaround. ↩︎

Building a terrible 'IoT' temperature logger

Tue, 31 Jul 2018 00:00:00 +0000

I had approximately the following exchange with a co-worker a few days ago:

Them: “Hey, do you have a spare Raspberry Pi lying around?”
Me: [thinks] “..yes, actually.”
T: “Do you want to build a temperature logger with Prometheus and a DS18B20+?
M: “Uh, okay?”

It later turned out that that co-worker had been enlisted by yet another individual to provide a temperature logger for their project of brewing cider, to monitor the temperature during fermentation. Since I had all the hardware at hand (to wit, a Raspberry Pi 2 that I wasn’t using for anything and temperature sensors provided by the above co-worker), I threw something together. It also turned out that the deadline was quite short (brewing began just two days after this initial exchange), but I made it work in time.

Interfacing the thermometer

As noted above, the core of this temperature logger is a DS18B20 temperature sensor. [Per the manufacturer] (https://www.maximintegrated.com/en/products/sensors/DS18B20.html):

The DS18B20 digital thermometer provides 9-bit to 12-bit Celsius temperature measurements … communicates over a 1-Wire bus that by definition requires only one data line (and ground) for communication with a central microprocessor. … Each DS18B20 has a unique 64-bit serial code, which allows multiple DS18B20s to function on the same 1-Wire bus. Thus, it is simple to use one microprocessor to control many DS18B20s distributed over a large area.

Indeed, this is a very easy device to interface with. But even given the svelte hardware needs (power, data and ground signals), writing some code that speaks 1-Wire is not necessarily something I’m interested in. Fortunately, these sensors are very commonly used with the Raspberry Pi, as illustrated by an Adafruit tutorial published in 2013.

The Linux kernel provided for the Pi in its default Raspbian (Debian-derived) distribution supports bit-banging 1-Wire over its GPIOs by default, requiring only a device tree overlay to activate it. This is as simple as adding a line to /boot/config.txt to make the machine’s boot loader instruct the kernel to apply a change to the hardware configuration at boot time:

1

dtoverlay=w1-gpio

With that configuration, one simply needs to wire the sensor up. The w1-gpio device tree configuration by default uses GPIO 4 on the Pi as the data line, then power and grounds need to be connected and a pull-up resistor added to the data line (since 1-Wire is an open-drain bus).

The w1-therm kernel module already understands how to interface with these sensors- meaning I don’t need to write any code to talk to the temperature sensor: Linux can do it all for me! For instance, reading the temperature out in an interactive shell to test, after booting with the 1-Wire overlay enabled:

1
2
3
4
5
6
7


$ modprobe w1-gpio w1-therm
$ cd /sys/bus/w1/devices
$ ls
28-000004b926f1 w1_bus_master1
$ cat 28-000004b926f1/w1_slave
9b 01 4b 46 7f ff 05 10 6e : crc=6e YES
9b 01 4b 46 7f ff 05 10 6e t=25687

The kernel periodically scans the 1-Wire bus for slaves and creates a directory for each device it detects. In this instance, there is one slave on the bus (my temperature sensor) and it has serial number 000004b926f1. Reading its w1_slave file (provided by the w1-therm driver) returns the bytes that were read on both lines, a summary of transmission integrity derived from the message checksum on the first line, and t=x on the second line, where x is the measured temperature in milli-degrees Celsius. Thus, the measured temperature above was 25.687 degrees.

While it’s fairly easy to locate and read these files in sysfs from a program, I found a Python library that further simplifies the process: w1thermsensor provides a simple API for detecting and reading 1-wire temperature sensors, which I used when implementing the bridge for capturing temperature readings (detailed more later).

1-Wire details

I wanted to verify for myself how the 1-wire interfacing worked so here are the details of what I’ve discovered, presented because they may be interesting or helpful to some readers. Most documentation of how to perform a given task with a Raspberry Pi is limited to comments like “just add this line to some file and do the other thing!” with no discussion of the mechanics involved, which I find very unsatisfying.

The line added to /boot/config.txt tells the Rapberry Pi’s boot loader (a version of Das U-Boot) to pass the w1-gpio.dtbo device tree overlay description to the kernel. The details of what’s in that overlay can be found in the kernel source tree at arch/arm/boot/dts/overlays/w1-gpio-overlay.dts.

This in turn pulls in the w1-gpio kernel module, which is part of the upstream kernel distribution- it’s very simple, setting or reading the value of a GPIO port as requested by the Linux 1-wire subsystem.

Confusingly, if we examine the dts file describing the device tree overlay, it can take a pullup option that controls a rpi,parasitic-power parameter. The documentation says this “enable(s) the parasitic power (2-wire, power-on-data) feature”, which is confusing. 1-Wire is inherently capable of supplying parasitic power to slaves with modest power requirements, with the slaves charging capacitors off the data line when it’s idle (and being held high, since it’s an open-collector bus). So, saying an option will enable parasitic power is confusing at best and probably flat wrong.

Further muddying the waters, there also exists a w1-gpio-pullup overlay that includes a second GPIO to drive an external pullup to provide more power, which I believe allows implementation of the strong pull-up described in Figure 6 of the DS18B20 datasheet (required because the device’s power draw while reading the temperature exceeds the capacity of a typical parasitic power setup):

By also connecting the pullup GPIO to the data line (or putting a FET in there like the datasheet suggests), the w1-gpio driver will set the pullup line to logic high for a requested time, then return it to Hi-Z where it will idle. But for my needs (cobbling something together quickly), it’s much easier to not even bother with parasite power.

In conclusion for this section: I don’t know what the pullup option for the 1-Wire GPIO overlay actually does, because enabling it and removing the external pull-up resistor from my setup causes the bus to stop working. The documentation is confusingly imprecise, so I gave up on further investigation since I already had a configuration that worked.

Prometheus scraping

To capture store time-series data representing the temperature, per the co-worker’s original suggestion I opted to use Prometheus. While it’s designed for monitoring the state of computer systems, it’s plenty capable of storing temperature data as well. Given I’ve used Prometheus before, it seemed like a fine option for this application though on later consideration I think a more robust (and effortful) system could be build with different technology choices (explored later in this post).

The Raspberry Pi with temperature sensor in my application is expected to stay within range of a WiFi network with internet connectivity, but this network does not permit any incoming connections, nor does it permit connections between wireless clients. Given I wanted to make the temperature data available to anybody interested in the progress of brewing, there needs to be some bridge to the outside world- thus Prometheus should run on a different machine from the Pi.

The easy solution I chose was to bring up a minimum-size virtual machine on Google Cloud running Debian, then install Prometheus and InfluxDB from the Debian repositories:

1

$ apt-get install prometheus influxdb

Temperature exporter

Having connected the thermometer to the Pi and set up Prometheus, we now need to glue them together such that Prometheus can read the temperature. The usual way is for Prometheus to make HTTP requests to its known data sources, where the response is formatted such that Prometheus can make sense of the metrics. There is some support for having metrics sources push their values to Prometheus through a bridge (that basically just remembers the values it’s given until they’re scraped), but that seems inelegant given it would require running another program (the bridge) and goes against the how Prometheus is designed to work.

I’ve published the source for the metrics exporter I ended up writing, and will give it a quick description in the remnants of this section.

The easiest solution to providing a service over HTTP is using the http.server module, so that’s what I chose to use. When the program starts up it scans for temperature sensors and stores them. This has a downside of never returning data if a sensor is accidentally disconnected at startup, but detection is fairly slow and only doing it at startup makes it clearer if sensors are accidentally disconnected during operation, since reading them will fail at that point.

1
2
3
4
5
6
7


#!/usr/bin/env python3

import socketserver
from http.server import HTTPServer, BaseHTTPRequestHandler
from w1thermsensor import W1ThermSensor

SENSORS = W1ThermSensor.get_available_sensors()

The request handler has a method that builds the whole response at once, which is just plain text based on a simple template.

1
2
3
4
5
6
7
8
9


class Exporter(BaseHTTPRequestHandler):
 METRIC_HEADER = ('# HELP w1therm_temperature Temperature in Kelvin of the sensor.\n'
 '# TYPE w1therm_temperature gauge\n')

 def build_exposition(self, sensor_states):
 out = self.METRIC_HEADER
 for sensor, temperature in sensor_states.items():
 out += 'w1therm_temperature{{id="{}"}} {}\n'.format(sensor, temperature)
 return out

do_GET is called by BaseHTTPRequestHandler for all HTTP GET requests to the server. Since this server doesn’t really care what you want (it only exports one thing- metrics), it completely ignores the request and sends back metrics.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 def do_GET(self):
 response = self.build_exposition(self.get_sensor_states())
 response = response.encode('utf-8')

 # We're careful to send a content-length, so keepalive is allowed.
 self.protocol_version = 'HTTP/1.1'
 self.close_connection = False

 self.send_response(200)
 self.send_header('Content-Type', 'text/plain; version=0.0.4')
 self.send_header('Content-Length', len(response))
 self.end_headers()
 self.wfile.write(response)

The http.server API is somewhat cumbersome in that it doesn’t try to handle setting Content-Length on responses to allow clients to keep connections open between requests, but at least in this case it’s very easy to set the Content-Length on the response and correctly implement HTTP 1.1. The Content-Type used here is the one specified by the Prometheus documentation for exposition formats.

The rest of the program is just glue, for the most part. The console_entry_point function is the entry point for the w1therm_prometheus_exporter script specified in setup.py. The network address and port to listen on are taken from the command line, then an HTTP server is started and allowed to run forever.

As a server

As a Python program with a few non-standard dependencies, installation of this server is not particularly easy. While I could sudo pip install everything and call it sufficient, that’s liable to break unexpectedly if other parts of the system are automatically updated- in particular the Python interpreter itself (though Debian as a matter of policy doesn’t update Python to a different release except as a major update, so it shouldn’t happen without warning). What I’d really like is the ability to build a single standalone program that contains everything in a convenient single-file package, and that’s exactly what PyInstaller can do.

A little bit of wrestling with pyinstaller configuration later (included as the .spec file in the repository), I had successfully built a pretty heavy (5MB) executable containing everything the server needs to run. I placed a copy in /usr/local/bin, for easy accessibility in running it.

I then wrote a simple systemd unit for the temperature server to make it start automatically, installed as /etc/systemd/system/w1therm-prometheus-exporter.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


[Unit]
Description=Exports 1-wire temperature sensor readings to Prometheus
Documentation=https://bitbucket.org/tari/w1therm-prometheus

[Service]
ExecStart=/usr/local/bin/w1therm-prometheus-exporter localhost 9000
Restart=always

StandardOutput=journal
StandardError=journal

# Standalone binary doesn't need any access beyond its own binary image and
# a tmpfs to unpack itself in.
DynamicUser=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Enable the service, and it will start automatically when the system boots:

1

systemctl enable w1therm-prometheus-exporter.service

This unit includes rather more protection than is probably very useful, given the machine is single-purpose, but it seems like good practice to isolate the server from the rest of the system as much as possible.

DynamicUser will make it run as a system user with ID semi-randomly assigned each time it starts so it doesn’t look like anything else on the system for purposes of resource (file) ownership.
ProtectSystem makes it impossible to write to most of the filesystem, protecting against accidental or malicious changes to system files.
ProtectHome makes it impossible to read any user’s home directory, preventing information leak from other users.
PrivateTmp give the server its own private /tmp directory, so it can’t interfere with temporary files created by other things, nor can its be interfered with- preventing possible races which could be exploited.

Pi connectivity

Having built the HTTP server, I needed a way to get data from it to Prometheus. As discussed earlier, the Raspberry Pi with the sensor is on a WiFi network that doesn’t permit any incoming connections, so how can Prometheus scrape metrics if it can’t connect to the Pi?

One option is to push metrics to Prometheus, using the push gateway. However, I don’t like that option because the push gateway is intended mostly for jobs that run unpredictably, in particular where they can exit without warning. This isn’t true of my sensor server. PushProx provides a rather better solution, wherein clients connect to a proxy which forwards fetches from Prometheus to the relevant client, though I think my ultimate solution is just as effective and simpler.

What I ended up doing is using autossh to open an SSH tunnel at the Prometheus server which connects to the Raspberry Pi’s metrics server. Autossh is responsible for keeping the connection alive, managed by systemd. Code is going to be much more instructive here than a long-form description, so here’s the unit file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


[Unit]
Description=SSH reverse tunnel from %I for Prometheus
After=network-online.target
Wants=network-online.target

[Service]
User=autossh
ExecStart=/usr/bin/autossh -N -p 22 -l autossh -R 9000:localhost:9000 -i /home/autossh/id_rsa %i
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Installed as /etc/systemd/system/autossh-tunnel@.service, this unit file tells systemd that we want to start autossh when the network is online and try to ensure it always stays online. I’ve increased RestartSec from the default 100 milliseconds because I found that even with the dependency on network-online.target, ssh could fail when the system was booting up with DNS lookup failures, then systemd would give up. Increasing the restart time means it takes much longer for systemd to give up, and in the meantime the network actually comes up.

The autossh process itself runs as a system user I created just to run the tunnels (useradd --system -m autossh), and opens a reverse tunnel from port 9000 on the remote host to the same port on the Pi. Authentication is with an SSH key I created on the Pi and added to the Prometheus machine in Google Cloud, so it can log in to the server without any human intervention. Teaching systemd that this should run automatically is a simple enable command away¹:

1

systemctl enable autossh-tunnel@pitemp.example.com

Then it’s just a matter of configuring Prometheus to scrape the sensor exporter. The entire Prometheus config looks like this:

1
2
3
4
5
6
7
8


global:
 scrape_interval: 15s # By default, scrape targets every 15 seconds.
 evaluation_interval: 15s

scrape_configs:
 - job_name: 'w1therm'
 static_configs:
 - targets: ['localhost:9000']

That’s pretty self-explanatory; Prometheus will fetch metrics from port 9000 on the same machine (which is actually an SSH tunnel to the Raspberry Pi), and do so every 15 seconds. When the Pi gets the request for metrics, it reads the temperature sensors and returns their values.

Data retention

I included InfluxDB in the setup to get arbitrary retention of temperature data- Prometheus is designed primarily for real-time monitoring of computer systems, to alert human operators when things appear to be going wrong. Consequently, in the default configuration Prometheus only retains captured data for a few weeks, and doesn’t provide a convenient way to export data for archival or analysis. While the default retention is probably sufficient for this project’s needs, I wanted better control over how long that data was kept and the ability to save it as long as I liked. So while Prometheus doesn’t offer that control itself, it does support reading and writing data to and from various other databases, including InfluxDB (which I chose only because a package for it is available in Debian without any additional work).

Unfortunately, the version of Prometheus available in Debian right now is fairly old- 1.5.2, where the latest release is 2.2. More problematic, while Prometheus now supports a generic remote read/write API, this was added in version 2.0 and is not yet available in the Debian package. Combined with the lack of documentation (as far as I could find) for the old remote write feature, I was a little bit stuck.

Things ended up working out nicely though- I happened to see flags relating to InluxDB in the Prometheus web UI, which mostly have no default values:

storage.remote.influxdb-url
storage.remote.influxdb.database = prometheus
storage.remote.influxdb.retention-policy
storage.remote.influxdb.username

These can be specified to Prometheus by editing /etc/defaults/prometheus, which is part of the Debian package for providing the command line arguments to the server without requiring users to directly edit the file that tells the system how to run Prometheus. I ended up with these options there:

1
2
3


ARGS="--storage.local.retention=720h \
 --storage.remote.influxdb-url=http://localhost:8086/ \
 --storage.remote.influxdb.retention-policy=autogen"

The first option just makes Prometheus keep its data longer than the default, whereas the others tell it how to write data to InfluxDB. I determined where InfluxDB listens for connections by looking at its configuration file /etc/influxdb/influxdb.conf and making a few guesses: a comment in the http section there noted that “these (HTTP endpoints) are the primary mechanism for getting data into and out of InfluxDB” and included the settings bind-address=":8086" and auth-enabled=false, so I guessed (correctly) that telling Prometheus to find InfluxDB at http://localhost:8086/ should be sufficient.

Or, it was almost enough: setting the influxdb-url and restarting Prometheus, it was logging warnings periodically about getting errors back from InfluxDB. Given the influxdb.database settings defaults to prometheus, I (correctly) assumed I needed to create a database. A little browsing of the Influx documentation and a few guesses later, I had done that:

1
2
3
4
5
6


$ apt-get install influxdb-client
$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 1.0.2
InfluxDB shell version: 1.0.2
> CREATE DATABASE prometheus;

Examining the Prometheus logs again, now it was failing and complaining that the specified retention policy didn’t exist. Noting that the Influx documentation for the CREATE DATABASE command mentioned that the autogen retention policy will be used if no other is specified, setting the retention-policy flag to autogen and restarting Prometheus made data start appearing, which I verified by waiting a little while and making a query (guessing a little bit about how I would query a particular metric):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


> USE prometheus;
> SELECT * FROM w1therm_temperature LIMIT 10;
name: w1therm_temperature
-------------------------
time id instance job value
1532423583303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423598303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423613303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423628303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423643303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423658303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423673303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423688303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423703303000000 000004b926f1 localhost:9000 w1therm 297.9
1532423718303000000 000004b926f1 localhost:9000 w1therm 297.9

Results

A sample graph of the temperature over two days:

The fermentation temperature is quite stable, with daily variation of less than one degree in either direction from the baseline.

I later improved the temperature server to handle SIGHUP as a trigger to scan for sensors again, which is a slight improvement over restarting it, but not very important because the server is already so simple (and fast to restart).

On reflection, using Prometheus and scraping temperatures is a very strange way to go about solving the problem of logging the temperature (though it has the advantage of using only tools I was already familiar with so it was easy to do quickly). Pushing temperature measurements from the Pi via MQTT would be a much more sensible solution, since that’s a protocol designed specifically for small sensors to report their states. Indeed, there is no shortage of published projects that do exactly that more efficiently than my Raspberry Pi, most of them using ESP8266 microcontrollers which are much lower-power and can still connect to Wi-Fi networks.

Rambling about IoT security

Getting sensor readings through an MQTT broker and storing them to be able to graph them is not quite as trivial as scraping them with Prometheus, but I suspect there does exist a software package that does most of the work already. If not, I expect a quick and dirty one could be implemented with relative ease.

On the other hand, running a device like that which is internet-connected but is unlikely to ever receive anything remotely looking like a security update seems ill-advised if it’s meant to run for anything but a short amount of time. In that case having the sensor be part of a Zigbee network instead, which does not permit direct internet connectivity and thus avoids the fraught terrain of needing to protect both the device itself from attack and the data transmitted by the device from unauthorized use (eavesdropping) by taking ownership of that problem away from the sensor.

It remains possible to forward messages out to an MQTT broker on the greater internet using some kind of bridge (indeed, this is the system used by many consumer “smart device” platforms, like Philips' Hue though I don’t think they use MQTT), where individual devices connect only to the Zigbee network, and a more capable bridge is responsible for internet connectivity. The problem of keeping the bridge secure remains, but is appreciably simpler than needing to maintain the security of each individual device in what may be a heterogeneous network.

It’s even possible to get inexpensive off-the-shelf temperature and humidity sensors that connect to Zigbee networks like some sold by Xiaomi, offering much better finish than a prototype-quality one I might be able to build myself, very good battery life, and still capable of operating in a heterogenous Zigbee network with arbitrary other devices (though you wouldn’t know it from the manufacturer’s documentation, since they want consumers to commit to their “platform” exclusively)!

So while my solution is okay in that it works fine with hardware I already had on hand, a much more robust solution is readily available with off-the-shelf hardware and only a little bit of software to glue it together. If I needed to do this again and wanted a solution that doesn’t require my expertise to maintain it, I’d reach for those instead.

Hostname changed to an obviously fake one for anonymization purposes. ↩︎

Considering my backup systems

Tue, 07 Nov 2017 00:00:00 +0000

With the recent news that Crashplan were doing away with their “Home” offering, I had reason to reconsider my choice of online backup backup provider. Since I haven’t written anything here lately and the results of my exploration (plus description of everything else I do to ensure data longevity) might be of interest to others looking to set up backup systems for their own data, a version of my notes from that process follows.

The status quo

I run a Linux-based home server for all of my long-term storage, currently 15 terabytes of raw storage with btrfs RAID on top. The choice of btrfs and RAID allows me some degree of robustness against local disk failures and accidental damage to data.

If a disk fails I can replace it without losing data, and using btrfs’ RAID support it’s possible to use heterogenous disks, meaning when I need more capacity it’s possible to remove one disk (putting the volume into a degraded state) and add a new (larger) one and rebalance onto the new disk.

btrfs’ ability to take copy-on-write snapshots of subvolumes at any time makes it reasonable to take regular snapshots of everything, providing a first line of defense against accidental damage to data. I use Snapper to automatically create rolling snapshots of each of the major subvolumes:

Synchronized files (mounted to other machines over the network) have 8 hourly, 7 daily, 4 weekly and 3 monthly snapshots available at any time.
Staging items (for sorting into other locations) have a snapshot for each of the last two hours only, because those items change frequently and are of low value until considered further.
Everything else keeps one snapshot from the last hour and each of the last 3 days.

This configuration strikes a balance according to my needs for accident recovery and storage demands plus performance. The frequently-changed items (synchronized with other machines and containing active projects) have a lot of snapshots because most individual files are small but may change frequently, so a large number of snapshots will tend to have modest storage needs. In addition, the chances of accidental data destruction are highest there. The other subvolumes are either more static or lower-value, so I feel little need to keep many snapshots of them.

I use Crashplan to back up the entire system to their “cloud”¹ service for $5 per month. The rate at which I add data to the system is usually lower than the rate at which it can be uploaded back to Crashplan as a backup, so in most cases new data is backed up remotely within hours of being created.

Finally, I have a large USB-connected external hard drive as a local offline backup. Also formatted with btrfs like the server (but with the entire disk encrypted), I can use btrfs send to send incremental backups to this external disk, even without the ability to send information from the external disk back. In practice, this means I can store the external disk somewhere else completely (possibly without an Internet connection) and occasionally shuttle diffs to it to update to a more recent version. I always unplug this disk from power and its host computer when not being updated, so it should only be vulnerable to physical damage and not accidental modification of its contents.

Synchronization and remotes

For synchronizing current projects between my home server (which I treat as the canonical repository for everything), the tools vary according to the constraints of the remote system. I mount volumes over NFS or SMB from systems that rarely or never leave my network. For portable devices (laptop computers), Syncthing (running on the server and portable device) makes bidirectional synchronization easy without requiring that both machines always be on the same network.

I keep very little data on portable devices that is not synchronized back to the server, but because it is (or, was) easy to set up, I used Crashplan’s peer-to-peer backup feature to back up my portable computers to the server. Because the Crashplan application is rather heavyweight (it’s implemented in Java!) and it refuses to include peer-to-peer backups in uploads to their storage service (reasonably so; I can’t really complain about that policy), my remote servers back up to my home server with Borg.

I also have several Android devices that aren’t always on my home network- these aren’t covered very well by backups, unfortunately. I use FolderSync to automatically upload things like photos to my server which covers the extent of most data I create on those devices, but it seems difficult to make a backup of an Android device that includes things like preferences and per-app data without rooting the device (which I don’t wish to do for various reasons).

Summarizing the status quo

btrfs snapshots offer quick access to recent versions of files.
btrfs RAID provides resilience against single-disk failures and easy growth of total storage in my server.
Remote systems synchronize or back up most of their state to the server.
Everything on the server is continuously backed up to Crashplan’s remote servers.
A local offline backup can be easily moved and is rarely even connected to a computer so it should be robust against even catastrophic failures.

Evaluating alternatives

Now that we know how things were, we can consider alternative approaches to solve the problem of Crashplan’s $5-per-month service no longer being available. The primary factors for me are cost and storage capacity. Because most of my data changes rarely but none of it is strictly immutable, I want a system that makes it possible to do incremental backups. This will of course also depend on software support, but it means that I will tend to prefer services with straightforward pricing because it is difficult to estimate how many operations (read or write) are necessary to complete an incremental backup.

Some services like Dropbox or Google Drive as commonly-known examples might be appropriate for some users, but I won’t consider them. As consumer-oriented services positioned for the use case of “make these files available whenever I have Internet access,” they’re optimized for applications very different from the needs of my backups and tend to be significantly more expensive at the volumes I need.

So, the contenders:

Crashplan for Small Business: just like Crashplan Home (which was going away), but costs $10/mo for unlimited storage and doesn’t support peer-to-peer backup. Can migrate existing Crashplan Home backup archives to Small Business as long as they are smaller than 5 terabytes.
Backblaze: $50 per year for unlimited storage, but their client only runs on Mac and Windows.
Google Cloud Storage: four flavors available, where the interesting ones for backups are Nearline and Coldline. Low cost per gigabyte stored, but costs are incurred for each operation and transfer of data out.
Backblaze B2: very low cost per gigabyte, but incurs costs for download.
Online.net C14: very low cost per gigabyte, no cost for operations or data transfer in the “intensive” flavor.
AWS Glacier: lowest cost for storage, but very high latency and cost for data retrieval.

The pricing is difficult to consume in this form, so I’ll make some estimates with an 8 terabyte backup archive. This somewhat exceeds my current needs, so should be a useful if not strictly accurate guide. The following table summarizes expected monthly costs for storage, addition of new data and the hypothetical cost of recovering everything from a backup stored with that service.

Service	Storage cost	Recovery cost	Notes
Crashplan	$10	0	"Unlimited" storage, flat fee.
Backblaze	$4.17	0	"Unlimited" storage, flat fee.
GCS Nearline	$80	~$80	Negligible but nonzero cost per operation. Download $0.08 to $0.23 per gigabyte depending on total monthly volume and destination.
GCS Coldline	$56	~$80	Higher but still negligible cost per operation. All items must be stored for at least 90 days (kind of).
B2	$40	$80	Flat fee for storage and transfer per-gigabyte.
C14	€40	0	"Intensive" flavor. Other flavors incur per-operation costs.
Glacier	$32	$740	Per-gigabyte retrieval fees plus Internet egress. Reads may take up to 12 hours for data to become available. Negligible cost per operation. Minimum storage 90 days (like Coldline).

Note that for Google Cloud and AWS I’ve used the pricing quoted for the cheapest regions; Iowa on GCP and US East on AWS.

Analysis

Backblaze is easily the most attractive option, but the availability restriction for their client (which is required to use the service) to Windows and Mac makes it difficult to use. It may be possible to run a Windows virtual machine on my Linux server to make it work, but that sounds like a lot of work for something that may not be reliable. Backblaze is out.

AWS Glacier is inexpensive for storage, but extremely expensive and slow when retrieving data. The pricing structure is complex enough that I’m not comfortable depending on this rough estimate for the costs, since actual costs for incremental backups would depend strongly on the details of how they were implemented (since the service incurs charges for reads and writes). The extremely high latency on bulk retrievals (up to 12 hours) and higher cost for lower-latency reads makes it questionable that it’s even reasonable to do incremental backups on Glacier. Not Glacier.

C14 is attractively priced, but because they are not widely known I expect backup packages will not (yet?) support it as a destination for data. Unfortunately, that means C14 won’t do.

Google Cloud is fairly reasonably-priced, but Coldline’s storage pricing is confusing in the same ways that Glacier is. Either flavor is better pricing-wise than Glacier simply because the recovery cost is so much lower, but there are still better choices than GCS.

B2’s pricing for storage is competitive and download rates are reasonable (unlike Glacier!). It’s worth considering, but Crashplan still wins in cost. Plus I’m already familiar with software for doing incremental backups on their service (their client!) and wouldn’t need to re-upload everything to a new service.

Fallout

I conclude that the removal of Crashplan’s “Home” service effectively means a doubling of the monthly cost to me, but little else. There are a few extra things to consider, however.

First, my backup archive at Crashplan was larger than 5 terabytes so could not be migrated to their “Business” version. I worked around that by removing some data from my backup set and waiting a while for those changes to translate to “data is actually gone from the server including old versions,” then migrating to the new service and adding the removed data back to the backup set. This means I probably lost a few old versions of the items I removed and re-added, but I don’t expect to ever need any of them.

Second and more concerning in general is the newfound inability to do peer-to-peer backups from portable (and otherwise) computers to my own server. For Linux machines that are always Internet-connected Borg continues to do the job, but I needed a new package that works on Windows. I’ve eventually chosen Duplicati, which can connect to my server the same way Borg does (over SSH/SFTP) and will in general work over arbitrarily-restricted internet connections in the same way that Crashplan did.

Concluding

I’m still using Crashplan, but converting to their more-expensive service was not quite trivial. It’s still much more inexpensive to back up to their service compared to others, which means they still have some significant freedom to raise the cost until I consider some other way to back up my data remotely.

As something of a corollary, it’s pretty clear that my high storage use on Crashplan is subsidized by other customers who store much less on the service; this is just something they must recognize when deciding how to price the service!

There is no cloud, it’s just someone else’s computer. ↩︎

Reflecting on Breath of the Wild

Sun, 11 Jun 2017 00:00:00 +0000

I’ve been enjoying The Legend of Zelda: Breath of the Wild recently, and reflected some on what makes it interesting to me from a non-gameplay perspective. This document is a version of those musings organized for publication, though perhaps less well organized than my usual writings- I am by no means a skilled critic, but spending longer in composing this would likely just delay its completion to little benefit.

Note that at the time of this writing I have not yet completed the game, but there are still some minor spoilers for the early portions of the game and general premise.

A bit of a callout to Metal Gear Solid 3, here.

I haven’t really played any Zelda games before Breath of the Wild. Once (long ago) I played a little bit of A Link to the Past but didn’t find it interesting (and was bad at it). Similarly, some time later I tried Ocarina of Time and failed to find anything compelling about it. Claiming that those games are simply not fun would be disingenuous, given both of them appear in multiple lists of “greatest video games ever” compiled by various parties. The correct question to answer here is then which of the differences between those earlier games and Breath of the Wild make the latter interesting to me, but the former not.

Having had extremely limited exposure to earlier Zelda games, I am in a poor position to comment on gameplay differences. In principle I appreciate the open design of Breath of the Wild that allows the player to go anywhere and do just about anything following the completion of a short tutorial section, but it seems to me that I have not been taking much advantage of that freedom. My general impression of the older Zelda style is that there is usually only one way to progress and that may be a frustrating factor that is not highly ranked in my mind however, whereas Breath of the Wild has given me as a player a sufficiently precise goal and the freedom to go after it at any time, but also hints as to things I may do to make success in that quest more probable.

It’s possible that merely offering more freedom to the player and applying contemporary popular game design principles (and resources) to the general Zelda formula is enough to capture the interest of the Present Me. Certainly when I tried the other above-described games they would have been relatively old so I may have been accustomed to games simply built with different style.

On the other hand, I have opinions about how Zelda has previously approached storytelling and how it does in Breath of the Wild which make for rather more interesting thoughts, so I will turn to those considerations instead.

The Big Bad

My general impression of the typical Zelda plot is “oh no stop the evil man from doing the evil thing.” As far as it goes, that’s fine but I find it uninteresting. Breath of the Wild takes a somewhat different approach to the general concept that I find much more compelling.

Elemental evil infests the seat of power.

Ganon, the usual Big Bad, is depicted as an elemental evil in Breath of the Wild, rather than some kind of dark wizard. If a mere dark wizard could be the source of an existential threat of the sort that requires a Great Hero (the player) to protect the world, it seems that there would be more frequent existential threats. Given the presentation suggesting that the player is a unique and noteworthy hero (for instance, collecting artifacts that are implied to be unique and offer powerful abilities to a chosen wielder), putting them in a world where the threat demanding heroism is as mundane as a dark wizard of some kind is rather unconvincing.

The ability of the player character to defeat this elemental evil is not (at least from the start) taken as self-evident, in comparison to how I see other Zelda games, which from the beginning seem to take it as given that the player character is destined to become a great hero and save the world.

A wild landscape with aspects of decaying grandeur.

Breath of the Wild starts with the player as the only sentient being in a wild landscape with traces of former glory, quickly introducing a guide sort of character who acts as a gently-guided tutorial before giving fuzzy backstory about how the player character is essentialy one in a long line of heroes. The details of this are revealed over hours of gameplay, largely leaving the player alone in a vast world scattered with suggestions of a long history.

I previously found the official canon for Zelda to be rather silly, because every new game is basically the same structurally (Link the hero must do hero things to assist/rescue Zelda the princess-heroine from the Big Bad). This has been justified as that they are these recurring characters over very long periods of time, but it struck me mostly as a convenient excuse for recycling characters that are well-liked.

A world with history

In Breath of the Wild the history of the world is used to good effect, including the recurring nature of the major characters. Lore-keeping NPCs allude to the distant past and what are effectively past incarnations of the main characters, while side characters are more defined by their roles so it is plausible to reuse them.

The elemental evil is eternally opposed by the guardian souls.

The world itself acts as a testament to the long history of the land, with ruined structures dotting the landscape (often large stone constructions that are impressive even as ruins) and even older structures (shrines) that serve as a kind of waypoints in what might otherwise be aimless wandering for the player. While the shrines are largely ignored by NPCs, they are not invisible- NPCs seem aware of their presence, but because they are not useful go largely ignored. If there is a weak point in this bit of world-building, the lack of interest or elaboration around shrines seems to be it- but this seems fair when these shrines are meant to be orders of magnitude older than any other relevant features of the landscape.

The art style might earn some credit for informing the feel of the world, but I don’t feel qualified to comment on that. I’ll leave comments on the art of the game to commentators better informed about art, leaving my thoughts at how it seems to be a good use of the limited computing power available to the game’s developers, recognizing that the Wii U and Switch are underpowered systems by the standards of other contemporary gaming platforms.

Characterization

I noted earlier that Breath of the Wild does not seem to take it as given that the player’s character is destined to become a great hero and save the world, but an alternate interpretation might be that the player already has hero credentials. It is shown early on that Link was some kind of royally appointed champion, and it is then implied that those selfsame credentials derive to some extent from Link being an embodiment of a spirit that protects the world through time immemorial.

Only the Chosen Hero can wield the Sword that Seals the Darkness!

Despite these heroic credentials, the player at the beginning of the game is effectively powerless, having been idle (unconscious) for a century, following a disastrous defeat at the hands of an encroaching Ganon.

Much of the characterization is done through flashbacks, which I find pretty effective. Putting this kind of backstory in-line might have been distracting and taken away from the feel of a desolate and ruined world, which makes for some of the most intriguing parts of the setting.

It is a bit unusual among games (those with this kind of budget available at least) that this one largely does without voice acting- most characters (the player included) “speak” in what amounts to grunts, except in flashbacks and with a few major characters who are fully voiced. This works well I find; they are not particularly wordy, and suggest some kind of connection between Link and the world’s past, making the player’s task something more resembling a quest to regain the glory of the land, rather than just preventing a cataclysm (which has already happened!).

I appreciate the gravitas of the voiced instances in general as appropriate to the events surrounding a kingdom under mortal threat. Despite that, there are welcome comedic and otherwise lighthearted instances that seem appropriate when considering that Link and Zelda are supposed to be relatively young. In particular, Zelda as a character does not come across to me as a nearly-omniscient plot device (“go here and do this, you must!”), but instead as a typical person thrust into a position of power as royalty doing what they feel is right in a difficult situation.

Zelda accepts a difficult role when the best efforts of the people have failed.

Other characters help the game illustrate that in the past Link is coming from, there were organized attempts to resist the impending doom embodied in Ganon. Compare this to my impression of other Zeldas, where if anybody else is aware of the existential hazard that must be defeated by the player they foolishly choose to put their trust in the abilities of the player who begins with no credentials to speak of.

The most guidance any person gives to the player regarding how best to achieve their ultimate goal of defeating the Great Evil hanging over the land amounts to general information, stating what happened in the past and suggesting what might be helpful. Not only does this support the feeling that nobody knows a magical secret that will defeat Ganon, but it also meshes nicely with the feeling that the player is responsible for restoring the former glory of the land (and themselves), leaving it up to the player what level of achievement is appropriate.

If desired, a sufficiently skilled player can go directly from the game’s introduction to defeating the Big Bad, in some fashion asserting that there is nothing that need be reclaimed.

Concluding

With a largely ruined and wild world in the grip of an elemental evil, Breath of the Wild suggests that there is a long and glorious forgotten history to be rediscovered and reclaimed. Whereas the feeling I get from the narrative setup of other Zelda games is “become a hero and save the land!”, this one is closer to “reclaim the forgotten glory of the land by rediscovering your own heroic nature.” Less a story of self-actualization, the player is fighting the universe itself to build a better world for themselves and every other character in it- but they are granted great freedom to decide what must be done and how.

Nice framing of a highly visible goal.

Recognizing my preferences in fiction, it is hardly surprising that I find Breath of the Wild’s approach to storytelling compelling. For instance, I quite enjoy the works of Alastair Reynolds, which often have themes relating to inhumanly long timescales and the inability of humans to comprehend all relevant aspects. Taken slightly differently, I greatly appreciate Lovecraftian horror with its bleak outlook and Breath of the Wild hits some similar notes (albeit in a much more positive fashion).

So, yeah. I like Breath of the Wild and really appreciate how it diverges from other Zelda games in presentation (though I lack the perspective to objectively judge how much it actually differs!). Even if you, dear reader, have found other Zelda iterations uninteresting, this one might be worth looking at.

At this point I’m just appreciating the scenery.

An illustrated guide to LLVM

Tue, 16 May 2017 10:20:43 +1000

At the most recent Rust Sydney meetup (yesterday, “celebrating” Rust’s second birthday) I gave a talk intended to provide an introduction to using LLVM to build compilers, using Rust as the implementing language. The presentations were not recorded which might have been neat, but I’m publishing the slides and notes here for anybody who might find it interesting or useful. It is however not as illustrated as the title may seem to suggest.

It’s embedded below, or you can view standalone in your browser or as a PDF, available with or without presenter notes. Navigate with the arrow keys on your keyboard or by swiping. Press ? to show additional keys for controls; in particular, s will open a presenter view that includes the plentiful notes I’ve included.

In case of curiousity, I built the presentation with reveal.js and its preparation consumed a lot more time than I initially expected (though that’s not the fault of reveal).