the djb way

In some recent background reading, I came across the djb way, a non-djb authored site that begins to describe the work of djb, specifically focusing on the basic building djblocks you might need to craft a "perfect server".

It's hard to say what the big wigs use to develop datacenter-scale applications, beyond Perl and Python, Linux and Apache, but there are some clues.

According to a recent ACM Queue interview by Jim Gray, Amazon's architecture is, so to speak, super service-centric, reportedly with Perl/Mason on the front-end.

Taking a look at Google-related tech papers, we could guess that Python and MapReduce-style domain specific languages run atop custom file systems and the like.

I haven't tried to guess how Yahoo works, but apparently Amazon at least used to rely on Oracle for OLTP, and its data warehouse is said to have been migrated to a cluster of Linux machines running Oracle as well.

In Jim Gray's conversation with Amazon's CTO at the time, the CTO asks, "How do you test in an environment like Amazon?"

How do you do that? I'm not sure Amazon knows, but I do have some ideas on what could be done.

Let's start with a super simple "server" app.

#/usr/bin/env ruby

puts "hello webby world"

It'd require another line or two to do anything useful on the web, but let's take a look at how you might test this app. Put this in front of a never-been-paged-at-night undergrad, and they might say, it obviously works.

Put this so-called application in front of someone who has been woken up by FastCGI-induced alarms and spent the evening watching "A Scanner Darkly" and they might say something else.

What if your cat breaks into your machine and suspends the process? If the box runs out of memory when somebody on the machine tries to download every single YouTube video at the same time? If an operator needs to take down the machine for repairs? If you need to uppercase a letter and deploy this to sixty servers in as many seconds?

Suffice it to say, things start to suck.

daemontools, at least, should make life a little easier.

Alas, we need a bit more than the ability to bounce a service reliably to do something useful, but it's a start.

On a larger scale, even if we could prove every possible algorithm correct, and every new piece of code was tested perfectly, systems would still continue to fail and klaxons would go off at all odd hours.

Why?

Even if each new subsystem could be deployed individually without any problem, as Zed points out, it's not the pieces that get you, it's their unexpected combinations.

You can fuzz the heck out of your new code and your neighbors too, but when you all submit your code on Friday night, too bad for the slower poke who breaks the build and trips the not-so-silent alarm.

It may be an easy target, but the classical software lifecycle doesn't work so well when you have a thousand developers constantly deploying to fifteen thousand machines at quasi-once.

Let's formalize the problem a little bit and propose a I-Don't-Know-If-It's-Russian (*) approach.

I came up with this abstract problem earlier this month, before news of yet more violence hit our airwaves, and as I think about it now, I do question how appropriate my abstract problem really is... On the other hand, I do think that a greater understanding of sociocombinatorics and associated algorithmics would help us to better teach our children and, perhaps, think more broadly about how isolated, narrow thinking must be incomplete.

So, here it is.

"Spy Contra Spy"

Let there be n distinct countries, each with their own secret agent. As the benevolent or not-so-benevolent dictator, you may choose which k secret agents are to be trained in the deadly art of [insert deadly art].

For the special case of k=1 -- that is, arming exactly one secret agent -- you can be sure of world peace, as all countries have agreed to be very careful in their process of testing and training covert operatives.

For k larger than one, however, things fall apart. If certain pairs (or generally subsets) of countries both arm their agents, bad stuff happens.

The question is, how do you figure out which are the "bad" subsets to arm at one time? For example, arming Penguins trained by Morgan Friedman might not be a bad idea in isolation, but if you arm the Sea Otters and Polar Bears too, a three-way gang war ensues.

Staying out of politics for the moment, I want to note that staged deployment, as possibly practiced by enterprise-y software engineers, does not nicely solve the problem in a way that scales in the number of users and software engineers (let's say a billion and three thousand, respectively).

Why? It's not just that "engineering distributed systems is an art that Ph.D. students in general do not yet possess" (notes Amazon CTO), but that a single, staged pipeline is going to break pretty easily when you have dozens of teams deploying at once.

You can enforce the strictest of regression testing, but even then, don't expect your software lifecycle to be anything but slow as tar cvf x /.

How so? Well, let's say the amount of time you need to build your system isn't trivial (Windows) or thirteen different teams deployed changes in between this afternoon's test and this morning's test. As you might guess, said pipeline will stall in a horribly messy way.

Can we do better? Maybe. I'm not sure how well the theoretical model of "Spy Contra Spy" applies to a VLSD-scale developer pipeline, but it's a start. Think of each team as a country and their subsystem as their secret agent. Pushing code from one stage to another is, in our metaphor, arming their secret agent and seeing if the world blows up.

If, as in combinatorial group testing, we try to minimize the number of tests we perform, it's not hard to see that a classical software pipeline results in an expected linear number of tests. Each team arms their code, and when some unlucky owner hits an integration snafu with a previous 'commit', they roll back each patch one by one.

Linear doesn't seem that bad, but in real life, one day is way faster than five.

There's a straight forward divide-and-conquery way to find the "bad" vertex pair (edge) in our graph theoretic architectural diagram, but I'm not sure how easily that theory applies to the practice.

I might as well try to name such a testing scheme. I'll call it shatter testing, as a nod to Timothy Chan, and because it amounts to throwing permuted subsets of new code at the wall and seeing which ones stick, and which ones break.

The idea behind shatter testing is that instead of integrating all VLSD-scale deployments into a single deterministic staged pipeline, we instead consider, at every stage, a distribution of virtual stages, so to speak. Instead of test and production, you might have test-A, test-B, test-C, and test-D. All virtual stages run the same systems, but contain slightly different versions. When a subsystem is upgraded, the new code is deployed to a fraction of the test environment.

By deploying to different (but overlapping) fractions of the test environment, we can then make guesses about what pieces (bad edges) have failed to integrate, with fewer integration-test cycles. At least in theory. I don't know if any VLSD-scale webdev pipelines implement "shatter testing" as a part of their process, but perhaps they'd spend less time in beta if they did :).

It may not buy you much if you don't have hundreds of devs tinkering with the pipes, but I'm guessing there be hard problems when you do have such a social scale.

When I think about it, social complexity is what differentiates VLSD integration from twentieth century supercomputing, grids and clusters. For example, a typical supercomputing application may involve several active programmers and a small number of scientists involved. In comparison, data and code and meta and /people/ are much more heterogeneous in a VLSD setting, when millions (billions?) of people might be part of the active human-computer I/O loop.

What happens, then, when every person in the web-abled world wants to say "hello webby world" at once? I'll leave that as an exercise to me, your sometimes gentle writer.

I thought about how this loops back to social justice, and I think, with technology as the lens, it might have to do with thinking about social complexity as rigorously and empathetically as we might think about spacetime computational complexity, and not shy away when we start to think about how actions, emotions, and interactions happen on the fractal scales of all such circles in our society.