The USS Quad Damage

We can look at programs in two different ways, firstly as the space inside a program, and secondly as the space between two programs. It has always been the case that more open systems concentrate more on the space between applications, and more closed systems concentrate on the space inside an application. We can see large monolithic pieces of proprietary software such as Adobe’s Photoshop have a lot of functionality built in, but do not communicate with applications outside themselves very well. On the other hand, small applications such as those on unix systems such as ‘cat’ or ‘cut’ can chain and connect to each other. If we want a dynamic and flexible operating environment, we must think deeply about the spaces between applications.

Importantly, though, is that we aren’t talking only about applications, but services, too. In fact, all data shared between applications, be it via files on a filesystem, or via a protocol, or via an API, all share a common thread: The data is either self-describing or it is not. Note that in reality, there’s no such thing as “self-describing data”. Mostly, the data is in a strict format, but that format might contain a description of the format. Notably, formats such as XML are strongly self-describing, formats such as JSON are loosely self-describing, and formats such as an IP packet are not at all self-describing. Also of note, even a binary format such as ASN.1 can be considered self-describing.

There are also grey areas such as protobuffers (or maybe even bson?), where a standardised externalised description will construct code that forms a generator / parser combination for non self-describing formats. Broadly, though, we may split formats into “self-describing” and “not self-describing”. Part of the reason is actually design intent. The design intent of a “not self-describing” format is to minimise the error space for a packet of information. A packet may, for instance, have a CRC which allows for rejection of the entire packet wholesale. On the other hand, a “size” field of a packet will be described as “n + 1”, so that a value of “0” is still considered valid. This is not being cheap on bits! If a value within a packet is invalid, what is a parser to do? In order to prevent this conundrum, all combinations of values in a packet should be valid, save for errors which allow an entire packet to be rejected.

By the same token, a “packet” of information can be defined as a piece of information which can be safely rejected. What “safely” means here is a little broad, but note that this doesn’t just apply for streaming protocols where both applications are present. A “packet-based” file format is quite common from compression formats to video codecs. The key is the safety and predictability of the parser: The parser simply doesn’t have many error cases to consider based on errors in the packet.

On the other hand, the design goals of a self-describing format are coping with change. These formats are generally only loosely sitting on top of a packet format, for instance XML or JSON which is just a long String in a particular encoding. Unlike the non self-describing format, the self-describing format leans heavily on both the program and the parser to ensure validity. An XML or JSON parser, for instance, have many checks that can be performed to ensure the validity of the message before it is even accepted, and even here there are many error cases that create grey areas for a potential parser.

For instance, if there are invalid characters, what does the parser do? What if there is a formatting error half way through a stream? What if the text validates but not under a more strict interpretation? What if the text validates but does not match the schema? What if all of that is true but there are other logic errors in the correctly formatted interpretation of the data? All of these are usually configurable options that a parser is usually initialised with. Even after all of that, there are various error conditions and corner cases to consider.

What does this have to do with statically typed and dynamically typed programming languages? Well, I’m about to argue that self-describing formats are most attuned to dynamically typed languages, and non self-describing formats are most attuned to statically typed languages.

This mostly has to do with the internalisation and externalisation of type information. In statically typed languages, as in non self-describing data, the type information is externalised. For data, by definition type information is externalised when it is not in the data. For programs, almost by definition, the closer type information gets to runtime, the more “dynamically typed” the language is. The whole point of a statically typed language is that you know the types at compile time.

There are, of course, various grey areas. Java has a runtime and reflection, but is “statically typed”. In my view, though, the runtime makes it dynamically typed when using reflection. Indeed, when you look at the error cases when using, say, Spring, it very much seems like a “dynamically typed language”.

On the other side of the coin, dynamic languages must keep all type information at runtime. This is as much about verifying type information that isn’t available at compile time as it is about determining the types of objects at runtime. While we are aware of Python and Ruby being fully dynamic languages with a full runtime, even languages like C++ and Java have RTTI and reflection. The easy way to think about this is that C++ and Java will throw Exceptions at runtime because the type information is incorrect, just like a dynamic language. In the same way, self-describing data describes its own type information within the data. To some degree, you do not need to know the structure of the data or what it contains.

Obviously, it is possible to use internalised data sources with statically typed languages and externalised data sources with dynamically typed languages, but it’s not an easy fit. For externalised data sources in dynamic languages, there’s a lengthy decomposition into their component parts, whereas in statically typed languages, it’s usually no more difficult than defining the data structure in the first place. Similarly, internalised data structures require complex parsers into static data structures in statically typed languages, whereas a dynamically typed language may not even have to care about the structures and types. It just inspects and alters them as if they were first class objects (in the case of JSON, they actually are).

The links go deeper though. In a static language and an externalised data format, you get the same guarantees: nothing has changed, by definition, since you compiled the code. The data format is the same, and so is the code interpreting it. Nothing can go wrong save for very specific error conditions. You effectively get static typing not only over the program, but also over the data it operates. Contrast with an internalised data format in a static language. All of a sudden you have error conditions everywhere which aren’t in predictable places. You may have noticed that static languages tend to have parsers that are very strict. The reason is purely to offer some clarity over how the program can fail. Having a statically typed program that can take a loosely formatted internalised data format and not explode (such as a browser) is no mean feat.

In the dynamic landscape, however, not only can this loosely formatted data be accepted, it can be passed directly into functions which carry out the actual computation. Even those functions need not know everything about the data structure. A module might be moving objects around or re-organising them, but it really doesn’t care what’s inside. As long as the structure is broadly the same, the code will continue to work. Even if the structure has changed completely, if the atoms remain intact then functions can operate over those atoms, keeping the structure the same. Even if both of those change, a dynamic language can introspect and deal with changes fairly elegantly.

This is where the idea of dynamic languages just being unityped static languages sort of falls down. If that were true, you couldn’t add two strings and two numbers as distinct operations. Once a value has been bound as an integer, it can be added, but importantly, if the language doesn’t know what the type of some data is, it doesn’t matter. As long as the transformations on that data don’t mess with the data the program doesn’t know about, the code just keeps on working. You can grab a bunch of XML data and pass it through a bunch of functions that do an addition operation, and the functions don’t need to know what data they’re adding, because that forms part of the internalised description of the data. Is it an integer XML attribute? The integers get added. Is it a String? They get concatenated. Is there other data or other data structures nearby that the code doesn’t understand at all? Doesn’t matter; executed correctly.

And this is where I’m getting at: In a world with many small apps, dynamic apps are king. This is why most scripts that keep a computer running nicely were written in Shell, then Perl, and now Python. These applications are effectively wiring data between applications. They need to know very little about the data, even though they may need to manipulate it before passing it on. Want to write a log parser? Python is probably far easier, more flexible, and more useful. Want to take a bunch of deep and complex JSON and parse out a simple calculation? Maybe Javascript is just the ticket. Having dynamic languages as high level co-ordinators of other applications is probably a very good idea.

It feels like people treat static or dynamic languages as a religion of sorts, but the fact of the matter is, these languages are more indicative of the kinds of problems we’re solving as opposed to the way in which we’re solving them. Static languages treat data as ingots of steel, and that’s good for when you want data that’s got to take a beating. Dynamic languages treat data like putty, and that’s good for when you need it to fit in that damn hole. In the end we need both kinds of languages sitting next to each other if we’re going to be able to process data correctly and flexibly, and we should be able to understand how to apply each to have the most powerful code.

In the end, the argument over dynamic and static languages is really an argument over which kind of data structures the program should be processing: data where the structure is expressed internally to the data structure, or data where the structure is expressed externally to the data structure. Ultimately, if we want the most flexible software, we need to know when to use which kind of data structure, and how to pass these data structures between programs in a landscape where the code is always changing. I feel that having static and dynamic languages co-operate in a multi-process environment will yield better and more flexible architectures than with a language monoculture.