Thanks to my dad’s wonderful prescience, I grew up with computers–and by grew up, I mean grew up. I think the first computer in the house was a Commodore (although for some reason I imputed a memory of my dad owning an Amstrad-Schneider). I was too young to remember much other than sitting on my dad’s lap and staring at the computer screen. Back in the day it was impossible (and I believe also illegal) to own “imperialist” equipment (I know one needed the governor’s permission to get a car) and so I am deeply impressed by my dad’s ability to do magic.

When I was six or seven (again, I don’t remember exactly), I got my very own ZX Spectrum + with a black-and-white monitor. I remember playing with it for hours (the fact that those years spent staring at a CRT screen haven’t made me blind confirms my theory that if you grow up with something you get used to it and it doesn’t harm you; ironically many of my friends who got their game consoles or computers when they were teenagers are nearly legally blind now).

Pretty quickly I started writing programs for it. Looking back, this was crazy–the user manual was in English and so at age seven I knew about eighty words–keywords used in programs I’d write–in English very well but no grammar or vocabulary that kids my age for whom English was their native language knew (a fascinating way to learn the language).

I don’t want to digress too much from the theme of this post; one day I’ll continue the rant. The important thing was that at the time the computer had 48k of memory available (an equivalent to the amount of information contained in the text of the Constitution of the United States of America; to put things in perspective, computers come with 2GB now, which is about 40 thousand times more). I didn’t feel I needed much, though, because the capabilities of the computer were limited and my brain was pretty small, too. Over time, however, I did start bumping into these limits more and more. I could define my own characters and sprites (for which I had to learn to operate in binary; knowing binary before I knew how to divide is a funny thing, now that I think about it) but I could define at most 256 of them.

Fast forward six years, to my first PC (getting close to that ominous greatest moment). Eternally curious how to make games rather than play them I’d continue programming. My brain, now more fully developed, could process more information and so I expected my programs to. The PC was also capable of much more (I think I had 16MB of memory at this point?) yet due to the hardware limitations I could use at most 640kB of it. You may think that a little more than 650 thousand bytes should suffice but that’s how much information is contained in a single uncompressed photo! I felt very constrained–my computer could play sound, display graphics, and perform calculations much faster than anything I’ve seen before. Yet I couldn’t take advantage of any of it. There seemed to be some kind of a rule that stipulated those omnipresent limits–party poopers. There seemed no way around it.

Then, one day, I discovered this utility called DOS/4GW. It wasn’t a discovery as much as a result of an investigation: I’d see more and more advanced games pop up that surely couldn’t have been subject to the 640k limit (they combined music with graphics and seemed to store a lot of information about the virtual worlds they were depicting… my intuition told me that must have been more than the measly 640k). All these games would launch a small application before they themselves started, and that application would simply pop “DOS/4GW” on the screen and disappear. So I started digging.

I found out (at that point I didn’t have any access to the Internet–searching for information was so incredibly painful back then) that this little utility allowed the game programmers to bypass the 640k limit, effectively taking advantage of all the memory that the computer had available. And to my shock it turned out that I could also take advantage of this application when making my programs.

This lifting of the limit was, in retrospect, the single most eventful day in my entire life of interaction with computers. For one, the limit was immediately increased from 640k to 16M (a 25-fold increase!). But, more importantly, it was a soft limit: I could simply buy more memory and have more available to my programs. I felt empowered^*. I went crazy–there were finally no limits to my creativity.

I like to do difficult things so I decided to try (explaining this to my mom):

1. This is a bug — which is just a phrase for an error that was caused by a mistake in the programming of the device

2. It happens whenever the newsfeed needs to display an ampersand — instead, that ampersand gets replaced with the word amp followed immediately by a semicolon

3. The device needs to somehow know what it is supposed to display. I don’t know for sure the way it is done but programmers like to reuse parts of code, and standards, and conventions that have been widely accepted, so I have an idea for how this works

4. The device is connected to some kind of a component whose task it is to receive information from the outside (after all, the newsfeed is regularly updated with fresh news) and pass it on to the device to it can be ultimately displayed on screen

5. It doesn’t matter what that component is — it may be a radio receiver which receives all the information periodically from a radio transmitter somewhere in the City (just like the RDS on the radio which allows you to see what band is currently playing), or a 3G receiver which receives the information from a cell tower (just like your cell phone is able to receive email), or maybe the cab driver is plugging the device in to some kind of box when he parks the cab at the end of his shift, and the data is transferred then

6. In any case, the data is transmitted to the device. There are many ways to transmit the data — remember that the transmission is digital which means that there has to be some code for each character of the newsfeed. This is similar to Morse code. A convention that is used a lot is to use a code called ASCII which assigns every character a combination of eight bits — each bit is either a zero or a one. I’m guessing this data is transmitted this way. I’m guessing this because I haven’t seen anything more complicated in the newsfeed than regular text — if I saw little icons, or Greek characters, I would have to go for a more complicated code (ASCII lets you encode at most 256 different characters)

7. But the newsfeed is not the only thing that gets transmitted. Weather information gets transmitted (and displayed as a number — the temperature — and an icon — rain, snow, etc.); those annoying commercials must also be transmitted the same way. The latter is a lot more information than the newsfeed, but the fundamental way of transmission is the same

8. Because there is more than one different piece of information to transmit, there has to be some way to organize this information. ASCII has no built-in way to do this because all it does it encode individual characters. For example, if the device received just a “stream of consciousness” information like this:

Crate & Barrel reports quarterly loss. President Obama arrived in Denmark. Monday 67 degrees sunny. Tuesday 70 degrees heavy rain.

it would be difficult for the device to decipher what belongs to the newsfeed, and what belongs to the weather forecast. Within each group, there are structures as well (the newsfeed has individual ticker items; each day of forecast contains the day of the week, the temperature, and the weather conditions). So programmers use another convention called XML which allows them to organize information in a way that computer programs can read. XML allows you to surround text with special words enclosed in brackets which are interpreted specially. There are a few rules that stuck (for example, to use angular brackets, and to use a slash for the word at the end). So, for example, the above transmission would look like this:

<news>
  <item>Crate & Barrel reports quarterly loss.</item>
  <item>President Obama arrived in Denmark.</item>
</news>
<weather>
  <item>
    <day>Monday</day>
    <temperature>67</temperature>
    <condition>sunny</condition>
  </item>
  <item>
    <day>Tuesday</day>
    <temperature>70</temperature>
    <condition>heavy rain</condition>
  </item>
</weather>

You will see that all information is there, it’s just highly structured. A computer program can then ask for things like, “give me every item in the weather block, and for every item, piece together what’s in the day block, what’s in the temperature block (adding the word “degrees” to the end), and what’s in the condition block.”

9. XML has some limitations. For example, an opening angle bracket cannot be used anywhere in the text because the program will think that it’s the beginning of a special block and probably not allow this transmission:

<item>To write that 2+2<5 will confuse the hell out of this program</item>

By “not allow” I mean, it’s possible that the part of the program doing the transmitting is expecting correct XML (that is, every opening angle bracket has a corresponding closing angle bracket, and so on). Perhaps the part of the program doing the receiving is expecting correct XML. Very likely both do (because programmers reuse code — somebody else wrote code for interpreting XML and they probably wrote it in a way that prevents common mistakes from happening).

10. To get around this problem, if you want to display an opening angle bracket, you have to use a special code instead. This code (again, by convention) is &lt; (which stands for “less than”): the ampersand denotes the beginning of a special code, and the semicolon ends it (you need both; otherwise the word “altitude” would be rendered as “a<itude”). Similarly, &gt; is the closing angle bracket (“greater than”). So the above really needs to be written as

<item>To write that 2+2&lt;5 will confuse the hell out of this program</item>

11. We’re almost there. This is a cool way to solve one problem, but unfortunately it introduces another one: you can’t display an ampersand! (because the program will think that you mean a special character). The way programmers solved this problem is to create a special code for ampersand itself — &amp;. So the news item that gets transferred to the device probably looks like this:

<news>
  <item>Crate &amp; Barrel reports quarterly loss.</item>
</news>

12. I’m pretty confident that so far I’ve been fairly right — &amp; is seen pretty much whenever XML is involved. One can come up with many theories at this point. Here is one.

13. I already mentioned that usually, programmers would include a frequently-used piece of code that does the proper decoding (i.e. turns &amp; into an ampersand). It’s possible that in this case, they didn’t use that common code, and instead wrote their own (thinking it’s easy to write something simple like this). That code may simply be ignoring all special codes and just passing whatever it encountered on. Then, down the road, another piece of code would pick up whatever it received (at this point it’s no longer aware that the text came from XML), and, as a safety measure, simply stripped any characters that it didn’t expect. This includes ampersands.

14. Why is this a safety measure? Programs are usually written in a defensive way, that is, they make very few assumptions about what they are given. Instead, they err on the safe side and double, triple check everything. One of the checks commonly performed is called sanitization, and it’s a process of turning possibly erroneous input into a correct one by stripping bad data or data that could be malicious (if interpreted literally). For example, suppose that the same newsfeed has a command that defines how fast the newsfeed is moving on screen (since the XML data is structured, we can just add a special block for this):

<news>
  <item>Crate & Barrel reports quarterly loss.</item>
  <speed>16</speed>
</news>

So if the program ever encounters a speed block, it knows to set the speed of the ticker to the given number. Now suppose that I can submit my own news items and they will be displayed. Suppose that this gets integrated with all the other news items by replacing the word CUSTOM below with the user-provided item:

<news>
  <item>This is the regular news item.</item>
  <item>CUSTOM</item>
</news>

This is all fine for simple news items (for example, if I provide “Hello world”, the word “CUSTOM” will be replaced with the phrase “Hello world” and everything is good). But if I knew about the speed command, I could submit an item that looks like this (read carefully!):

My news item.&lt;/item&gt;&lt;speed&gt;99999&lt;/speed&gt;&lt;item&gt;

All the &lt; and &gt; will be interpreted as opening and closing brackets and so here is what the transmission will look like:

<news>
  <item>This is the regular news item.</item>
  <item>My news item.</item><speed>99999</speed><item></item>
</news>

I just added a speed command to the feed even though I wasn’t allowed to! If this is not caught, I could crash the program by, for example, passing in a really large value for speed (or a negative value, or zero, or some nonsense). If the program didn’t strip certain characters (such as angle brackets), it could be vulnerable to attacks like this (this, by the way, is called an injection attack because I’m injecting special code into the data that I’m allowed to provide). Ampersands can also be interpreted as special characters (because &lt; is translated into an angular bracket which I can use to form a news item that changes the speed!) so they are stripped.

15. Hence, &amp; becomes amp;. Now the newsfeed contains no special characters and hence we see amp; on the screen.

Hopefully now my mom can see where all those years of computer science education went… and at the same time she learned about injection attacks. Pretty good for one post.