home
on exploration, introspection and creation

Archive for the ‘technology’ Category

Trends

Wednesday, September 1st, 2010

What’s the next big thing — many of us ask themselves this question. How do we predict how trends will shape in the future so we can be on the cutting edge — and presumably benefit from it?

A big problem with predicting a new trend is that we are fundamentally shaped by the current one. Just like revolutions are as much a result of an individual’s actions as the pressure from the Zeitgeist, our collective behaviors define a trend. Moreover, we are not really aware of being a part of a trend, which makes it very difficult to step outside of the proverbial box.

Let’s use an example. My generation defines several trends, most of which we are probably not aware of. First, there is a huge movement towards a hyperconnected-but-also-hyperisolated life. We don’t use land lines anymore. Email is the preferred method of communication — most times you can get a faster response by sending an email than by calling someone (this generational gap is most evident with the software vendors I interact with at work — a funny pattern emerges, where they call me and I email them back). More and more of us are eating well. CNN only makes us angry (recently I realized that ESPN is a much preferred station to watch at lunch despite the subject matter). This happens. Most of the time we don’t stop to think about all these things that are our everyday tasks; but it is exactly these things that draw us to social networking sites like Facebook. Twitter is another great example of a product that capitalizes on the schism between hyperconnectedness (you use Twitter to blast information about yourself to the entire world) and hyperisolation (you use Twitter to blast information about yourself to the entire world). The fact that I’m not quite sure what to call this trend shows how difficult it is to define it.

So if you want to create the next big thing, the thing that will be the next trend setter, you have to first understand the current trend. That’s not enough, though. Twitter started operation in 2006. It’s fascinating that a product which I hypothesized my generation is already too old for was created at the peak of what I believed to be my capacity for cultural absorption. This means that the things you’re thinking about as “the next big things” are probably already behind the new trend. It’s a little bit like sitting in traffic: you see cars moving faster in the lane next to you so you quickly switch lanes hoping to move faster. But to your frustration, you end up moving slower. This is because when you came up with the idea to switch lanes, statistically most other drivers had already done so so you’re just lagging behind and receiving the due punishment — joining the lane that is about to slow down to a crawl while opening the lane you were just in for the new trendsetters.

So what should we do? When you understand the Zeitgeist and finally come up with an idea for the next big thing, think about what the big thing after that would be.

Mankind’s local view of history

Tuesday, August 31st, 2010

I talked about it before and some, but the concept is so interesting to me that I need to elaborate.

I believe that it is intrinsic in human nature to possess a kind of confirmation bias, extrapolating severely limited set of data points (say, the time span of one generation) to make statements about timeless truths, concepts lasting for very long time frames. In other words, we all have a kind of local view of history which is causing us to make incorrect assumptions about the past, or is preventing us from questioning things enough to make good predictions about the future.

There are numerous examples to support this idea. Until about 2006, the general populace was convinced that housing prices will always go up. We fear of terrorists taking over planes but forget that in the seventies, plane hijackings were rather common. We love good food but just fifty years ago Americans considered food to be a rather utilitarian exercise.

It is commonly thought (and by “It is thought” I mean “at some point we all thought that, even if now we may not admit it because of a certain pressure from those who tend to expose common conceptions as myths who may make us appear stupid”) that the Middle Ages were by and large a waste — several centuries of backwardness. However, I believe that we think that only because we live in a technophiliac age where one assigns value to a very specific kind of progress that was, admittedly, absent in the Middle Ages. This is most likely coupled with a phenomenon in which we conveniently forget that progress has an exponential nature — sure, the Renaissance seemed like a huge step forward, but it very likely needed to be bootstrapped by a much slower progress that was brewing in the centuries before it. But, even more fundamentally, our belief in progress is an even more deeply rooted assumption about the human race. Can we extrapolate it into the future? Will we always aim towards progress? What if we deplete our natural resources?

And how about political systems? We think of monarchy as a less developed, more primitive form of government that suffers from a lot of problems that we have since managed to mitigate with other systems. But again, this is just confirmation bias: we were raised in a culture that relishes in the importance of democracy. We seem to think that democracy will reign forever — or at least, that monarchy will never come back. But that doesn’t have to be the case (Frank Herbert was onto something). It doesn’t have to be the case because the path between concept and execution is chaotic — small variations in the original assumptions can lead to massive differences in execution, so what seems like a massive problem with monarchy in the context of eighteenth-century world citizen’s worldview may easily be solvable with twenty-first-century’s tools and mindset. For example yes, there is potential for abuse, but what if we use technology to limit it just as technology today ensures we don’t abuse some of the privileges given to us. Once we solve these problems (which centuries of oppression have brought to the foreground), the underlying framework may provide a much more efficient way to stabilize the society (in fact, I’ve always thought that democracy is ostentatiously very inefficient — just think about the years of training, campaigning, and money spent by the candidate who loses the elections).

That’s exactly the problem with extrapolation: it assumes that some phenomenon can be approximated well with a linear model. So long as the extrapolation is minimal, most phenomena do behave pretty nicely: if today we’re burning up coal like crazy, tomorrow we are also likely to use up a lot of it. But it does not necessarily follow that in fifty years we will. Extrapolation over a long time window is unable to take into account the interactions of the thousands of variables that affect the outcomes we’re trying to predict.

History, or social sciences in general, suffer from this problem particularly much, precisely because there are so many variables at play (i.e. the systems they try to explain are so chaotic). In a way, one of my biggest realizations was that social sciences are enormously more complex than exact sciences because the latter have the liberty of operating on kiddie worlds where everything can be controlled and measured and there are relatively few degrees of freedom. It’s only when we move on to sciences that rely on inaccurate, nondeterministic and complex objects, such as economics (struggling with the complexity of human nature when applied to incentive-driven behavior), politics (struggling with the complexity of understanding the impact of policies on e.g. economics) and history (struggling with the complexity stemming from the interplay of a number of economic, political and social factors with individuals and their decision-making process), that the number of variables increases beyond our comprehension. Those models are nowhere near linear, so let’s not extrapolate.

There are some positive consequences of the abandonment of superfluous extrapolation. It is not necessarily the case that food will look less and less like food — in fact, we are slowly starting to see people go back to natural sources of food, from organic food to CSAs to growing their own vegetables. We may run out of rare earth metals in twenty years, but that doesn’t necessarily imply armageddon.

The Actual Boston Subway Map

Tuesday, August 3rd, 2010

Being a son of a seafarer, I developed a kind of fascination with being on the sea, and with maps. It is because of the latter (and because I happened to live in Boston, and because I didn’t quite like how MBTA imitated Harry Beck, and because I always wanted to know how far it actually between the different subway stops) that in 2005 I decided to make an actual Boston subway map, that is, a geographically-accurate map of all subway stops.

It was several years ago — I believe MBTA may have added a few subway stops since then, and you can also see all these stops on Google Maps, but there’s something elegant in the simplicity of my diagram. It’s also a good case study of Google Maps, scripting and LaTeX.

The idea was to find all the subway stops on a map downloaded from Google Maps using the locations of the stops as reported by MBTA (as you can imagine, it was a humongous pain to click on every single station map to figure out where to actually plot each station), and put the coordinates of each station in a LaTeX file that would generate the pdf image of the subway map. I used pstricks, which is a great LaTeX package for drawing graphics.

The following tcsh script downloads the relevant quadrants from Google Maps and creates an HTML file that displays all the quadrants on one large page. The URL format for the quadrants has changed since 2005 but you get the idea:

get.tcsh

Then I opened the large map in Photoshop and figured out the coordinates of each subway station and turned them into a LaTeX file:

mbta.txt and mbta.tex

Finally, I ran LaTeX to generate the following pdf file (click on the file to download the pdf):

The actual Boston T Map

IE team crashing the CSS standards meeting

Tuesday, July 20th, 2010

IE team crashing the CSS standards meeting

IE: Sorry we’re late guys, we didn’t know about the meeting until this morning
Everyone else: Uhmm, we have had this meeting in the same place every week for the past four years
IE: Oh really?
Everyone else: Yes. You know, we’re trying to set some standards so that web developers can write clean code that is interpreted the same way by every browser.
IE: Of course we think it’s a good idea for all browsers to interpret websites the same way IE does.
Everyone else: No, no no no! That would be a very bad idea.
IE: How come?
Everyone else: It’s stupid for one company to set the standard, especially if the standard is not really a standard but a collection of random fuzzy rules that aren’t consistent, are hard to interpret, and consist of hundreds of exceptions.
IE: We don’t buy it. You’re just jealous of our 95% market share.
Everyone else: We are a standards group, we don’t care about your market share. We’ve invited you to these meetings from day one. In fact, didn’t Microsoft push for technology standards in the first place?
IE: Of course, provided that they come from Microsoft.
Everyone else: That’s retarded.
IE: Look, we’re innovating here and you’re just slowing things down. Like when we introduced image effects.
Everyone else: Oh, you mean the Wave effect?

IE: Yeah, isn’t it awesome?
Everyone else: Looks like something from a 1990s website.
IE: Our design group sees nothing wrong with that filter.
Everyone else: Oh, you have a design group now? But, we digress. You’ve got to start complying to some of these standards, complying to both IE and other browsers is a headache for most developers.
IE: What are those standards?
Everyone else: We’ve published them and even sent you a paper copy. What the hell are these proprietary filters doing in your dynamic HTML? filter:progid:DXImageTransform.Microsoft.Alpha(opacity=20). The notation is awful and does not conform to any of the other style properties.
IE: You’re just pissed because it has Microsoft in its name.
Everyone else: And what about your lenient interpretation of identifiers? Why does Microsoft continue, over and over again, to put so much emphasis on case insensitivity? Every programming language other than… let’s see… BASIC! is case sensitive.
IE: You’ve got to allow mediocre developers to make some money too.
Everyone else: And the conditional comments in HTML? <!–[if lte IE 6]> This is only encouraging people to add IE-specific stuff.
IE: Okay, it seems we won’t be able to convince you guys to embrace innovation. Maybe at least you’ll approve our OOXML standard?
Everyone else: Are you serious? The spec that ignore industry standards like SVG, MathML, XForms and even XML?
IE: There’s no reason a reinvented wheel must be worse.
Everyone else: Oh yeah, how about how within the spec, the same element has multiple conflicting definitions (‘e‘ has eighteen of them!) and some elements are just aliases of one another? It’s almost as the people in your team didn’t even talk to each other when they wrote the spec!
IE: Screw you. We’ll win anyway. We offer the world’s most popular operating system and browser family. Can’t beat us.

Telephones

Sunday, July 11th, 2010

The telephone has some pretty impressive, clever ideas behind it. First is the concept of tone dialing which older people will be familiar with, where the number to dial is expressed as a set of digits in unary expressed through quick breaking of the phone circuit. For example, to dial 911, you could quickly press and release the “hang up” button that the headset normally resides on nine times, then (after a pause) once, and finally (after another pause) once more. This is why old phones had these funny circular dial pads — each rotation of the pad from one digit to the next would break the circuit once so if you reached for the digit “9″ and then released the pad, it would break the circuit nine times.

The next-generation — used until cell phones showed up — system was just as clever. The idea was for the exchange to listen to the sound from the headset and interpret certain frequencies as digits. The frequencies that would stand for digits were actually sets of two sounds which were not harmonics of one another — i.e. if you tried to play the sound on any instrument, or whistled, you couldn’t generate these frequencies. What a brilliant way to minimize conflicts!

For posterity, and because I’m strangely attracted to it, here is the Dual-Tone Multifrequency keypad — the row/column headers correspond to the frequencies, in Hz. The cells tell you which two frequencies are used (so the digit “7″ generates a tone that’s a combination of an 852 Hz and a 1209 Hz sounds):


·····1209·1336·1477·1633
687···1····2····3····A
770···4····5····6····B
852···7····8····9····C
941···*····0····#····D

Betting on the Timing of an Event

Sunday, June 13th, 2010

There are times when there is a disagreement over what time a particular event will happen and people want to turn the different in opinions into money. It is common to place “over-under” bets — if the event happens before time T, Andrew gets the money, otherwise Bob gets it. Usually the further the event is from T, the more money exchange hands.

I don’t like this style of betting because it’s simply not expressive enough. Instead, I prefer to bet by specifying my probability distribution of the timing of the event, and then using these distributions to determine payouts. My friend and I made this bet once and it was a fun activity, despite the math involved behind-the-scenes, not that frustrating and requiring very little mathematics to actually place the bet.

Essentially, each party draws a probability distribution of the timing of the event — a histogram with the time on the horizontal axis and the probability density function on the vertical axis. The latter can simply be intuited as “the relative probability that the event will happen around the time specified on the horizontal axis”. So if the histogram is twice as tall around 8pm than around 7pm, the event is twice as likely to happen around 8pm than around 7pm.

That’s all each person really needs to do. No need to worry about the area of the histogram summing up to 1 since the vertical axis can be scaled up appropriately. The two people should also agree on how money they are willing to bet — say k dollars each.

When the event actually occurs at time T, the two people compare the value of the probability density function (the height of the bar) at time T on their graphs (the height will be scaled appropriately so that the area adds up to 1–so of course you can’t cheat by making your graph taller) and pay up based on the difference in these values.

Executing such bets is a little difficult since it involves calculating areas under the graph which may be very irregular. Those who are mathematically masochistic can constrain themselves to piecewise linear functions, or, in the extremely, easily integrable functions; otherwise the graph can be scaned and a simple graphics editing application (like Photoshop) can be used to determine the area under the graph (using the flood fill and histogram tools).

Scope Creep

Friday, June 11th, 2010

Scope creep — the phenomenon of an ever-expanding surface area that your product needs to cover — happens when you don’t convey a vision, and don’t collect requirements correctly.

My variation on “the customer is always right” is a — in my view — more accurate “the product should always be right for your customer, but the customer never knows what this means”.

If you don’t collect requirements correctly, you will end up with a product that is not right for the customer, and this will lead to scope creep (if you don’t want to lose the customer you will have to implement the delta between what the product ended up doing and what the customer wants it to do). The customer won’t know that — a common beginner’s error is in assuming that the customer knows what they are talking about when they tell you what they want — so it’s your responsibility to make the need tangible and coherent. Poor business analysts simply listen to what the customer says and write down what they hear. Good BAs understand the requirements (which often means truly understanding the needs, and with them, the habits, behind the product). Great BAs drive the requirements, providing to the customer something he or she never knew he or she needed.

“But it’s hard to do,” you’ll probably remark. Of course it’s hard to do, and this is why good BAs are hard to come by! Don’t confine yourself to conventional methods of doing requirements analysis — i.e. talking to the customer and writing down notes. Mock something up for, or — even better — with your customer, come up prepared with theories for what the customer might actually need, or start with the anti-requirements.

Much more important — especially for a product past its infancy — is the vision you set for the product. It defines the universe of needs that the product will address, and, consequently, the universe of features the product will contain. The vision is what allows your product to be unique and remembered; the vision creates the brand. It also makes the product easier to develop for the majority of its lifecycle — you know which customers to talk to, you know what skills you’ll require your engineers and marketers to have, your product is easier to implement because all features have something in common.

The vision for all products is fluid — it has to respond to the Zeitgeist, new information, other products — and the features in your product will change based on that vision. However, vision brings about a certain inertia — it gets harder to add features if the vision changes — and this is why all products die. So just get used to it: don’t be afraid to kill products, knowing that you can bring new ones to life.

To part with, an anecdote. Imagine (in a weird future-in-the-past universe) that a group of Dutch settlers came to you and said “we want you to encode a map for our new city”. You ask, “what will the streets look like?”. “Oh, it’s a very simple system; we’ll pretty much have a grid, streets going east to west, and avenues going north to south”. “Trivial!,” you exclaim and encode every intersection as a set of two coordinates (x, y). For example, an intersection of 23rd Street and 5th Avenue would be encoded as (23, 5).

A little later you realize that there are avenues which aren’t numbered and fall in between numbered avenues. You scratch your head — your elegant system doesn’t quite capture this — but you realize that you can just map these names to non-integral coordinates. For example, (42, 3.5) would be the intersection of 42nd Street and a street that falls between the Third and Fourth avenues (which the settlers decided to call, oh I don’t know, say, Lexington Avenue). You also need non-positive coordinates because there are avenues to the West of First Avenue (which the settlers called Avenues A, B, C, …)

Then you realize that there is a special avenue that is diagonal, i.e. it intersects other avenues! Your elegant system certain does not capture all those intersections, so you implement a special set of coordinates (x, D) where x is the street that avenue intersects, and (D, y), where y is the avenue, knowing that that diagonal avenue — which the settlers called Broadway — can be represented as a piecewise curve of four straight lines, and with some clever mathematics you can still compute distances, and so on.

Then a large park is built in the center of the city, which removes some intersections but adds another irregular street. You’re visibly frustrated now, but with an additional mapping of which intersections don’t actually exist, and with an additional “special” road that connects two ends of the block taken up by the park, your system still supports the use cases. Of course, it’s pretty complicated now and it’s getting difficult to add features to it.

Finally the settlers tell you that actually, south of the 1st Street, there is a completely irregular grid of streets that are named, not numbered. That’s when you quit.

If only you had gathered requirements well in the first place…

Surprising Origins of Things (part III)

Monday, June 7th, 2010

Many people who know what they are talking about claim that humans’ abandonment of the hunter-gatherer lifestyle — the invention of agriculture — is one of the most meaningful events in the history of mankind. It is that invention that led to the rise of civilization and thus all the good things we know of — mathematics, medicine, poetry, computers, space travel, microwave ovens, Twinkies, and so on.

This is a fascinating topic and one post will not do it justice. For one, I find it interesting that what we now see as examples of mankind’s intellectual superiority and conceptual ability is simply a by-product of necessity, all stemming from agriculture. For example, agriculture increased the importance of land — suddenly the concept of ownership developed. Ownership necessitated the need to keep track of what belongs to whom, which led to accounting which led to writing which in turn led to higher, more abstract thought (and, yes, also poetry). But agriculture increased the importance of schedules which led to the idea of a calendar, which led to astronomy and then to mathematics. What if we hadn’t adopted agriculture? Would we still have mathematicians and poets? Would some other need necessitate the development of these abstract disciplines? I think so — it seems to me that fields such as mathematics lie on top of the pyramid of abstraction, and as mankind solves its problems — any problems — and becomes more efficient at whatever it does, it travels up that pyramid, ultimately arriving at those high abstractions.

What is more interesting than asking about the consequences of agriculture is questioning it in the first place — there are dissenting voices — that agriculture was, in fact, mankind’s greatest mistake. There is some compelling archaeological evidence, such as the fact that we’re just now beginning to match our pre-agricultural ancestors’ average height and disease rate, plus it’s not particularly clear to me what short-term benefit an agricultural life that demands much more of humans has — just like today you’d think I’m crazy if I tried to convince you to become a hunter-gatherer, it wouldn’t have made much sense to the humans eight thousand years ago to do the opposite.

Whether or not you subscribe to the dissenting view or the prevalent one, I think this is an extreme example of what I would call a local view of history, a phenomenon related to a kind of confirmation bias we have about events that have happened recently. In other words, we may be blind to a fact that what we’re experiencing is not what it appears to be because we haven’t stepped back enough and considered a broad enough historical perspective. Often, what appears as fundamental properties of our society are just arbitrary results of a chain of events that seem irrelevant and relatively insignificant.

For example, a lot of Americans consider eating food very pleasurable, but — arguably — prior to Julia Child’s efforts, food had a much more utilitarian function in the American society. Similarly, we value financial success but that has not always been the case, as the pursuit of material comfort has its roots in Calvinism.

Taking this notion to the extreme, perhaps agriculture — and with it the notion of progress — is also arbitrary; zooming out enough (more than eight thousand years to the times of our hunter-gatherer ancestors) displays an entirely different flavor of society that may not necessarily be inferior to ours. What, therefore, is progress, and is it a good thing?

On a total tangent, the topic of agriculture as an unlikely villain has been raised in a freakish display of coincidence in three very distinct sources I happen to be a subscriber of — PRI’s To The Best of Our Knowledge, BBC World News, and a Stuff You Should Know podcast. It happens every so often and I am beginning to form a theory that it’s a phenomenon rather than just coincidence — similar to how studios sometimes come up with near-identical movies within few months of one another.

The latter source remarks that it is not the desire for efficiency that caused us to adopt agriculture, but the desire to brew alcohol. Beer, it is quite likely, may have led to human civilization.

The Hairball of Air Travel

Sunday, June 6th, 2010

Why is air travel significantly more complicated than alternative forms of transportation? This is true not just for the actual duration of travel (I don’t think I have to say how uncomfortable airplanes are relative to, say, cars and trains), but the entire system, from booking to departing.

Airlines overbook their flights, which means that it’s possible that you could pay for a flight but not be able to fly.

Booking has always been difficult. In the past, travel agents would make money helping you find the right schedule amidst all the constraints. I remember when you could get a better deal on tickets if you got this special student card, but it all smelled like some kind of scheme since nobody at the airport verified that I actually had the card.

Now, web startups have taken up that role and booking is not as hard as it used to be but it still took me several hours to find a good way to fly form New York to London, stay there for a day, then fly home, then fly with my family to Italy, finally to fly my family back home, and fly myself back to London, and New York.

There are all sorts of booking codes, miles, elite points, brownie points, statuses, colors and precious metals associated with flying, unlike for other forms of transportation. Like, when you can redeem miles but if you only have gold status you will have to use up 60% more miles to fly during blackout dates. Blackout dates? Huh??

If you miss your train, you can still use the ticket later. The same is not true for flights.

There are all sorts of limitations regarding changing your flights or canceling them.

Airlines have all these funny luggage restrictions.

I’m violating the airline rules by writing this right now because apparently it’s unsafe to have electronics turned on when the plane is about to land (why, exactly, is that? Will my laptop cause interference with the plane’s electronics? Really? If that’s an actual risk then I don’t want to by flying!)

The food has become so bad, and so scarce — a snack for a seven hour flight?!

There are a couple of explanations. Either air travel is fundamentally a different kind of beast, a highly constrained problem, which requires it to be governed by highly complex rules. Or, air travel is the form of transportation that has seen the most corporate development (in the pursuit of profit) and the others like rail travel simply haven’t caught up yet. Or, it’s a relatively immature form of transportation and everyone’s running like chickens with their heads cut off.

It’s probably a little bit of each of these. The worst thing is that we all seem to have become numb to all this.

Fake Shutter Sound

Monday, May 10th, 2010

It’s absolutely ridiculous, when a small square device like an iPhone makes this over-the-top sound that comes from some 80s-era Soviet camera. I know, for some reason it’s reassuring to people to hear a shutter sound when they take a picture, but since we have departed from the idea of a camera sufficiently already (Gedankenexperiment: how many of the parts found in a camera from two decades ago can be found in an iPhone?), we may just as well drop that ridiculous one.