On Layers

Something I saw somewhere about cutting layers out from a software stack reminded me of something else that has been niggling me for a while. Most likely what I saw was someone talking about the Vishal Sikka keynote at SAP's TechEd conference, talking about SAP's HANA in-memory application server removing layers from the software stack.

SAP has been doing this layer-collapsing move a fair amount lately and I think this is attributable to their innovation push over the last few years. I think that's all good, though we'll see where it brings us. More on that later. But what niggles me is this idea that removing layers is always a good thing. Them there layers are actually doing something you know!

So what are layers? On the server side of a client-server (two layers right there) architecture we might be talking about database servers, application servers, and front-end rendering servers. In application the application space maybe we talk about model, view, controller architectures. Every system of any complexity has some sort of layering concept.

Why do these layers exist? What are they doing? Primarily lowering costs and allowing people to get their work done. Layers server as an abstraction, as a way to get work done within a layer without having to worry too much about what is going on outside of the layer. They are a protective bubble that allows me to do things like write this post or tweet without experiencing (increasingly) existential angst about how the bits are stored.

Yes, layers look really complex when viewed from a systems perspective, but that is usually because they are codifying an incredibly complex system into a form that we can actually comprehend. Layers force structure onto a system in order to make spaces where complexity is hidden and people can live and work.

Which brings us to the point that it's not really about what is in the layers so much as it is about the transitions between the layers - those shields that create the spaces where actual work and life gets done. Transitions between layers are interfaces of some sort. They are negotiated ways for layers to talk to each other. In programs, these are called APIs. In real life, this is called bureacracy.

APIs and bureacracy share a few things in common:

• They have lots of rules about who you have to talk to and how you have to talk to them
• Everyone thinks they are terribly designed
• They're usually pretty slow compared to going "stright to the source"
• If they didn't exist and everyone went "straight to the source" then there would be total chaos

In short, everyone hates them but nothing would get done without them.

Sometimes, layers outlive their usefulness and those who hate them overwhelm the inherent inertia of mass acceptance. When this happens in technology, it's a "technical revolution". When it happens to companies, it's a "reorganization". And when it happens to governments, it's a real "revolution.

Here's the thing: Revolutions are pretty nasty affairs.

In the political sphere, lots of people die. In the industrial sphere, people lose their jobs. And in technology people experience angst about bits.

Seriously, there have been a ton of questions about how HANA persists data. People are literally experiencing angst over how their bits are stored.

So, I know that tearing apart layers is sometimes necessary in order to get past some institutionalized road-blocks that have outlived their usefulness. Just remember that those layers you just ripped through actually were doing something useful.

Comments

[Thomas Z.]

Hi Ethan,
I follow your argument completely. I think that there is a permanent struggle between useful abstraction and performance optimisation with the latter frequently boiling down to overcome the rules, constraints, forces etc imposed by the APIs - as you rightfully raise above. In the end, it is a trade-off between the usefulness of the abstraction (i.e. the layer) and performance. The problem is that it is a trade-off / a compromise. Consequently, there will always be counter arguments (against the layer as performance is always positive). But this is reality.
On the other hand, some abstraction layers have clearly made it. Nobody argues moving back to programming in assembler as hand-crafted assembler code can perform so much better. I'd argue that a software like the one underlying HANA would not be possible with such an abstraction-free approach. Similarly, SQL has been widely accepted despite optimiser hick-ups here and there. And even when you look at LSA there is so much well established and best practice incorporated.
To close with a funny, albeit real story on the inbound layer of LSA (also frequently challenged as being unnecessary): some years ago we did analyse a latency problem for one of our customers loading data into PSA. After a few cycles it came down that the non-regular loads were caused by the fact that the data were downloaded in the evening to a USB stick, the guy then cycled through traffic to the next shop to download more, to then go to an internet cafe in order to upload the files to a central server. Sometimes traffic was bad or the cafe was busy. This example shows that even in these modern times it is of value to have all your relevant data in one location and thereby relieve you from network problems, system downtimes and other unforeseen problems that are out of your own control. The PSA layer serves this purpose.
Regards
Thomas