IMPLEMENTATION MATTERS

Conventional programming techniques tell us when creating a large software system that we should isolate components from each other using the ‘black box’ technique.

This idea behind this is simple, “why expose all the complexity of the system to third parties?”, instead you can define an abstraction for the complex system and enable interaction through this. Utilizing the abstraction provides us with many advantages; the most relevant advantage is the ability to interact with this component without understanding everything about it.

But the technique of abstracting out a complex system generates problems of it own; problems I believe cripple our ability to move (the engineering of) software forward. The following are some of my thoughts (ramblings) on this:

MONOLITHIC SOFTWARE

We cannot engineer software that doesn’t depend on implementation.

Even though we create systems that may have hundreds or thousands of interactions through approximate interfaces, this software still only works when treated as a single piece of unchanging code.

We can only engineer software we are confident works if at some point of time we lock down all codebase and make it monolithic.

While some systems (like Firefox or Eclipse) do support plug-ins at runtime, these changes are cosmetic at best, these components typically don’t have other (unexpected) components depending on them, these are simply plug-ins at the extremities of the dependency graph.

IMPLICIT CONTRACT

When we work with interfaces, we are really working with the implementation.

The myth is that if we define an interface completely enough, then the implementation doesn’t matter.

However during the development and testing cycles we are testing against an implementation of the contract, and by doing this we really no longer working with a black box system we are working with an implementation, so we are working with a contract implicitly derived from the implementation. This is sometimes referred to as a leaky abstraction.

COMPLETE CONTRACT

There is a way to define a contract so completely that its implementation will be unambiguous.

The only time you can successfully interact with a system without caring what the implementation of a black box, is if the contract is defined so completely that there is no ambiguity in the implementation. The good news is that there is such a contract! The bad news it that this contract is the implementation itself!

Even a logically equivalent implementation would not be sufficient, i.e. code that for every possible combination of inputs would give the same outputs as a different implementation. Since there could be different internal failure points and dependencies.

MINOR FLAWS MILTIPLED

Inconsistencies in implementations create flaws, which become magnified in dynamic systems.

Currently we have no way to create software that can be truly assembled at runtime with differing implementations, because it would just not work.

Imagine if using today’s technology we defined a contract for 100 different components, and all these components interact and leverage each other. We then give these 100 different components to 2 different groups of people to implement, these implementations fulfill the contract.

If we then try to run the system and for each component randomly choose which of the two implementations to use for each component, this system would never work.

SPECIFICATIONS

Speicifcations are just a contract.

Specifications are really another face to the same problem, just another way to try to define a contact without defining the implementation.

A great example of this is the attempt by Sun Microsystems to try to define the J2EE spec to make EJBs vendor neutral. The idea was that you could build an EJB for IBM’s WebSphere Application Server and then you could then deploy it on BEA’s Weblogic Server and it ‘would just work’.

In practice this was simply just not the case, even the hugely increased detail in the subsequent EJB specifications has not made them portable.

INTENTIONAL PROGRAMMING

DSLs have implementations too.

The current metaprogramming technologies also have this problem. A metaprogram is eventually implemented, the DSL itself can and will be implemented in an ambiguous way as well.

However there is good news, capturing intent on many different levels (layers) gives us the potential to have so much more information to work with.

IMPLEMENTATION MATTERS

And it’s ok! We should stop trying to pretend that everything works the same.

Thanks to all this extra information we now have a broader semantic description of a programmer’s intent.

Once we have entire systems written this way, we will then be able to do so much more.

Interacting components will not only be able to interface with each other through a well defined contract but also have a conversation about each others implementation details.

If we think of the optimizations that a game programmer typically does to get high performance rendering, even though the programmer is interacting through a generic interface (like DirectX) she is only able to truly achieve great performance if she knows a lot about the implementation of the graphics card behind the generic API.

While critical to high performance these optimizations are typically not some great insight they are just the application of gained knowledge. There is nothing here that couldn’t be automated, if the graphics card could communicate to the calling program the most performant way to structure its graphics data (texture size, byte alignment, etc…) then this system could be as fast as one coded by hand.

FUTURE

This is what I see in the future thanks to metaprogramming technologies like intentional programming, this will be one of the first times where a new technology will actually be able to run faster that the previous one. Think of a complex system which runs as fast as if it had been completely coded by hand at every level. No only this but also a system that would truly be dynamic, since it would be able to change implementations behind abstractions without the loss of stability or performance.