Encapsulation Considered Harmful

You heard me. Encapsulation is an obstacle to the reuse of code.

When I say encapsulation, I mean having a region of your program that knows or has access to some information about the implementation of something, and hiding that information from the rest of the program. If you have another definition of encapsulation, I’m not arguing against that.

Why do software engineers encapsulate? I claim it is for two major reasons: (1) to reserve the right to change the encapsulated code later without breaking anything, and (2) to minimize the propagation of assumptions through the program. To illustrate this, consider the following C code which uses a (supposedly) abstract List type:

int sum(List* list) {
    int i;
    int accum = 0;
    int length = List_length(list);
    for (i = 0; i < length; ++i) {
        accum += ((int*)list)[i];
    }
    return accum;
}

This code breaks encapsulation. It assumes that List is represented as an array. So if we wanted to go back and change List to a linked list because it is more efficient in the majority of cases for which we are using it, sum would break. (In this example, it would break very badly — i.e. it might not even segfault but instead return some nonsense number)

Here is an example of code that correctly respects encapsulation.

int sum(List* list) {
    int i;
    int accum = 0;
    int length = List_length(list);
    for (i = 0; i < length; ++i) {
        accum += List_get_elem(list, i);
    }
    return accum;
}

If you’re about to object that we should have exposed an iterator interface instead of a indexed get interface because we knew we wanted to change it to a linked list, I preemptively respond that we didn’t know that at the time — as in most code, when you are encapsulating, you don’t usually know what you’re going to come back to change — if you did, why didn’t you just write it like that in the first place?

The two reasons above are noble. They promote flexibility and simplicity, so that the minor decisions we make do not ripple their way through our architecture, making it impossible to change or understand. It takes the pressure off.

Although I agree with the goals, I believe this is the wrong solution. I think we should turn the barriers inside-out, and use only encapsulation’s dual: abstraction. When I say abstraction, I mean code that is defined polymorphically with respect to its assumptions. The above C code is abstract with respect to the list argument, since we can pass it any list we like. It is not, however, abstract with respect to the the List type and its operations — it fixes a single choice.

Encapsulation and abstraction are duals in the following sense: let’s say your program is defined in two parts A and B. A knows some information C, and B does not. There are two ways to look at this. A encapsulates C, or B is abstract with respect to C. That is, if you change C, you have only affected the specification of A, and B reacts polymorphically. So just divide your program at an information boundary — one side is encapsulating, the other side is abstracting.

While mathematically they are two sides of the same coin, there is a very real software engineering difference between them. Let’s say you are have three functions which use your abstract interface: sort, reverse, and frobnicate (some complicated business logic that can’t be written in 5 lines). With encapsulation, you might have this sort of usage graph:

And then, in one fell swoop, you can change it to:

That’s power, that’s flexibility. But… it’s lacking something. Now we have another part of our program which uses arrays, and I sure wish we could use that sort code we’ve already written. But we need that sort code for lists. Hmm, well we could copy and paste. But that sucks, maybe we could go back and make sort polymorphic in the list type. Yeah, that’s the right way to do it.

But, then why didn’t we just do that in the first place? Look what happens when we do:

Look at all those combinations. Those are the possibilities for code reuse without changing anything. Imagine if every encapsulation you made was an abstraction instead. Your usage graph would be basically black with arrows. But these aren’t bad dependency arrows, these are good reuse arrows. These are your possibilities for correctly using code that has already been written.

So I suggest: instead of advertising your guarantees and hiding details, advertise your assumptions and abstract over details. This has a pretty profound effect on the way your code is structured — like I said before, it turns all the encapsulation barriers inside-out.

My inspiration for this idea came from studying the form of mathematical theorems. They advertise their assumptions prolifically: “given a ring R with (+) and (*) as operations, …”, as opposed to “a real number has (+) and (*) and some implementation details”. This allows theorems to be maximally reusable, since although the mathematician was thinking about real numbers when he proved the theorem, he realized the same logic could work on any ring — including the integers, modular integers, matrices, and new things that were discovered later.

Object oriented programming seems to get close. One can consider a class (or an interface) as the specification of an assumption, and an instance the implementation of an assumption. Then you just take parameters that correspond to your assumptions, which are nicely bundled into classes so you don’t have to say “I use (+) and (*) and (-) and absolute value and …”. But this essential idea is muddied up by all sorts of crap, both in the language design and in popular usage.

I started to list the features we need to add/remove from OO languages to make them support this style, but it just got long and nitpicky. So I’ll just say this: I think we should be using objects differently. Objects as implementations of assumptions, not as “agents”. A natural number is not an object, The Natural Numbers is an object. As generics gurus are noticing, this:

interface Number {
    Number operator + (Number y);
}

has got to go. The assumptions are not specified properly — you are requiring that + take any kind of number whatsoever, when we probably mean the same kind of number as on the left side. What does it mean to add 2 (mod 4) to 7 (mod 13)? Instead we are learning to write:

interface Numeric<N> {
    N operator + (N x, N y);
}

That’s the right way. Notice that, although it is specified with different notation, N is just another abstract member of Numeric. We specify the set and its operations together, but the set is distinct from the interface. The interface is the name for the assumption of such a set. In some cases the interface and the set can be bundled into one, and it is these cases that provide most of the study material for OO design. But inside the soul of pop OO lies a form of modeling that is so much more powerful.

To reiterate, this is not breaking the guarantees of encapsulation — anybody who uses Numeric gets encapsulation for free. They are not allowed to see the details of N, just as if you had encapsulated it. It’s just that now you can swap out different N’s in the same code, where you couldn’t before.

I am pleased to see software engineering practice slowly converging on this already. I just thought I would point out the uniform rule behind it. Next time you write an encapsulation, ask yourself, what would it look like to make the users of this code abstract instead? Is it a small enough notational burden to do it that way? If not, what kind of language would allow it to be?

Did that give you something to ponder about? Flattr this

About these ads

31 thoughts on “Encapsulation Considered Harmful

  1. So, if I understand you correctly, encapsulation is when the provider forces decoupling, and abstraction is when the consumer forces decoupling.

    One argument for encapsulation is that it allows you to ship a library used by people you know nothing about, change internal details, and be reasonably confident that you didn’t break their code. Of course, this is assuming they don’t reverse engineer your interface and use undocumented functions (hi Windows!), so this may not actually work in practice. :-)

  2. glib lists have a foreach function for lists which makes the whole discussion pointless. You don’t need to be polymorphic on the list type. You don’t need to be polymorphic at all since you don’t have to know how is implementing the list. Your function submitted to the g_list_foreach could have just as easily been submitted to g_nodetraverse and your code doesn’t have to know.

    Encapsulation with good interfaces mean you have decoupled software. Decoupled software is maintainable because you can change a library without anyone else having to recompile. If this didn’t happen glib could never change without breaking the whole world.

    Take some time to examine the interfaces presented in glib and how it’s decoupled. Consider the implications for compile time, link time, and run time. Also, take a look at what Rob Pike has to say on the subject of Function Pointers: http://doc.cat-v.org/bell_labs/pikestyle

    Otherwise, this is a well written post, imo.

  3. I think the C++ STL is a good example of this idea: set, vector, list, deque and so on all expose enough of their internals to nail down the implementation pretty well (due to requirements on the element type they abstract over, iterator invalidation, contiguous layout guarantees, performance guarantees and the like) without letting you violate their invariants. And conversely, the algorithms abstract over the type of container they’re used on, requiring only (say) a forward iterator. And the result is a very effective, efficient, reusable library.

  4. Elements of Programming by Stepanov and McJones is basically a book written around these ideas. They do exactly “My inspiration for this idea came from studying the form of mathematical theorems. They advertise their assumptions prolifically: “given a ring R with (+) and (*) as operations, …”, as opposed to “a real number has (+) and (*) and some implementation details”.” throughout the book. If you’re not already familiar with it I would strongly suggest reading it.

  5. Edward Z. Yang, you didn’t actually give an argument for encapsulation over abstraction, because if we already assume that our consumer does not use undocumented functions then we can just rename any internal details we change so that any consumer that relies on the internal details we changed will automatically break.

    Ewan, you missed the point. Write the sum function using g_list_foreach then write the sum function using g_nodetraverse; the same program might need both sum functions. How do you eliminate the code duplication between the two sum functions, even if they both submit the same loop-body callback function to g_list_foreach and g_nodetraverse?

  6. Maybe I’m being dumb, but I don’t see what the difference is. The client of an encapsulated module has type (∃α.A(α)) → B, and the type Luke is proposing we use is apparently ∀α.A(α) → B. But these two types are isomorphic! (∃α.A(α)) → B ≃ ∀α.A(α) → B. What am I missing?

  7. @Neel, that isomorphism is exactly the point! If only our programming languages saw it so clearly, this post would not be necessary. The trouble is that most languages allow you to instantiate α at different points in the latter case and not in the former (in the same program — the former usually has “link time polymorphism”, which is substantially less flexible).

  8. @Neel: While, theoretically, those two types are isomorphic the point is that we should stop and think about what isomorphism means. In particular, it does not mean equality. Isomorphisms are non-trivial, and since we have to deal with actual implementations in bits (unlike mathematicians) we cannot simply use isomorphisms freely.

    Moreover, OO programming doesn’t actually use the existential type; the name of the classes is hoisted out to the top level of the program, not bound locally. Which means that given our program parts A and B, in the encapsulation implementation the interaction has the type (∃α. A α → B (A α)), which is to say that there exists an implementation, α, of A and B is specified to use the *one* implementation available. What we really want is the interaction type (∃α. (a : A α)) → B a, where B is not predicated on α, but rather is predicated on any particular instance, a, of the type A. With this latter interaction type, the type of B becomes universally polymorphic over α and so we can have multiple Bs with different implementations.

    What this means is that in the encapsulation version, we can replace α by α’ without breaking B, but we cannot reuse B for both α and α’ simultaneously. Whereas in the abstraction version we are free to reuse B simultaneously for multiple implementations of A. This, IMO, is the crucial difference between encapsulation-oriented and abstraction-oriented programs.

  9. Chung-chieh said:
    “Ewan, you missed the point. Write the sum function using g_list_foreach then write the sum function using g_nodetraverse; the same program might need both sum functions. How do you eliminate the code duplication between the two sum functions, even if they both submit the same loop-body callback function to g_list_foreach and g_nodetraverse?”

    How do you mean? You use the same loop-body callback function with user_data pointing to your collected state. There is no code duplication at all here.

    In fact Luke’s original ‘sum’ code is the same thing twice but just replaces the assumption that lists are implemented in terms of arrays with the assumption that there exists a List_get_elem function which understands how to get the element. The net number of assumptions hasn’t gone down. (Though now it’s possibly worse since accessing i’th elements of simple linked lists is an n squared traversal when summing the contents of the list).

    In other words, he’s just replaces (*) in the Ring (or Group) being defined as operator[] with being defined as List_get_elem.

  10. Sure.

    #include
    #include
    #include

    static void sum_cb(gpointer data, gpointer user_data) {
    *(int*)user_data += *(int*)data;
    }

    static void node_helper(GNode* node, gpointer data) {
    sum_cb(node->data, data);
    }

    int main(int argc, char* argv[]) {
    {
    GList *mylist = 0;
    for(int i = 0; i < 10; ++i) {
    int* x = (int*)malloc(sizeof(int));
    *x = i;
    mylist = g_list_append(mylist, x);
    }

    int sum = 0;
    g_list_foreach(mylist, sum_cb, &sum);

    printf("mylist sum: %d\n", sum);

    //g_list_free_full(mylist);
    g_list_free(mylist); // can't use free_full as I don't have 2.28 on this machine
    }

    {
    GNode* mytree = g_node_new(0);
    for(int i = 0; i < 10; ++i) {
    int* x = (int*)malloc(sizeof(int));
    *x = i;
    g_node_insert(mytree, -1, g_node_new(x));
    }

    int sum = 0;
    g_node_children_foreach(mytree,
    G_TRAVERSE_LEAVES,
    &node_helper,
    &sum);

    printf("mytree sum: %d\n", sum);
    g_node_destroy(mytree);
    }

    return 0;
    }

    So here we have a single sum function which does the collection of values based on the underlying values and can be run against a list and a tree. The foreach function signatures differ and we get around that using node_helper.

  11. Ewan, “int sum = 0;” (and its placement before the traversal(s)) is the code duplication I meant.

    Even though it’s just one line of code in this example, I hope you see that there would be more code duplication in a larger example, such as replacing every element of a collection by the minimum element or comparing the element sequences of two collections.

  12. @Neel Krishnaswami

    Neel, loved your comment, but am a little confused.

    “Maybe I’m being dumb, but I don’t see what the difference is. The client of an encapsulated module has type (∃α.A(α)) → B, and the type Luke is proposing we use is apparently ∀α.A(α) → B. But these two types are isomorphic! (∃α.A(α)) → B ≃ ∀α.A(α) → B.”

    Could you write in english what that means?

    Here’s my interpretation: please correct me if I’m wrong:

    (∃α.A(α)) → B : There exists an implementation, alpha, of type A which means to type B. (In OO, this would be: there exists an object, alpha, of class A which maps to class B.) And I presume the mapping is done by some function.

    Also, Luke, great post. I’ve read it three times now and would love to ask you a question or two; I just haven’t understood it well enough yet. Any day now …

    Regards,

    Ed.

  13. Hej, Luke,

    Again, thanks for the great post.

    Unless I’ve misunderstood your point (and I’ve won medals in misunderstanding), I must, however, argue respectfully that your conclusion does not follow from your premises.

    Firstly, congratulations on your definition of, “Encapsulation.” The Reference Model of Open Distributed Processing [10] – the International Organisation for Standardization’s International Standard – defines encapsulation as, “The property that the information contained in the model of an entity is accessible only through interactions at the interfaces supported by the model.” Taking a pinch of consensus, we can say that information that is not accessible through these interfaces is then, “Information-hidden.” This startling closely resembles your definition, so we’ve at least begun on common ground.

    I hope you don’t mind if we switch languages to Java, the only language with which I’m proficient.

    Let’s say my Java program concerns an automated car assembly and I have two packages (subsystems, if you like): the engine package and the gearbox package. Both are well-encapsulated in that all their classes are non-public, i.e., they are information-hidden within their packages. One class of each package, however, implements a public interface whose sole function is to return a component from each package.

    A class in the engine package thus supports this interface:

    public interface EngineCarPart {
    public Engine getComponent();
    }

    The gearbox supports this interface:

    public interface GearboxCarPart {
    public Gearbox getComponent();
    }

    I think – and again, please correct me if I’m wrong – that what you’re suggesting is that this encapsulation is considered harmful and that polymorphically typing the getComponent() method with the returned generic car-part type would be a better approach, so that we could scrap the two interfaces above and replace them with one:

    public interface CarPart<Part> {
    public Part getComponent();
    }

    And, indeed, I think you hint that this interface could now be re-used by all the other subsystems, too, for example the electrical or hydraullic subsystems, thus showing that this polymorphic typing of the function is better than encapsulation.

    For me, if you say that, “Encapsulation [is] considered harmful,” and show an alternative, then you must demonstrate how encapsulation and its harmfulness has, in some sense, been reduced or eliminated. Yet I cannot see that you have demonstrated this.

    Going back to the definition, encapsulation says that you can only access packages via their interfaces, and it is understood that the fewer interfaces there are to a package, the more well-encapsulated it is (the least-encapsulated package you can have is a package whose every class and interface are public and thus accessible to the entire system). In the first car example above, there are two interfaces in two packages; in the second, which I claim uses your encapsulation-alternative, there is now one interface shared between two packages, and thus there is half an interface per package.

    This reduction in the number of interfaces per package represents an increase of encapsulation, not a decrease.

    Have I completely missed your point?

    Sincerely,

    Ed Kirwan.

  14. Ed, I think you have missed the aspect of my argument concerning the duality of abstraction and encapsulation. Grok the paragraph beginning “Encapsulation and abstraction are duals in the following sense:”. It is important to the argument.

    I would like to respond to your example program, but I am not seeing it completely. Do you mind if I ask you to fill in a few details by answering these questions?

    EngineCarPart.getComponent() returns an Engine, which I believe you said was hidden. How does that work? Is Engine treated as abstract to the client code — no methods, just an opaque identifier that can be passed around? Or are its methods now available, and hiding it only prevents the client code from creating new Engines of its own? If the latter, what are a couple of its methods? Can you give an example of those methods being used, and tell me where the usage occurs in your program?

    Sorry for being nitpicky. The transformation from encapsulation to abstraction is a whole program transformation. It’s not possible to look at a library interface and change it from one to the other — the code using the library is profoundly affected in the process.

  15. Nice post, Luke– an intuitive exploration of parametricity from a software engineering perspective.

    Here’s the part I think you don’t take into account: separate development. When I write a library, and I have thousands of clients, I can’t control whether they write their code polymorphically. So by using encapsulation (roughly, by using an existential type), I can *force* my clients to be abstract, thereby retaining flexibility to change implementation and managing internal invariants without requiring (hoping) my clients preserve them.

    BTW, I don’t buy your argument that you never know which parts of your code are likely to change (“if you did, why didn’t you just write it like that in the first place?”). Planning for change is a part of engineering. There are all sorts of reasons why you can expect a particular piece of code will change: getting it correct first and optimizing later, tuning based on changing conditions, etc.

    Dave

  16. @Dave, you are treating your thousands of clients as if you own them. They are using your library to make their software. It is their responsibility to ensure that their code is resistant to your changes, not yours. Maybe the current version of your library does just what they need, they don’t need to worry about upgrades. Or maybe they have extremely strict requirements, and what you consider to be an innocuous bugfix, they consider to be a compatibility breakage that needs their attention.

    When you encapsulate, you only ever prevent people from using code that has already been written. If a client needs code that you have encapsulated, they will get it. Perhaps by copying and pasting from your library, perhaps by writing it themselves. Both of those are harder than letting them use it from your library. If you change your library in a way that breaks their code, well, then they probably won’t use the latest version of your library. Or they will pick and choose the pieces they need from different versions of your library and use them all at once.

    Instead of forcing your code to be used in a certain way, simply express properties about your code. If you need plugins to remain stable for your next release, just say so: “for code to be considered a plugin, it needs to be abstract in the following parameters”. And then you are free to change those parameters, and plugins are guaranteed to continue to work. Programs that reuse your code in other ways might break, but the alternative may be for them not to exist at all. It’s important to remember that not all reuse of your code is going to be in the form of a plugin — so characterize what it means to be a plugin, don’t prevent your clients from writing things that aren’t plugins.

  17. Luke :
    @Dave, you are treating your thousands of clients as if you own them.

    Wow. I’m not really sure how to respond to that. You got me: my true motivation is to use software engineering in the service of human bondage.

    Seriously, I’m treating my clients as people who have a need I am trying to serve. If I can serve that need without exposing the mechanism for doing so, I make the solution more amenable to change.

    They are using your library to make their software. It is their responsibility to ensure that their code is resistant to your changes, not yours.

    Given enough clients, they will depend on leaked details. And if your only response to the bug reports that ensue from incompatible change is “that’s your fault” your clients will not be satisfied.

    When you encapsulate, you only ever prevent people from using code that has already been written.

    This is where the hyperbole of your argument really loses me. If X makes use of Y internally, and Y is useful, why not release Y independently? The internal use of Y in X doesn’t need to be exposed.

    If a client needs code that you have encapsulated, they will get it.

    Indeed they will.

    Dave

  18. @Dave, sorry, I didn’t mean for my argument to be hyperbolic, but my first sentence kind of indicated such a tone.

    If X makes use of Y internally, and Y is useful, why not release Y independently? The internal use of Y in X doesn’t need to be exposed.

    I suspect this is the source of our disagreement, and probably has something to do with our talking about abstract names like X and Y and the vague word “use”. Let’s make this discussion more formal and concrete. You are a Haskeller, no?

    Let’s take an example that is close to home for me, a drawing library. We have a type Image, which is implemented as a compositional interface to OpenGL: Image = IO (). We may wish to change this implementation in the future, so we would like to export Image as an abstract data type. Here’s the thing though, when I released this library, I didn’t write very many primitives. Just circle and square. One of my clients needs triangle, what are they to do? One rule — you can’t ask for the drawing library to have been designed differently. Bad design abounds in software engineering, and we should be able to achieve code reuse nonetheless.

    They could write triangle as OpenGL commands themselves, but then they can’t use my combinators to combine that with images constructed with my library. They could fork my library and add triangle to my source code. The former is clearly not optimal, the latter is the same as breaking my encapsulation, because I won’t be there to change their triangle to my new representation (this is unavoidable, no matter which way the client gets triangle this will happen).

    Let’s now go to an alternate universe where I was not allowed to encapsulate. The way to expose an abstract data type would be to describe its interface: data ImageModule = ImageModule { Image :: *, transform :: Affine -> Image -> Image, circle :: Image, square :: Image }. Then I expose alongside it an implementation: imageIOModule = ImageModule { Image = IO (), ... }.

    The normal user would phrase his code like: forall Im :: ImageModule, .... Any code under such a quantifier is just as abstract as it was when I had encapsulated. Except now the guy who needs triangle can get it by coupling to imageIOModule: Im = imageIOModule; triangle :: Im.Image = .... If I change my representation it will break, but as I already said, this is unavoidable, since I have no knowledge that triangle even exists. At least he gets his application to work, and he can use the combinators I have written.

    Did that clarify things? If not, could you use the triangle dilemma as an example?

    I realize that no popular languages make this kind of program structure pleasant to work with. Agda does it pretty well. I’m writing in order to inspire a desire for language features that make this pleasant.

  19. Hmm, I realize I didn’t really respond to your concerns. But at least we have a concrete example to discuss.

    Maybe this is what is going on. We are assuming different things about the parties involved. You trust the library creator to be responsible and to design well — for example, releasing Y independently — and are assuming the worst of your clients. Perhaps you have been in charge of a library with a thousand clients and have seen the crazy shit they do.

    I am arguing from exactly the opposite perspective: I am assuming the worst of the library designer and trusting the clients to be responsible — for example, abstracting over ImageModule instead of using imageIOModule directly. This is because I have had the displeasure of attempting to reuse code from some really badly designed libraries. So my arguments are coming from a place of more immediate meaning for me, thus I am considering them more important.

    How do we reconcile these? How can we assume the worst of both parties and still get the modularity properties we desire?

  20. Honestly, I’m not really sure what I think. The idea is interesting.

    BTW, I’m not really a Haskeller, though I’ve written a reasonable amount of Haskell code. I’m coming at this from the perspective of the web and particularly JavaScript libraries, where the quality of the libraries is typically much better than the quality of the clients. In JS the decks are stacked even worse, given that exposing representation also exposes you to third-party mutation.

    (A tangent… this is a *very* common misunderstanding in the OO world, given the ubiquity of mutation: people often think encapsulation is just about protecting your implementation against mutation, rather than about representation hiding.)

    But back to your point: another way to look at this might be the idea of providing multiple layers of abstraction. In your Image example, you allow clients to choose: do I want to operate at the higher level or the lower level? Even better, by making those levels distinct, i.e. by providing two separate interfaces (ImageModule and imageIOModule), you make it clearer which level the user is operating at. This stratification also makes for a clearer story you can tell to your clients:

    - If you operate at the higher level, the API will be more stable — it may grow in the number of operations it supports, but backwards-incompatible changes will be less common etc. etc.

    - If you operate at the lower level, the API is less stable and more subject to change.

    A design like this just looks very nice. What I am having a harder time doing now is mapping it to the encapsulation providing by closures (or in the OO world, private fields). The idea that clients should be able to peek inside closures seems absurd on its face. But closures provide implementation hiding at the value level just like existential types do at the type level. If encapsulation is considered harmful, what would you suggest for library writers instead of closing over private implementation details?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s