The “Whole Program” Fallacy

This StackOverflow question has generated a buzz of zealous reactions in the Haskell community. Here are the important bits of the question:

I often find this pattern in Haskell code:

options :: MVar OptionRecord
options = unsafePerformIO $ newEmptyMVar

doSomething :: Foo -> Bar
doSomething = unsafePerformIO $ do
  opt <- readMVar options
  doSomething' where ...

Basically, one has a record of options or something similar, that is initially set at the programs beginning. As the programmer is lazy, he don’t wants to carry the options record all over the program. … Now each part of the program has to use unsafePerformIO again, just to extract the options.

In my opinion, such a variable is considered pragmatically pure (don’t beat me). …

In this post I will give my own zealous reaction.

To ask a question like this assumes something about the nature of software. The assumption is hiding in these phrases: all over the program, each part of the program. Here, the poster assumes that a program is a large, monolithic beast such that every part of it will need access to this variable, and yet the definition of this variable is not known to the programmer. That means that the program depends on this value. If we purely model the structure of the above example program, we see that every function depends on OptionRecord. So we have (taking the context of a compiler):

parse :: OptionRecord -> String -> AST
compile :: OptionRecord -> AST -> LinkerObject
simplify :: OptionRecord -> AST -> AST
freeVars :: OptionRecord -> AST -> Set Variable
safeName :: OptionRecord -> Set Variable -> Variable -> Variable

These are perhaps not the cleanest signatures for parse, compile, and simplify, but they are conceivable in the real world. There is some junk — surely not all three of those functions depend on every option of OptionRecord. It would be cleaner to declare that they depend on exactly the things they actually depend on.

But the problem becomes much more unsettling at freeVars. freeVars takes an OptionRecord — staying true to the original problem description, it must, because it or a function it calls may end up depending options. But what on earth could a global OptionRecord determine about a free variables function? Perhaps there are multiple ways to find free variables — do we count type variables, what scoping mechanism to use — but those are not global options. Different functions will require different behaviors out of that function depending on what they are doing.

We even get such pathologies as shortestPath :: OptionRecord -> Graph -> Node -> Node -> [Node] — a plain, simple, reusable graph algorithm which somehow depends on this global options record. We have no way of telling the compiler — or, more importantly, ourselves — that this algorithm really has nothing to do with the specific compiler we are implementing. Somewhere deep in shortestPath‘s call chain, there is a call to some function which calls an error function which depends on one of the options. Suddenly this beautiful, well-defined function is not reusable. To take it out and use it in another project means to include OptionsRecord in that project, and OptionsRecord has things about compiler and type system extensions, configuration files, who-knows-what, but certainly having nothing to do with graphs. Sure, we can go and dig out the OptionsRecord, replace it with a record that is more suited to the program we are reusing the code in. But you have to go read, understand, mutate code that you just want to work please so you can get on with your project. We have all suffered the head-throbbing pain of integration problems. This is their source.

When I think of software as thousands of lines of specification for something, my mind jumps to problems like the original question. How am I going to write something so huge purely without it being really inconvenient? I see the need for global options, often global state, things ending with Manager (often a global trying to convince you it is a good abstraction), big systems talking about big ideas which are only applicable to my project.

But I have begun to think about software another way. Consider 100 lines. That is potentially a lot of information. The only reason 100 lines is not very much in the code world is because we lack the vocabulary to say what we mean. We are caught up in the details of manipulating lists of identifiers, building sorting trees, defining what we mean by “first” in this or that context. Could you describe your project in 100 lines of English? Perhaps not, but you could get a lot closer than with 100 lines of code.

I’m beginning to think that my latest greatest software project should be as small as possible. I need to build up vocabulary so I can describe it in a small space, but that vocabulary is not a part of my project. That vocabulary belongs to everyone in the same-ish domain of software. And nobody else cares about the meaning of OptionsRecord.

When I think of software this way, the idea that I need to pass around an OptionsRecord as a parameter to every function in my project starts to make sense. Every part of my project depends on its options, but my project is just a little thing that is standing on the shoulders of the giants of vocabulary used to get it there. I don’t mind passing options around between a few functions after I load them.

This is an ideal. I hope to move software closer to this ideal with my CodeCatalog project. Your current project probably cannot be phrased in a few hundred lines right now. But think about what it would look like if you structured it as a small main program + a lot of supporting vocabulary. Think of the supporting vocabulary as something any project could use. What does that mean for the modularity, reusability, and adaptability of your code? What does it mean for the clarity of your specification? What does it mean about the concept of “architecture”?

Flattr this

About these ads

10 thoughts on “The “Whole Program” Fallacy

  1. Interesting post on unsafePerformIO. If you ask my I am a bit of a purist to and I would much rather make uglier functions with more arguments than they strictly need then make unsafePerformIO operations.

  2. I thought that this kind of thinking is the reason people designed programming languages which have abstractions in the first place.

    The main reason we don’t see programs designed in this way, is that they tend to be the work of one person. This is for two reasons. Firstly, building a concise program takes time. Usually a commercially acceptable solution can be put together faster using copy and paste, so concise code is a work of love, or the result of decades of experience. The second reason is that as soon as multiple people start to collaborate, they must all understand the reasoning behind the concise description. By definition, the concise description is a highly dense structure that may be beautiful, but still takes time to comprehend properly. Usually the collaborators will fear the process of building this understanding because they are used to things being fairly obvious, and code that is hard to understand because it’s concise is not easy for the inexperienced person to distinguish frok code that is hard to understand because it is confused.

    Thus, there are emotional barriers to this way of coding that only a determined individual or truly inspired group will overcome.

  3. You make a good point. Personally, I have trouble imagining why I would need to pass Options through every function in programs I write. If a program is modular, it should not happen. If it does happen, obviously your code is not as reusable as it could be. Part of reusability is minimizing dependencies.

    There is another way to think about this issue. Even a small program usually depends on lots of libraries. If you substituted the library code into your small program instead of importing it, you would have a big program, and the vast majority of it has evidently (because it was in a library) been written in such a way as not to depend on your Options. So in a way there are no small programs, yet there are evidently big programs which avoid depending on Options.

    Two more things.

    First, I think many people pass Options around monolothically, because it is not convenient to invent new sets of datatypes which are factorizations of Options. But in a good program it ought to be the case, say, that only 50% of your modules/functions depend on only 50% of the configuration options, so you should be splitting it up as you pass it down.

    Second, I think it is worth observing that Options will necessarily be quite ad hoc, because the language of the command line (or whatever) is far less sophisticated than Haskell itself. That means that the way in which you can parametrize the program/functions via flags, etc. is going to be very crude compared to the way in which you can parametrize it in Haskell. So it should often be the case that your functions can do more than what you can express via Options.

    Your freevars is a good example. If it is the case that freevars depends on Options, it is probably also the case that you can write a more general function freevars’ which does not depend on Options, and the freevars factorizes through freevars’. So, for example, freevars’ might find free variables of all sorts, while freevars only finds free variables of one sort, and freevars could be expressed as a specialization freevars = freevars’ LinearVariables, or something.

    That said, I don’t see any harm in making configuration options global and pure. While it is true that, say, argv could be updated externally during a program run, I don’t find this relevant: you can just define the semantics of a configuration global as being a copy of argv made just before the Haskell program launches. That is, IMO, the intended semantics anyway.

  4. Wow! The reaction to my SO question is surprisingly high. In the first place, I asked it because I saw this pattern quite often (though not always with that many calls to unsafePerformIO). My thought was: Isn’t it possible to add global “state” in a more convenient way? So I asked this question.

    In my opinion, the best solution is implicit parameters. They give you exactly what you need – adding some sort of state but avoiding cluttering the function’s definition.

  5. Can one solve this with Template Haskell?

    If you could look the current definition of a function up and modify it, then couldn’t you add arguments to it?

    I often find myself writing (\x -> trace (“State ” ++ show x) x) in my code and then commenting it out as {-(\x -> trace (“State ” ++ show x) x) -} because I want to keep the argument lists short.

    Instead it would be nice if I could write “debugMsg State”” and not change the function arguments but have the behavior change depending on command line flags. I could do that with unsafePerformIO which should be quite safe since the command line flags are set only once. However this is a general problem and I’m wondering if there is a better solution.

    An alternative would be if I could use template Haskell to update the functions that invoke debugMsg to have an additional argument. Basically I’d write some template haskell to look for all functions in which debugMsg appears. The template haskell would update these functions’ argument lists and add a “options” parameter. It would replace debugMsg by “debugMsg’ options”. (And it would do this recursively for all functions invoking the functions calling debugMsg). The result would be a program with the options thing threaded through implicitly.

    Can Template Haskell do this?

  6. Actually it seems I’ve just suggested implicit parameters which apparently already exist.

  7. Unfortunately even Simon Peyton-Jones suggests using unsafePerformIO in order to handle configure files (that are much like global options) in “Tackling the awkward squad”, page 15. So far I was very happy not using unsafePerformIO for this purpose.

  8. You happen to have rediscovered a great PL idea of the last years. You probably want to read the keynote speech “Growing a language”, from Guy Steele, or watch the video.
    The paper starts out really crazy, but it’s worth it: I also find it amazing that when you are (as famous as) Guy Steele, you can afford leaving your reader/audience totally confused for 10 minutes to make your point in such an impressive way.
    This paper is also important in having inspired a big line of research to develop growable languages.
    Furthermore, the general idea of “having a small program using a wide vocabulary” is nowadays often expressed through Domain-Specific Languages, and DSL-oriented programming. The main-difference is that vocabulary extensions are domain-related.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s