Tuesday 11 December 2007

Bottom-Up Development - Embracing Change

I have just read Raganwald's post, Something's Fishy, and it made me start thinking hard. One statement in particular caught my attention:
"Now maybe this [bad software] has absolutely nothing to do with programmers."
which spurred me to start writing. I was going to add my thoughts as a comment on Raganwald's blog but once I started writing I couldn't stop! Also, I have just started this blog so I thought, why not - I'll publish my brain-dump here.

In my experience, the fact that writing software is hard is the ultimate driver for all the bureaucracy trust upon developers in medium to large software houses.

Managers are scarred by past experiences maintaining bug-ridden, badly designed systems so they inevitably try to find ways minimize risk by enforcing rule upon rule. They embrace the fear of "getting it wrong" and introduce layers of process from pre-planning to lengthy design phases, to requirements re-re-re-analysis and so on. Often the rules put in place are ad-hoc, subjective and designed to prevent recurrence of specific nightmares from the past which will probably never reoccur again, at least not in exactly the same way.

I believe putting blame on bad management and bureaucracy is generally futile because, while not necessarily incorrect in specific circumstances, they are just symptoms of the real problem. The real problem in my opinion is that - even when using popular dynamic languages like Python - writing bad code is way too easy to do and maintaining it can become a nightmare as time goes on and feature creep has it's wicked way with an infrastructure never designed to support the features it is being asked to support.

That's why I am so excited about discovering the power of Lisp and in particular when Paul Graham talks about bottom-up programming. I believe the main players in the software industry focus far too much on top-down design of extendible and reusable systems. This is the basis for OO and it has never, ever sat well with me. How can we, in the real world, really expect anyone to come up with a system that is designed for change when we have no idea what the change will be? We can guesstimate and plan ahead and sometimes we may get it right, but there always comes a time in a system's lifetime when the customer requests a feature that "the system wasn't designed for".

I am very new to Lisp and more generally, functional programming concepts, but when I read about bottom-up programming in On Lisp, it all makes so much sense! Maybe we shouldn't be designing for change? Maybe we should accept that code will change in ways we cannot predict, then embrace it, then begin to use tools (like Lisp) that help us to manage and maintain such code elegantly.

If tasked with writing a geographical mapping product that tracks ship movements, do we spend months writing an infrastructure to support generic objects and potential features that the customer hasn't asked for, but may well do in the future? Or do we write a geographical mapping product that tracks ship movements? Your typical OO enthusiast we say, "Oh, you can't create a Ship class, you should be more generic than that! Study your Ship class for a while - I could pick several attributes and behavioural aspects of your class that aren't specific to a ship and come up with about five base classes. Wham Bam! An extendible and reusable system!".

But I just want to write some software that meets the requirements and more importantly... works. I want to keep it simple to reduce risk and minimise the chance of it not working properly. If I break-down my Ship class into multiple base classes I am not going to use (and repeat this approach throughout the system), then I am unnecessarily complicating the code and arguably wasting time and money.

Nevertheless, the OO guy has a point. If I limit my code to exactly what is required now, then it will be difficult to extend in the future. But, crucially, if the OO guy had his way and generalised the code using a top-down approach, unless he possessed psychic powers he would undoubtedly end up with a system that was just as difficult to extend than mine (except maybe in certain specific ways he had thought of), yet his system would be far more complicated and more difficult to maintain.

I believe the bottom-up approach is better because I end up with a simple system that works NOW! But I still have the problem that my code is very difficult to extend. What I need is a tool designed to aid, manage and maintain code written in a bottom-up style. Although I have no experience using Lisp in production environments, from what I have read so far I am pretty sure this is precisely where Lisp shines, primarily because of macros. Macros combined with other powerful features of Lisp provide the programmer with the flexibility to generalise code after the fact. It allows us to write very specific code that doesn't go any further than meet current requirements (you know - the ones that actually exist!). As the system evolves and new requirements are requested by the customer, Lisp allows programmers to elegantly mould their system into a new beast capable of meeting those requirements.

Now, if I were a Lisp expert I would dive into an example of it's power at this point. But I'm afraid I am just at the beginning of my journey towards Lisp Enlightenment. I have just purchased The Little Schemer, I am almost finished my first attempt at Pratical Common Lisp and I have read the first few chapters of On Lisp. So I have a long way to go, and time permitting I will try to record some of my learning experiences here.


Unknown said...

Nice post. I have definitely had the same epiphany not that long ago. Except that I wouldn't say that Lisp is the answer for bottom up development. You can do bottom up development in really any language you choose. LISP has a lot of strengths, but it doesn't give you bottom up development for free. Personally I found the answer in Test Driven Development, but that's just one approach.

Anonymous said...

Welcome to the party...there seem to be a lot of us regular programmers learning Lisp these days. I've found Learning Lisp (notes from an average programmer studying the hard stuff) to be an interesting read. I also blog (infrequently) at Invisible Blocks about software, which for me (lately) means learning Lisp and functional programming.

Have you read any of Structure and Interpretation of Computer Programs yet, or seen the lectures? Well worth the time...I'm still working on them.

Unknown said...

Thanks for the link to lectures, I'll definitely look at those. No, I haven't read the book, but it's on my list.

My post might have come out wrong, I was not dissing learning LISP in anyway, by all means, learn it. You will become a better programmer for learning it. I was just saying that taking up the bottom up approach doesn't necessitate learning a whole new language.

Anonymous said...


I was thinking the same kind of thing about Lisp and bottom-up. I'd guess for bottom-up, a dynamic language, preferrably with some kind of meta-programming ability, would be best, all other things equal.

But I can imagine TDD providing the same kind of flexibility, if you're using a static, meta-programming-free language. I suppose it comes down to, is my team more comfortable with dynamic languages, or unit testing?

Paul Drummond said...

Thanks for the comments!

While I am very interested in learning more about TDD (I tried it in a production environment once and cocked it up! I will blog about it when I get time), I still feel the right programming language is crucial to bottom-up programming.

For example, if I had a complex Java enterprise system and was asked to add a new feature that "it wasn't designed for" which required substantial changes to the core infrastructure, due to the nature of the Java language the effort required to refactor the code could be enormous regardless of the quality of the unit tests! Unit tests help give a developer confidence to "mould the system into a new beast" without regressing but they certainly don't help make it easy or elegant.

BTW: I have read several chapters of SICP but haven't finished it yet, nor have I seen the lectures. It's high on my todo list though. Going to read The Little Schemer first, and maybe finish PCL over Christmas.