The standard way of working on software development is for a business analyst to talk to users, look at processes and work out what is needed, setting it out in some kind of large document. An IT analyst then takes this document and produces a design and specifications for how the computers will produce the effect the business person described. A bunch of programmers then follow the specification to produce a working system that fulfills the IT analyst's specification. At the end the working system is handed over to the users, who cry with joy at the sheer wonderfullness of what's been created.
What tends to _actually_ happen is that the Business Analyst has a pretty good idea of what they want, knows that anything they miss out will cause people to whine at them incessantly, and that IT projects come along once every five years. They therefore throw vast amounts of 'nice to have' items into the spec, leaving out lots of things that they believe are obvious, and put in a variety of things they can't have, as they don't actually understand the current system.
The IT analyst then mostly understands this document, works out a good way of implementing bits of it, hides the bits that look impossible, writes up a design based on their (limited) understanding of the current system and hands it to a bunch of developers who all work independently to produce something which then doesn't really hang together when they come to integrate it. It doesn't quite do what the IT analyst wanted (because the spec wasn't 100% perfect), it doesn't really do what the business analyst wanted (because the IT analyst didn't understand the business spec 100% before writing the IT spec), and it really doesn't do what the actual business user wants (because they've been pretty much left out of the whole process).
The business users then complain loudly about the programmers, who are terribly upset, because _they followed the spec_. At this point everyone goes down (separate) pubs and cries into their beer.
The problem here is not the quality of specs, the quality of the understanding of the various people involved or the laziness of the various workers. The problems is that the whole methodology is a big pile of poo.
The _current_ methodology we're trialling (to rapturous response) is this: The business people come to us with a problem. We then spend 1-4 weeks going over their processes with them and seeing where we can help, before agreeing a general, over-arching goal/direction for the project. We then agree what we are going to do _in the first month_. We do this using a very simple method - they organise their priorities into a list (with our help) and then we tell them how far down the list we can get in a month.
We then beaver away for a month, producing the stuff we said we would. We do this _with a business representative sitting with the team_. This business rep may well bring work with him to do while they're sitting there, but they're constantly available when we ask awkward questions like "When you said you wanted X, did you want X1 or X2?", the answer frequently being "Y, of course." This keeps us on general track during that month, in general.
At the end of the month, we then have a system that _does something_. This is important. It may not be finished. It may not be ready to show the world. But it _is_ in a state where we can show it off to a bunch of business people and ask for their feedback. They can then look at what we've done and decide (a) what changes they want, because of miscommunication and misunderstanding and (b) what ideas they've now had as to what it could do. We now go back to the priority list, they add, remove and move things around and we then agree what will happen in the second month.
By doing this, it is impossible to produce a system they don't want. We may go off on the wrong track for a month (although this is unlikely with a business person sitting there), but we can't go wrong for longer than that. We can't focus on functionality they don't care about, rather than the things they do, because they set the priority.
And so far, in general, it seems to work extremely well. I never have a feeling of working in a vacuum, or that I'm not doing the right thing, or that what I'm making makes no sense - because there's always someone around to give me a sense-check, and at the end of each month we get feedback about how well we're doing.
If they try to make me go back to the old way of working, I'm going to cry. A lot.