Having taken a position with a firm that recently launched an Agile Development initiative, I have been giving a lot of thought to how software is developed.
First off, I will say that I don't hold out much hope for Agile. Every company has its own work culture which is entrenched and not easily shifted. Some may be ripe for accepting an approach like Agile Development but most are not. If Agile is launched an AVP with a memo and training program, just sit through the seminar and wait. It will go away just like Who Moved My Cheese.
More generally, I think we spend much too much time and effort in agonizing over organization and methods instead of producing quality programs. I don't mean we need test-driven development or a million test cases in the QA group. I mean that little attention is paid to the actual code that is produced. Is it efficient? Is it secure? Is it robust? Is it easy to understand and modify? I believe W Edward Deming said that the problem with management is that it reads too many management books and not enough statistics. Same with programmers. My last organization did a Lunch-and-Learn series on best practices in coding specific situations. It was nice. Most of what I see, though, is google-the-problem/copy-a-sample-from-a-blog/compile-the-result. And that is when they aren't trying to stay on the bleeding edge.
I have rediscovered an admiration for COBOL programmers. I have to assume that the senseless dash toward "progress" seen in Java programming circles is less an issue in COBOL, which is considered a dying computer language. These guys and gals receive a request or specification and then code a solution. End of story. No need to impress anyone with the latest framework, there.
I read a comment thread recently which mentioned the "80% of anything is crap" rule with regard to programmers. Why don't we attack this statistic by attempting to make that crap a little less aromatic? What we need is a movement to improve the quality of programming by teaching theory and best practices. We all know that well written code is less prone to errors. But do we pay any attention to coding? No! We spend most of our time talking about patterns and pair-programming. Perhaps with enough theory and programming philosophy, developers might recognize that moving something from the Java source file to an XML file does not reduce the code, it merely translates it into XML. Or that using JUnit increases your code-base because your test modules are programs too that may need to be tested themselves.
I have been a programmer my entire working life but I trained as an engineer. I often wonder when will the time come when software "engineering" lives up to its name and drops the hacker culture and becomes a true discipline.