Where Scrum went wrong
Scrum might work for you and I don't dispute that. I am sure it works for many teams, I will not be claiming otherwise in this post. But I do think that Scrum has led many of us down the wrong path and I'll set out my argument in this post. Oh and note that I am talking about Scrum not Agile.
Agile started out as a set of principles. It was a reaction to the waterfall approach which clearly is not the right way to go for the vast majority of projects. It listed a set of principles that define what Agile software development is. Just like Lean Thinking has also done.
But Scrum is not a set of principles, it is a set of rules and they are pretty prescriptive. I think Scrum was needed in the beginning because people read the principles and now wanted concrete steps to follow. Scrum laid it all out for everyone to follow and was a good first step. But here is where it all started to go wrong. Instead of using Scrum as an example of how to take the principles and apply them to your organisation, Scrum became THE set of rules to follow and if you veered away from those rules suddenly you were doing ScrumBut - we do Scrum "but" we don't do rule X or Y. ScrumBut was marketed as a negative thing, something to be avoided. Keep to the script people.
I liken Scrum to a monolithic N-Layer architecture and Agile to good software development principles. Some principles that we live by in software development are weak coupling, strong cohesion, separation of concerns, dependency inversion, clean code etc. N-Layer was an implementation of some of these principles. But so too is SOA and micro-services but they are very different. I am not an N-Layer consultant and I don't think that N-Layer architecture is the right architecture regardless of what you are building. Instead, I get paid to translate core programming principles into good architectural choices based on available technology, budget and also the environment in which the software is being developed. And yes, ultimately I get paid to deliver value to the business.
Think of all the industries, technologies, software, organisational cultures, organisational structures out there. May be you're doing embedded firmware for a chip manufacturer, or you're a C++ real-time trading systems developer, or R&D, or a web developer, or an app developer, or doing AI research etc. Some of us are in start-ups, some in big organisations, some at not for profits or the government. And yet Scrum is pushed as the right project management process for all of it. Scrum is the way and when you evolve your use of it, well now you're not doing Scrum anymore, you're making a big mistake.
Somewhere along the way the goal has been lost, the original principles, while not lost are rattled out to demonstrate why Scrum is the way. Thankfully software development, while it goes through its crazes and hype cycles, tends to keep the core principles in mind and come back to them again and again.
So how did Scrum get such a strangle hold? It was inevitable really. First of all it started with the Scrum Master Certificate. People could get a certificate after a pretty short course that certified them as an agile and Scrum expert. Agile really was working, it really is better than waterfall. Suddenly Agile was something that all self-resspecting companies needed and the Agile consultant was born. These days there are a lot of them and they produce the majority of the Agile and Scrum content out there. This has led to the hijacking of Agile for profit. I find software process very interesting and I have spent a long time over the years reading about Agile and Lean. But these days trying to get good impartial advice on Agile is like doing a Google search for a cure for back pain. There is no shortage of "informative" websites that will explain about any health condition.
Back to the Agile Principles
Let's take a moment to read the current Agile principles:
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity--the art of maximizing the amount of work not done--is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Let's compare that to the rules of Scrum. Scrum has a lot of rules and a lot of nuisances, and I'm not going to list them all here, just the bare bones:
Devs/testers, product owner, Scrum master, outside stakeholders. Pigs and chickens.
All work carried out in time-boxes (sprints). The development teams makes a "commitment" to complete all the work in that time-box. At the end of the time-box the team demos only the finished work to the client.
During a sprint, the work is protected from change, but change in the next sprint is expected.
Requirements are broken down into short User Stories which are refined in Refinement meetings and finally committed to in the Planning Meeting just before kicking off the time-box.
User stories are very lightweight, more reminders of previous conversations than actual documents. However, via the acceptance criteria, you can be more rigorous and strict if you want.
Each user story clearly defines the Definition of Done so we all know when the story is complete. Great for identifying those tasks that often get forgotten about but involve a fair amount of work.
User Stories are estimated as a team based on the "What" not the "How" using story points. Story points are a relative number and exactly what they mean depends on the team. Calculating the story points of a story is based on looking at previous work to gauge the size.
Retrospective meetings get the team together to identify what went well and what didn't. There is some tension here as you can make changes to how you operate but Scrum rules are not to be touched.
The above list isn't supposed to be a definitive description of Scrum, seriously you need a book for that. So don't waste your time trying to "educate" me. Most of us know what Scrum is already. My point is that most of the above seems reasonable and I can see how the practices could help support the principles. But in no way could anyone say that these are the only rules that support the Agile principles above.
People don't get it, it's the principles not the rules! Read those principles again and tell me that they don't sound like wise advice! So why are we being strangled by the Scrum process, the one true way!? My only theory is that there are a lot of consulting firms heavily invested in Scrum and they have an interest to keep it going.
But I for one can think for myself, I've done research and used a few different Agile techniques out there over the years. Does Scrum have any good points? Sure and I'll use them if I think they are appropriate for my organisation and support the principles of Agile and Lean. But just like I'm not going to force every client and every solution into an N-Layer monolith, I'm not going to force anyone into the one true Agile implementation of Scrum. Having said that though, sometimes you don't get a say in using Scrum or not, that is what you get and that's it.
You don't need Scrum, you need Agility
Learn the Agile principles, learn the different Agile techniques that support those principles, apply techniques in support of those principles in a way that works for your team and your organisation. If you get in a consultant, make clear you want an Agile consultant, not just a one trick pony.
If you're looking for agility don't think that a software development process is the whole answer, it isn't. You want agility? How about Continuous Deployment (CD)? How about infrastructure as code? How about cloud native applications that can be deployed and scaled at will? How about evolvable architecture? These create agility! You could apply all of those in addition with a super simple iterative development process and you'd be far more Agile than doing 100% Scrum with a legacy architecture with special snowflake servers and applications that have large manual deployment scripts.
Do yourself a favour, don't serve Scrum, choose a process that serves you and your goals. And don't forget that there is way more to agility than a software development process.