Scrum – the hard parts

Scrum has taken the software development world by storm since its inception in the 1990s. This lightweight agile framework promises to help teams deliver value in short cycles, stay responsive to change, and empower team members. Thousands of organizations have adopted Scrum and many report big benefits in productivity, quality and customer satisfaction.

However, Scrum is not a silver bullet or a simple, prescriptive methodology. Scrum is based on a different way of working and cultural values that are often at odds with traditional management practices. Many teams and organizations struggle with the difficult mindset shifts and technical practices needed to realize the full benefits of Scrum. Let‘s explore some of the hardest parts of Scrum and how to overcome them.

Accurate Estimation

In Scrum, the Development Team is responsible for estimating the effort required for backlog items and how much they can deliver in a sprint. Relative estimation techniques like planning poker and story points have become popular to help teams estimate quickly without fretting over hours and days.

However, many developers feel immense pressure during Scrum planning to come up with estimates that fit the Product Owner‘s expectations. The PO and other stakeholders often push for lower estimates and more aggressive commitments. No one wants to look slow or lazy by giving conservative estimates.

The unfortunate result is that teams chronically overcommit and end up accruing technical debt as they rush to meet unrealistic deadlines. Quality suffers, morale tanks, and the benefits of a sustainable development pace are lost.

To address this anti-pattern, the Scrum Master and PO must act as servant leaders and defend the development team‘s ability to estimate their own work. Estimates are just that – uncertain guesses about the future, not binding contracts. When a PO pushes estimates down, the Scrum Master should teach them to either reduce scope or extend timelines rather than pressuring the team. The team must feel psychological safety to give honest estimates without fear of retribution.

Over time, a Scrum team‘s estimation accuracy should improve as they gauge their velocity sprint over sprint. But perfectionism is misplaced – the point is to deliver something of value, get feedback, and adjust course as needed. Trust the team to do their best given the time and information they have.

Defining "Done"

One of the key ideas in Scrum is to deliver a potentially shippable product increment at the end of each sprint. The increment should be "Done," meaning it meets the team‘s quality criteria and Definition of Done. In an ideal world, a Scrum team could deploy their work to production every sprint if the customer and market demands it.

However, many teams fall short of this bar and struggle to truly finish features within a sprint. They may build functionality but skimp on testing, integration, documentation, or other criteria needed for something to be release-ready. A sort of "ScrumBut" pattern emerges where teams do development within a sprint but need additional hardening, testing or stabilization phases before releasing.

This is arguably worse than a staggered waterfall process since it creates a dangerous illusion that development is done earlier than it really is. Technical debt, defects and risk get pushed downstream where they are harder and more expensive to address.

Some teams try to work around this by watering down their Definition of Done to make it more achievable. But this is the wrong approach – it‘s better to deliver less but with higher quality than to do many half-baked features. The Definition of Done should reflect what it really takes to be potentially shippable.

Organizations serious about agile need to invest in building quality in throughout the development process. Techniques like test-driven development, continuous integration, pair programming, and DevOps automation make it more feasible to achieve a stringent DoD in a short sprint. Quality is everyone‘s job in Scrum, not something to tack on at the end.

Empowered Product Ownership

In Scrum, the Product Owner plays a pivotal role in guiding the team to build the right thing for the customer. The PO is responsible for creating and prioritizing the product backlog, answering questions from the team, and accepting sprint work.

This is a tall order that requires vision, business acumen, customer empathy, decisiveness and communication skills. The PO must balance many competing concerns and make hard prioritization choices. Should we do this high-risk/high-reward feature or knock out several smaller enhancements? Fix technical debt or develop something new? Pursue a strategic bet or respond to customer escalations?

Unfortunately, many Product Owners lack the organizational authority and support to truly fulfill this important role. Stakeholders frequently bypass the PO and try to directly sway the team. Sales throws in last-minute deal sweeteners. Executives dictate pet projects that trump the PO‘s carefully groomed backlog. Everything is top priority!

In these situations, the Product Owner is reduced to a glorified admin, merely rearranging the backlog to reflect the latest orders from above. The PO cannot be a true voice for customer value and sustainable product development. The team feels whipsawed and loses faith in Scrum to provide focus and predictability.

As with estimation, the Scrum Master must vigorously defend the Product Owner‘s authority over the backlog. That doesn‘t mean the PO can be a tyrant – they must actively collaborate with stakeholders to create alignment around product strategy and priorities. But when push comes to shove, the PO must feel empowered to make the call based on their expertise and customer knowledge.

Senior leadership must reinforce this by redirecting stakeholders who try to pull rank back to the PO. If everything is top priority, then nothing is. Organizations get the product leadership they deserve by how they treat their Product Owners.

Focused Scrum Events

Scrum prescribes several events for inspection and adaptation throughout a sprint – sprint planning, daily scrum, sprint review and sprint retrospective. When done well, these create a regular cadence of alignment, progress checks, demos and continuous improvement. But they also take a lot of time and discipline to do properly.

Many teams struggle to keep Scrum events short and productive. Daily scrums turn into rambling status updates where people tune out. Sprint planning drags on for hours as the team gets mired in design details. Reviews become boring readouts of completed work with little meaningful feedback. Retros raise the same old issues but nothing changes.

Agile values individuals and interactions over processes and tools, but teams often fall into the trap of going through the mechanical motions of Scrum events without the heart and soul. They miss the real purpose – to frequently come together, realign on goals, honestly assess progress, gather input, and reflect and adjust their process.

Scrum Masters can coach the team to keep events lean and meaningful. Timeboxing is key – when participants know they have limited time, they are more likely to focus on what‘s most important. The daily scrum is for quickly aligning on the next 24 hours of work and removing blockers, not general project discussions. Sprint planning should center on what can be done and how to start doing it, not hashing out every detail and "planning the sprint."

Reviews are for showing real working software, not vanity metrics, and gathering actionable feedback. Retrospectives are for identifying a couple key ways to improve as a team for next time, not endlessly rehashing the past. Continually evaluate if Scrum events are adding value or just sucking time and energy. Experiment with formats to keep things fresh.

Cross-Functional Collaboration

Scrum is built on the idea of cross-functional teams who have all the skills needed to deliver a product increment. Developers, testers, designers, etc. plan together, collaborate throughout the sprint, and share responsibility for shipping quality software. This avoids wasteful handoffs and enables faster feedback cycles.

Many organizations struggle to achieve true cross-functionality and fall back into silos and tossing work over the wall. Developers write code and then hand it off to QA to test after the fact. Testers don‘t have time to automate since they spend all sprint manually checking new features. UX gets brought in way too late to iterate on designs. Each discipline does their own thing on their own timeline.

This fragmentation is exacerbated when teams aren‘t staffed properly with the roles needed to build a vertical slice of the product. Organizations try to matrixed workers across multiple Scrum teams to utilize them efficiently. But this destroys self-organization and creates competing priorities for team members.

At the root of these issues is often an unwillingness to let go of industrial-era functional groupings and management structures. Leaders want the benefits of Scrum but aren‘t willing to redesign teams and departments around the work to be done. They pull people on and off teams to match short-term demand rather than cultivating stable products and people.

Collaboration is also stifled when teams sit apart, don‘t include non-coders in ceremonies, or allow hero culture and rockstar developers to dominate. Team members must feel jointly accountable for outcomes and comfortable working across boundaries.

Scrum is an invitation to learn, grow, and become more "T-shaped" to better serve the product and customer. A strong Scrum Master can help foster this mindset by setting an example of servanthood and facilitating interactive workshops and co-working sessions. Given support and space to grow, developers can learn to write automated tests, testers to pair on development tasks, designers to solicit ongoing customer feedback, and so on.

Self-Organization Over Command-and-Control

Perhaps the deepest and most fundamental shift in Scrum is from command-and-control management to self-organizing teams. In Scrum, the team decides how much work to take on and how to do it. Managers don‘t assign tasks, dictate technical solutions, or micromanage execution. The team owns their process and outcomes.

This is a huge change that requires managers to let go of traditional levers of control and team members to step up as true owners of their work. It‘s uncomfortable for many organizations used to predictable plans and hierarchical decision making. Managers feel anxious about deliverables and deadlines. Developers wait to be told what to do.

Self-organization requires a foundation of transparency, trust and psychological safety. When the team makes progress and problems visible, managers and stakeholders can gain confidence to give them more autonomy. Managers must learn to give guidance around strategic goals and constraints rather than diving into the weeds of execution. Teams earn trust through performance.

The Scrum Master plays a delicate role in coaching managers to be more hands-off and team members to be more self-sufficient. It‘s about creating a culture of ownership and agency, not anarchy. Certain boundaries, ground rules and support structures can actually enable more self-organization by creating clarity and safety.

Over time, high-performing Scrum teams and managers find a sweet spot where the team has goals and guardrails but freedom to operate within them. Managers spend more time on servant leadership activities like securing funding and resources, removing impediments, and developing people. They don‘t tell the team how to do their job.

Embracing Agility, Not Just "Doing Agile"

The recurring theme in all these hard parts of Scrum is that true agility requires deep and pervasive changes to mindset, culture and ways of working. Organizations can‘t just change their process and keep doing everything else the same. They must embrace the scary, exhilarating, messy implications of cross-functional teams rapidly iterating to deliver customer value.

Unfortunately, many organizations approach Scrum as a set of rituals to be performed rather than a catalyst for transformation. Teams go through the motions of Scrum events and artifacts but don‘t really live its values and principles. Work is a little more incremental and iterative but the essential dynamic is the same – stakeholders throw requirements over the wall to delivery teams.

"Doing agile" without "being agile" creates dysfunction and disillusionment. People feel the added overhead of meetings and ceremonies without reaping the benefits of faster learning and value delivery. Velocity and burndown charts create an illusion of control while quality, morale and innovation suffer under the surface.

Leaders who want the real benefits of agile must go all in on creating a culture of psychological safety, transparency, experimentation, learning and customer obsession. They must model vulnerability and openness to change. Practices like Scrum won‘t make you agile but they can create structures and habits to reinforce agile principles.

Ultimately, the hard parts of Scrum are invitations to transform your organization to be more fit for the future. True business agility comes from focusing on outcomes over outputs, collaborating across boundaries, rapidly experimenting to innovate, delivering value early and often, and harnessing the collective intelligence of everyone touched by your product. That can‘t be reduced to a methodology – it‘s a new way of being.

So don‘t despair if your organization struggles with Scrum at first. Like any new skill, agile takes time and deliberate practice to internalize. Stay focused on Scrum values even if you can‘t do all the mechanics perfectly. Continually eliminate impediments to being more agile and Scrum will start to feel more natural and powerful. The challenges are worth it to build shared consciousness and thrive in an uncertain world.