2013/06/04 RallyON Conference Day 2

by Gene Kim on

#rallyon13

(I'll be presenting on "Why Everyone Needs DevOps Now" on Tuesday 10:45am, Boulder Library; Scaling Execution Track)

Up: "State Of The ART": Drew Jemilo, Rachel Weston Rowell

  • How do we integrate Agile into PMO, release mgmt, product mgmt, quality, integration, architecture and design, governance, waterfall team
  • Agile adoption is like Innovator Dilemma: started with small innovative team, now at high quality use
  • Weston: "Agile at scale are the robust practices to support demanding environments like Twitter dev"
  • Jemilo: "Teams are high performing when they're empowered, and know what needs to be implemented to achieve the common goal"
  • Jemilo: "Deliver fully tested, system level solutions every 8-12 weeks"
  • @PrettyAgile: We stood on the shoulders of Scrum and Kanban - @DrewJemilo talking about #SAFe #RallyON13
  • @RallyOn: RT @stephen_younge: @DrewJemilo on SAFe: "Until a concept is proven 3 times, it can't go into the SAFe framework."
  • @RallyOn: RT @PaulBratcher: Listening to Rachel and Drew discussing SAFE and the future of agile. #rallyon13 http://t.co/h1DPoWMUI2
  • Jemilo: "Leffingwell created Agile Release Train to solve problem of integrating flows of multiple project teams; 5-12 teams (50-100 individuals) that plans and commits and executes together
  • Jemilo: "Common cadence and normalized story point estimating
  • @cdric: @DrewJemilo on SAFe: "What kill you the most in Agile are dependencies." #rallyon13 #SAFe #ScalingAgile
  • Haha. Nice. @DrewJemilo: "You can't scale crappy code"
  • @DrewJemilo quoting Reinerston: "More value is created by creating overall alignment than local excellence."
  • .@DrewJemilo: "In agile at enterprise scale, product teams must demo of fully integrated code every 2 weeks. Kind of scary"

  • Great talk by @DrewJemilo and Rachel Weston! My notes for Day 2 posted at http://scribes.tweetscriber.com/RealGeneKim/144

  • I'm giving my talk on DevOps at 945 in Boulder Library. See you there!

Gene Kim, Why Everyone Needs DevOps Now

  • @jeantabaka: #rallyon13 @jeantabaka: sooo freakin' cool @RealGeneKim speaking here about #lean #devops
  • @CamTurner: RT @erwilleke: "Thank you for your puppy, it's very cute" @RealGeneKim on sending code back to Dev until it's ready #rallyon13 http://t.co/2ZwIc5e4GJ
  • @PavelDabrytski: RT @agilelynn: Lots of laughs! "DevOps could be called .*Ops" @RealGeneKim #RallyON13
  • @RallySoftware: RT @PrettyAgile: @RealGeneKim Thanks for the book and an Excellent talk #RallyON13 http://t.co/C5IEZd8iGy
  • @agilelynn: Demonstrate user value on DevOps User Stories via cycle times and emergency response. @RealGeneKim #RallyON13
  • @PrettyAgile: @RealGeneKim Thanks for the book and an Excellent talk #RallyON13 http://t.co/C5IEZd8iGy
  • @servemecom: RT @agilelynn: "Break things before production." @RealGeneKim #RallyON13
  • @jeantabaka: “@RallyOn: . @RealGeneKim thanks for the paper copy of Phoenix project - now I can share it #RallyON13” ENVY!
  • @RallyOn: . @RealGeneKim thanks for the paper copy of Phoenix project - now I can share it #RallyON13
  • @agilelynn: !! Injecting $2.5T into the economy would dramatically change the world our children live in ($ value of IT fails) @RealGeneKim #RallyON13
  • @benbuxton: RT @stephen_younge: @DReinertsen quoted at #rallyon13 - "Centralize decisions if they are global and infrequent. Decentralize all other decisions." @RonicaRoth
  • @erwilleke: RT @RallyOn: . @RealGeneKim #RallyON13 allocate 20% to non-functional to pay down debt continuously or spend 100% time doing it http://t.co/CCEOPCHuYH
  • @agilelynn: The longer teams are doing DevOps, the better they do (puppet labs survey). @RealGeneKim #RallyON13
  • @RallySoftware: RT @AgileRanger: Love this! Shared pain drives shared goals between Dev and Ops!
  • #RallyON13 #devops http://t.co/p1DUsgmZqn
  • @benbuxton: Confront. A bold word that means "to face". @ChristopherAver #RallyON13
  • @PrettyAgile: You need a culture that keeps pushing into the danger zone and the habits that enable you to survive @RealGeneKim #devops #RallyON13
  • @RallyOn: . @RealGeneKim #RallyON13 third way - most important - culture of continuous experimentation and learning - keep pushing into danger zone
  • @agilelynn: Integrate Ops User Stories into agile projects. * @RealGeneKim #RallyON13
  • @benbuxton: Responsibility is the place where we refuse to feel trapped. @ChristopherAver #RallyON13
  • @RallyOn: . @RealGeneKim #RallyON13 the google checklist for hand-off reviews - if you #fail you get your puppy handed back http://t.co/V2h6P0K2Yj
  • @RallyOn: . @RealGeneKim thanks for the paper copy of Phoenix project - now I can share it #RallyON13
  • @agilelynn: !! Injecting $2.5T into the economy would dramatically change the world our children live in ($ value of IT fails) @RealGeneKim #RallyON13
  • @cmar55: "on the freeway, it's hard to know how fast you're going until you pass someone" - ted bardasz #rallyon13 #agile
  • @erwilleke: RT @RallyOn: . @RealGeneKim #RallyON13 allocate 20% to non-functional to pay down debt continuously or spend 100% time doing it http://t.co/CCEOPCHuYH
  • @tintegrity: RT @erwilleke: "Thank you for your puppy, it's very cute" @RealGeneKim on sending code back to Dev until it's ready #rallyon13 http://t.co/2ZwIc5e4GJ
  • @agilelynn: The longer teams are doing DevOps, the better they do (puppet labs survey). @RealGeneKim #RallyON13
  • @RallyOn: . @RealGeneKim #RallyON13 allocate 20% to non-functional to pay down debt continuously or spend 100% time doing it http://t.co/CCEOPCHuYH
  • @RallySoftware: RT @agilelynn: "Break things before production." @RealGeneKim #RallyON13
  • @cdric: "People don't buy WHAT you do, they buy WHY you are doing it!"
  • @PrettyAgile: You need a culture that keeps pushing into the danger zone and the habits that enable you to survive @RealGeneKim #devops #RallyON13
  • @RallyOn: . @RealGeneKim #RallyON13 third way - most important - culture of continuous experimentation and learning - keep pushing into danger zone
  • @agilelynn: Integrate Ops User Stories into agile projects. @RealGeneKim #RallyON13
  • @PrettyAgile: "The goal is to have shared pain" @RealGeneKim suggests waking Devs up at 2am will improve defect resolution cycle time #devops #RallyON13
  • @mark_richdata: "if we have shared pain, we're much more likely to have shared goals" @RealGeneKim #RallyOn13
  • @RallyOn: .@RealGeneKim #RallyON13 Andon cord gets pulled up to 3300 times per day to swarm issues in some and create technical debt downstream
  • @BMC_DevOps: RT @RallyOn: #rallyon13 @RealGeneKim how to achieve high performance in #devops in concert with Jez humbolt http://t.co/p6nYvWF1Za
  • *
  • @RallyOn: . @RealGeneKim 2nd wave - amplify the feedback or close the feedback loop - best example is a Toyota Andon cord #RallyON13
  • @RallyOn: . @RealGeneKim #RallyON13 was ops pick through poop for peanuts to package #fail
  • @agilelynn: "With shared pain you can have shared goals." Developers supporting their own code can help. @RealGeneKim #RallyON13
  • @stephen_younge: @RonicaRoth - "If everybody on the team doesn't understand the vision and strategy, then they can't help you achieve it!"
  • @RallyOn: .@RealGeneKim #RallyON13 Andon cord gets pulled up to 3300 times per day to swarm issues in some and create technical debt downstream
  • @agilelynn: "Toyota plant stops production 3300 times a day - it's better to swarm on all issues than to let them get through." @RealGeneKim #RallyON13
  • @RallyOn: . @RealGeneKim #RallyON13 was ops pick through poop for peanuts to package #fail
  • @RallySoftware: RT @RallyOn: #rallyon13 @RealGeneKim how to achieve high performance in #devops in concert with Jez humbolt http://t.co/p6nYvWF1Za
  • @agilelynn: "Show that at the end of every sprint we have shipable code in the environment it runs in." @RealGeneKim #RallyON13
  • @RallyOn: RT @agilelynn: "Unplanned work (system down) is like the dark matter for DevOps" @RealGeneKim #RallyON13
  • @RallyOn: RT @cdric: "How long does the average company on the Standard & Poor's list survive? 1958: 60 years, 2012: 18 years!"
  • @agilelynn: "How long does it take to get a test env, 46 weeks? That's when we start to horde test envs like Klinger on MASH" @RealGeneKim #RallyON13
  • @PrettyAgile: High performing organisations deploy code 30 times more frequently and 8,000 faster @RealGeneKim #DevOps #RallyON13
  • @agilelynn: "Toyota plant stops production 3300 times a day - it's better to swarm on all issues than to let them get through." @RealGeneKim #RallyON13
  • @RallyOn: . @RealGeneKim 2nd wave - amplify the feedback or close the feedback loop - best example is a Toyota Andon cord #RallyON13
  • @RallyOn: . @RealGeneKim #RallyON13 was ops pick through poop for peanuts to package #fail
  • @agilelynn: "Show that at the end of every sprint we have shipable code in the environment it runs in." @RealGeneKim #RallyON13
  • @RallyOn: RT @agilelynn: "Unplanned work (system down) is like the dark matter for DevOps" @RealGeneKim #RallyON13
  • @agilelynn: "How long does it take to get a test env, 46 weeks? That's when we start to horde test envs like Klinger on MASH" @RealGeneKim #RallyON13
  • @cdric: "How long does the average company on the Standard & Poor's list survive? 1958: 60 years, 2012: 18 years!"
  • @benbuxton: RT @PrettyAgile: High performing organisations deploy code 30 times more frequently and 8,000 faster @RealGeneKim #DevOps #RallyON13
  • @benbuxton: Responsibility is about how we respond when things go wrong @ChristopherAver #RallyON13 http://t.co/mgzuzO5FZG
  • @agilelynn: There is a difference between deployment and release events. @RealGeneKim #RallyON13
  • @PrettyAgile: Amazon deploys every 11.6seconds! @RealGeneKim #DevOps #RallyON13
  • @benbuxton: What has more mindshare at your company?...the problems of silos or the org's shared responsibility? @ChristopherAver #RallyON13
  • @mark_richdata: When we break promises we tend to compensate by making even more audacious promises #RallyOn13
  • @agilelynn: "This is really bad tech debt" @RealGeneKim #RallyON13 http://t.co/HioHcEHIL3
  • * @AgileRanger: "Orgs that adopted DevOps practices 12+ months ago 5x more likely to be high performing.." #RallyON13 #realgenekim http://t.co/AzdBtuQ7x8
  • RT @AgileRanger: "Orgs doing DevOps for 12+ months 5x more likely to be high performing" #RallyON13 http://t.co/AzdBtuQ7x8

  • @jeantabaka: #rallyon13 read "The Phoenix Project" NOW! It will change your view of #devops WIP and pull forever. Tell me if u r reading it.

  • @agilelynn: Tee hee! How Telstra started their scaled agile approach: http://t.co/q9e8t4wWNh @PrettyAgile #RallyON13

  • @agilelynn: Telstra has Epic Owners (exec), Feature Owners (business), and Product Owners (tech leads) @PrettyAgile #RallyON13

Organizational Change for Beginners: Hacking the Human System, Bob Gower

  • @LeadershipGift: RT @benbuxton: We can cope or we can grow. It's your choice @ChristopherAver #RallyON13
  • @andreanolanusse: Let's start the change if you wanna your company agile some day #rallyon13 http://t.co/bmluC4e3gY
  • @ChristopherAver: RT @benbuxton: Confront. A bold word that means "to face". @ChristopherAver #RallyON13
  • @benbuxton: Cognitive dissidence is a necessary component of change...we have this but we want that. @bobgower #RallyON13
  • @CoachAtPlay: The "Turn" in your change effort is created by a vision of how it could be. #RallyON13 @bobgower
  • @LeadershipGift: RT @benbuxton: Responsibility is the place where we refuse to feel trapped. @ChristopherAver #RallyON13
  • @LeadershipGift: RT @benbuxton: You are not born with perfect knowledge, nor will you die with it...@ChristopherAver #RallyON13 @RallySoftware
  • @CoachAtPlay: When "we" decide we won't succeed anyway. #rallyon13 @bobgower http://t.co/DXLUUbM8pJ
  • @benbuxton: ...and before that...I was a human being @bobgower #RallyON13 @RallySoftware
    • Gower citing AA Steps. Love it. "Admit we are powerless; We have hit bottom" (and seek to right those we have wronged)
    • .@bobgower: "AA creates support structure: holds mtgs on Fri/Sat night, when problematic behaviors most likely
  • @CoachAtPlay: Can you identify the positive aspects of the thing you think should change? How is it bringing value to the business?#RallyON13 @bobgower
  • @agilelynn: "The more you invest in communication, the more you impact the culture" @PrettyAgile #RallyON13
  • @CoachAtPlay: "Let's commit to a higher class of problems in our lives!!" #rallyon13 @bobgower

Agile Mythbusting

  • @stephen_younge: @LMaccherone Mythbusters insight: Scrum Teams who don't task out outperform those that do - but sizing Stories is still vital.
  • @RallySoftware: RT @cdric: Agile Mythbusters - Fact #1: Lightweight Scrum perform better than any other types of Scrum!
  • @RallyOn: #rallyon13 @LMaccherone and the data says: lightweight scrum performs best on SPDI #agile http://t.co/hqT6PYDCWF
  • @jeantabaka: #rallyon13 metrics reveal that iterations w/story points only perform better than other #Scrum approaches with task-level estimates
  • @RallySoftware: RT @jeantabaka: Time in Process - a metric u can find in @RallySoftware data if you use the tool effectively, also "say/do" #rallyon13 @LMaccherone
  • @PrettyAgile: RT @jeantabaka: #rallyon13 @LMaccherone "Start extracting data to prove to your business that #agile & #lean really are better."
  • @jeantabaka: Time in Process - a metric u can find in @RallySoftware data if you use the tool effectively, also "say/do" #rallyon13 @LMaccherone
  • @RallyOn: #rallyon13 @LMaccherone details on the four calculated SDP index items - see attached http://t.co/D9XD8rODS7
  • @RallyOn: #rallyon13 @LMaccherone details on the four calculated SDP index items - see attached http://t.co/D9XD8rODS7
  • jj
  • @RallySoftware: RT @nataliewarnert: #RallyON13 correlation does not equal causation http://t.co/bKZxZxhxD2
  • @jeantabaka: RT @RallyOn: #rallyon13 @LMaccherone SDPI - 4-7 dimensions - responsiveness, quality, predictability, productivity - from anonymous @RallySoftware data
  • @stephen_younge: RT @RallySoftware: #RallyON13 Live! - "Agile Mythbuster - Metrics & Insights" w/ @LMaccherone, 12:45PM MT Reg: http://t.co/PkxtNqlO9s
  • @jeantabaka: #rallyon13 "Dedicate to ONE team" most predictively valuable metric for better responsiveness/productivity/predictability/quality
  • @cdric: Agile MythBusters - Fact #2: 95% of the work done by dedicated teams is better than when members are working on multiple teams.
  • Finding: confirmation of Little's Law: "as WIP goes up, Time in Process goes up."
  • Analysis of finding: "teams that have discipline to limit their WIP have better performance"
  • Interesting! These measures are highly correlated: WIP, Time in Process, Quality, Responsiveness
  • Great talk by @LMaccherone: finding causal factors for quality, time in process, responsiveness, etc, from real customer data

Continuous Delivery at Rally: It's Easy! Just change all your engineering practices (Well, maybe it isn’t that easy); By Steve Neely, Steve Stolt

  • Stolt: "Continuous delivery: ability to release features on demand; we know engr discipline reqd to support this is good"
  • Stolt: "worst case of continuous delivery is still good: more feedback, faster, fail more safely, more feedback, etc.
  • Neely: "Every checkin takes you into production; developers are smarter"
  • Neely: "why? smaller batches (50, 100 lines) makes easier to debug; 2 months w/thousands of lines of code, multiple teams;
  • Neely: "we used to ship code so infrequently, we could never remember all the steps to go into production, w/bad conseq
  • Haha! Neely: "Why did we as Dev want continuous delivery? It would be fun, & all the cool companies were doing it"
  • Neely: "We [Rally Dev] read about Etsy deployment practices on their blog; that sounds like fun" /cc @allspaw @mrembetsy
  • Stolt: "[Prod Mgmt] wanted value to customers faster/ASAP, and less stress about missing a release; 3rd feature that slipped to next 8 weeks"
  • Stolt: "We also had intellectual curiosity about kanban and Lean; value stream map, Reinertsten"
  • Stolt: "Why continuous deploy? We wanted to outflank competition, deliver as fast as we write code, image of excellence
  • Stolt: "Before: less than 300 employees; in business for 8 yrs; 8 week release cycle, which was too long
  • Neely: "Horrible old world: painful & lengthy code merges; preparing tar balls, complex migrations & db chgs, stop/start svrs
  • Neely: "..and if your feature missed the release, we'd have to wait 8 weeks until the next release"
  • Neely: "We had Scary Saturday mornings at 5am when we released code; everyone watching charts; lose Sat when it goes wrong
  • Neely: "Copy this, tar that, ask DBAs to that, call me when you're done, what went wrong
  • Neely: "Monday Morning Hangover: ...and we'd have to come in early on Monday, to see how site handled higher load
  • Stolt: "It's horrible when you're at work on Sat morning, & you're 6 hrs in, & u'r supposed to be at soccer game w/kids
  • Stolt: "Also caused Scope Cut; features cut down to bone and further just to get feature shipped"
  • Neely: "First step: fully understanding our process: from pulling a story to deployment; get build, test, stop server, etc. No one could agree what was the way"
  • Neely: "Once we documented our entire deploy process (which no one agreed on), then find longest step, usually manual"
  • Haha. Neely: "We did thought experiment: rolled die: odd = deploy now; even = go back to work; what would prevent a deploy?"
  • Neely: "Reasons: manual test phase; manual deploy steps; cold db migration; unknown performance implication
  • Neely: "[Rally] deploy pipeline runs on Jenkins: 5 steps: unit/int test, deploy to Test, run more tests, Engr use, Prod use
  • Neely: "Feature toggles; dark deployments; canary deployments
  • Neely: "Suddenly we noticed that team flow chgs when cycle goes from 8 wks to daily; lots more adhoc mtgs; consume backlog, plan, add enough stories for the next week; heartbeat does emerge; about weekly now
  • Stolt: "[PM] canceled mtgs that we've held for yrs: it felt great to delete our iteration & planning mtgs"
  • Stolt: "We also burned all our roadmaps: all those annual/quarterly were gone: they were never accurate anyway"
  • Neely: "As Dev, we didn't look at production monitoring charts just on Sat morning; we watch them all the time"
  • Neely: "In order to deploy faster, you have to know what's going on in Production: what changed, why"
  • Neely: "We have posters of unicorns/rainbows on our walls: Ops & Dev working together, everyone happy & has work/life balance
  • Neely: "Easier/faster to onboard new engrs: safe to fail environment; good test coverage enables new people to hv less fear=
  • Neely: "Surprise: continuous deployment scared QA; but tester expertise is essential; tester minds are weird & wonderful
  • Neely: "Required new phase: early test planning; enumerates automated tests that need to be built to complete story
  • Neely: "Speed is hard: used to take 9 hrs to run GUI test; which meant fastest deploy was 9 hrs! Identify/optimize those tests
  • Neely: "We constantly chart the slowest test, and figure out how to speed them up, parallelize them as much as poss
  • (Yes!) Neely: "Lesson: design every test to be parallelized; very painful to retrofit tests after the fact"
  • Stolt: "Stakeholders freaked out when we no longer gave feature ship dates: 'when can I get feature X?'"
  • Stolt: "Countermeasure: set clear expectations and be transparent by showing backlog, kanban board"
  • Stolt: "We now publish 3 month and 6 month roadmap & then vaguer list of 'this is what we're thinking'"
  • @stephen_younge: Packed session on Continuous Delivery and Continuous Flow by our own @stevestolt and @stenvoon at the Boulder Library! #RallyON13
  • .@stevestolt: "Enablement: internal: push responsiblity out; external: mtgs > emails; webinars > blogs; badge for new features
  • .@stevestolt: "Feature toggles are awesome: gives you control, allows staged rollout, easy rollback, easy market release
  • .@stevestolt: "All features are fully QA certified before feature is toggled on: been thru full testing, etc."
  • .@stenvoon: "All our features are toggle enabled; they're that awesome"
  • .@stevestolt: "Have the mission clarified
  • .@stenvoon: "Our call to action: do continuous delivery: you will end up in a better place than where you are now"
  • .@stenvoon: "Where to start: get automated testing going, document deploys

Agile Business Book Authors Panel: A Leader's Guide to Harnessing Complexity

  • Speakers: Jean Tabaka, Bob Gower, Ryan Martens, Zach Nies, Brent Barton, Ronica Roth

  • Zach Niles: "What is agile business? Enable biz to change as quickly as outside world, competitive environment, etc."

  • @agilelynn: Author panel for new Agile Business book! http://t.co/wr4diFxqWu #RallyON13

  • @benbuxton: Agile Business authors panel...great stuff! #rallyon13 @RallySoftware http://t.co/JEGixV2swI

  • (Nice) Niles: "Also, Agile Biz allows early founders & explorers to continue to do valuable, meaningful work, vs raw execution

  • Jean Tabaka: @jeantabaka: wrote section "People, Not Resources"

  • Ronica Roth commenting on energy levels as orgs adopt Agile practices; more collaboration, winning, etc.

  • @benbuxton: How do you build the right thing? Oftentimes the bottlenecks are upstream @zachnies #RallyON13 @RallySoftware

  • .@RallyOn: "I think Agile Businesses spend a lot of time thinking, testing theories, learning"

  • @agilelynn: "Planning to get lucky - setting yourself up to take advantage of crises." @RallyOn #RallyON13

  • @agilelynn: "Interesting paradigm where I'm pointing out all the mistakes I've made in order to show the value..." Brent Barton #RallyON13

  • @benbuxton: Ryan always says...will you get on with building something crappy so you can learn? @RallyOn @RonicaRoth #RallyON13 @RallySoftware

  • .@RallyOn: "Tipping points: when people mv b/c walls are in the way; when PMs abandon roadmaps; when engrs own testing

  • .@jeantabaka: "Only 2 metrics matter: customer happiness and employee happiness/engagement

  • @agilelynn: @RonicaRoth: Agile tipping point - when multiple stakeholders negotiate with each other to prioritize needs. #RallyON13

  • @agilelynn: Agile tipping points: moving furniture; others taking over; eng. own test infrastructure; can't find facilitators fast enough. #RallyON13

  • @agilelynn: @jeantabaka (quoting Jeff Southerland): metrics that matter most = employee happiness and customer happiness. #RallyON13

Misc

  • @agilelynn: Yes, she is! "Rachel Weston Rowell is one of the best coaches in the world!" #RallyOn13
  • @jeantabaka: #RallyON13 State of the ART keynote on scaling agile using #SAFe #agile @rallysoftware Excited! http://t.co/EW2vE3G500
  • @agilelynn: RT @stenvoon: Find out how easy it is to implement continuous delivery. Engineering meets Business, in the library at 3pm today #RallyON13
  • RT @cdric: @DrewJemilo on SAFe: "What kill you the most in Agile are dependencies." #rallyon13 #SAFe #ScalingAgile
  • Haha... RT @benbuxton: ...and before that...I was a human being @bobgower #RallyON13 @RallySoftware
  • RT @RallySoftware: RT @cdric: Agile Mythbusters - Fact #1: Lightweight Scrum perform better than any other types of Scrum!
  • RT @jeantabaka: #rallyon13 "Dedicated team (vs mult proj)" best predictor of better responsiveness/productivity/predictability/quality
  • RT @cdric: Finding: 95% of the work done by dedicated teams is better than when members are working on multiple teams.
  • Yes, great talk! RT @stephen_younge: Packed session on Continuous Delivery by our own @stevestolt/@stenvoon! #RallyON13 Neat. RT @agilelynn: "Planning to get lucky - setting yourself up to take advantage of crises." @RallyOn #RallyON13