2013/06/02 RallyON Conference

by Gene Kim on

#rallyon13

(I'll be presenting on "Why Everyone Needs DevOps Now" on Tuesday 10:45am, Boulder Library; Scaling Execution Track)

Up: Tim Miller, CEO, Rally Software

  • Miller: "We are all gathered here to solve some of the largest problems in the space"
  • Miller, CEO, Rally Software; 130 companies gathered here; industry and financial analysts here for first time;
  • @cdric: RT @RallyOn: Welcome to the tent @RallySoftware! Only way to fit 400 in #boulder for conference in downtown #rallyon13 http://t.co/QTAvqtKy3P
  • @BryceDay: Tim Miller - Rally CEO #RallyON13 http://t.co/lwiO7KOwWR
  • @benbuxton: Listen, Encourage, Challenge, and Teach. - Tim Miller #RallyON13
  • @BryceDay: Tim Miller - Rally CEO #RallyON13 http://t.co/lwiO7KOwWR

Up: Ryan Martens, CTO, Rally Software

  • "We have our entire engineering team participating in Hackathons, especially around customer events; it's a great way to time-bound efforts and get great results"
  • Showing video of 48 hour hackathon; "we spent entire weekend coding"; showing 3 finalists based on demos
  • @jeantabaka: #RallyON13 I love @rallyon passion about our customer hackathon event here and Nat'l Day of Civic Hacking http://t.co/EW2vE3G500
  • @RallySoftware: Watching and voting on #hackathon #RallyApps at #RallyON13 http://t.co/RyJoiXb7By
  • Finalist 1: Cambia Healthcare (Luke Mueller, Conner Reeves0
    • Goal: show cumulative flow across project portfolio
  • Finalist 2: National Instruments (Ben Ridge, Matt Cotton)
    • Goal: make sprint planning easier: by combining feature, defects that are in tree view, and enabling drop/drop into sprint columns
  • Finalist 3: John Deere (Brand Fryslie, David Wire)

    • Goal: show risk to cycle time due to large batches/features
    • Showing cycle time of story point sizes, trimming by sizes
  • Rally Software: Social mission: Citizen Engineer: #hackforchange: 95 events, 20 govt partners, 75 data sets, 10K+ expected participants

  • National Day Of Civic Hacking: 6/1-6/2: a national event; sponsors: Rally, Intel, Code for Good, Edelman, Socrata, Elance

Up: Seven Deadly Sins of Agile Measurement; Larry Maccherone (St. Julien Tent)

  • Contact info: @LMaccherone, LMaccherone@rallydev.com
  • Started factory floor process control software company during college
  • Did time at CMU at intersection between quality & software with Watts Humphrey; did talk on Agile Metrics in 2009 (got 3 job offers from the talk)
  • Rally Software wasn't one of those companies, but sought them out, because they had actual data from real customers
  • Contrasts
    • Process: formal & heavy vs. light & often
    • Product & Design Feedback: varies vs. early/often
    • Plan: heavy, upfront, longer iterations vs. light throughput
    • Dominant Value: Control for the Org vs. Collaboration within Team
  • All of agile techniques and tools are just ways to specifiy Feeedback & Values for some Context
  • Why measure?
    • Feedback :)
    • Diagnostic :)
    • Forecasting :)
    • Lever :( (by use by management to modify behaviors)
  • Sin 1:
    • Using metrics for Feedback to improve Own Performance; try to steer clear to change somone else's metric
    • .@LMaccherone: "Use metrics for feedback to improve your own performance; not to change someone else's behavior"
    • (Showing cycle time scatter chart: all good until red line overlaid; "mgmt oversight causes wrong behaviors")
    • (Better reporting: Time In Process chart: calendar day vs. cycle time: no red line)
  • Sin 2: Unbalanced Metrics
    • 4 quadrants: Do it fast vs. do it right vs. do it on time vs. keep doing it
    • Need attributes from all four
  • Sin 3: Believing metrics can replacing thinking (of people closest to the work)
    • "Too often we seek knowledge not wisdom; as a result, we are becoming a nation of technological giants and ethical infants" -- Vernon Davis and Omar Brandly
  • Sin 4: Too costly measures
    • 15 min timecarding was supposed to only be a 1-3% burden of time; felt much higher ("was guilty of doing this with Watts Humphrey)
    • "imagine how much data you can get from passively looking at source code repos" -- "I propose a Net Promoter Score can get you 80% of data you need; 1 question + comment field" (vs. "horrible 12 question surveys we always get)
  • Sin 5: Using convenient metrics (but drive wrong behavior"
    • "When NOT to take a shot"
    • Monta Ellis: 9th highest scorer (8th last season)
    • Carmelo Anthony (Melo): 8th highest scorer (3rd last season)
    • .@LMaccherone: "In NBA teams, some teams w/high scorers win more when they're out sick; highest scorers often steal shots"
    • .@LMaccherone: "Lesson: don't use easy metrics, b/c they can drive wrong behavior (top NBA scorers)"
    • ODIM: Outcome, Decision, Insight, Measurement (Think L->R; Effect R-> L)
    • .@LMaccherone: "I think most failures not due to bad strategy; instead, it's bad execution, wrong decisions made on daily basis"
  • Sin 6: Bad analysis
    • Misuse of control charts: pick outliers, then chose SLA minus outliers: "We'll eat the penalties for the outliers"
    • "heavy tail" is very different than standard distribution: Risk management: could be 3-10x off using Normal, 2.5-5x off using Poisson, 7-20x off is you use Shewhart
    • Note cycle time distribution graph on right: doesn't look at all like normal distribution!
      https://pbs.twimg.com/media/BL3OSKQCcAAugCN.jpg
  • Sin 7: Forecasting w/o discussing probability
    • Stmt to management: "we'll be done in 7 sprints"; likely they'll hit it, but will hide risks to mgmt
    • percentile coverage, cone of uncertainty, and Monte Carlo simulation (made popular by Nate Silver, 538 election work)
    • Randomly introduce delays based on historical record: generate probability of hitting the date
    • New technique: planned backlog, perf issues + vendor delay
  • Watts Humphries biggest contribution was integrating into psychologist into software development; advocated use of third party coach ("I work for the team, not the manager")
  • Q: "how to sell common agile metrics who care about date, time and cost"; A: "I'll point to the coach, as sometimes managers don't want to listen to you"
  • Exercise: 100 boxes in warehouse: mix of boxes from 1/99 to 99/1 red to green; if you pull out one ball and it's green, what is guess of whether more red; 75% of chance being right if you guess Green

Up: Tim Miller, CEO, Rally Software: "Culture Of Responsible Product Leadership"

  • Talking about transitioning from "command and control" leader to "servant leader"
  • Goeffry Moore: Horizon 1, 2 and 3
    • 1: making money
    • 2: where do we need to go
    • 3: places to invest for 5+ year timeframe
    • We separate Horizon 3, b/c otherwise it would wreak havoc on everything
  • We have to take execs on offsite: 4 day trip to Mexico to create team, understand shared goals, etc.
  • "stop yourself from blaming other people; good job, congats on taking risks, we learned a lot; to do otherwise will kill learning organizations" "instinct is highest when the stakes are high"
  • "why did we wear suits to NYSE IPO day?" "respect for the institution"
  • "organizational design research showed that #1 complaint from workers is not enough information shared"
  • "biggest mistake?" "we pivot so much, all the mistakes are indistinguishable! Everything is a mini-experiment, no big releases or big gamble"
    • Miller on biggest failure: "We pivot so much, all our mistakes are indistinguishable! Everything is a mini-experiment"
    • "many years ago, we had to achieve a goal or bad things would happen; for 24 hours, we needed financing or partnerships, and that was stressful; but we had smart people"
  • "compensation and sharing" "sent out everyone's stock option info by mistake; one quit b/c not as high above everyone else as he thought"
  • Miller: "In next company, I'd remove vacation tracking policy right away -- you need high trust"
  • 'CEOs have too much stress; they need to know that a high functioning team w/high trust will always outperform"
    • "Culture: It's completely in your control, it will help you outperform your peers"
    • Miller: "Investing in culture is best investment: it's completely in your control, and you'll outperform your peers"
    • "Small teams can create culture of their own, despite CEO"
  • Miller on hiring: "We interview 200 interview candidates to hire 1 engineer. Quickly ran out of candidates here in Boulder"
  • On retention: "I think if someone isn't happy in companh, they should leave and encourage it" "Just create great place to work where people want to come every day"
  • Tim Miller, CEO, Rally: "I believe an onshore Agile team will outperform a non-Agile team by 3:1"
  • Miller: "I'm a big believer in recognition: small prizes, not spot bonuses or compensation"
  • Miller: "not sure public company is harder than private! before, any 3 of 5 investors could take me out! now there's a ton of process."
  • Miller: "Even as public company, we still don't provide product ship dates until entire team has committed to it"
  • Up: Keynote

  • .@RallyOn: "At USWest, we built screen scraping apps for mainframes; 6 week cycles"

  • .@RallyOn: "At BEA, we shipped software on CDs; never want to do that again; BEA couldn't handle 6 wk release cycle"

  • .@RallyOn: "WebLogix 7 product had horrible 8 month release cycles; like anti-thesis of Agile"

  • (Listening to Ryan Martens, CTO, Rally Software; founder Rally Software)

  • .@RallyOn: "To steer towards Agile is to steer away from mediocrity and amateurism, and steer towards mastery"

  • .@RallyOn: "Our Agile phases: batch -> flow -> pull -> innovate" "single team -> multiple team -> multiple programs"

  • .@RallyOn: "Initial goal: F4: Factor of 4 better: Agile and Lean"

  • .@RallyOn: "For 4 years, our goal was to master Iteration and Release; constant 1-4 week cycle"; 8 yr history!

  • .@RallyOn: "In 2010, we shipped an 8 week batch, with disastrous consequences; made decision to 1 wk batch"

  • .@RallyOn: "Now most of our services are doing continuous deployments; every commit goes to production" (awesome story)

  • (This is an amazing talk by @RallyOn, describing the 8+ yr journey of Rally Software software dev and deploy processes)

  • .@RallyOn: "Now we're on kanban to manage flow, constantly learning... we still get the allocations [wildly] wrong"

  • .@RallyOn: "How to create greatness now?" "Step 1: A/B Testing in free Community edition, constantly testing features"

  • .@RallyOn: "Step 2: Allow organic feature adoption in Enterprise & Unlimited Editions via apps" (like HackDay outputs)

  • .@RallyOn: (answering question of "if you're so nimble, why aren't we seeing a ton of new features in our instances?")

  • .@RallyOn: "Step 4: Build up these apps to fully meet the needs of enterprises" (to prevent drastic user experience chgs)

  • .@RallyOn: "Step 5: Roll out big improvements to all of our users"

  • .@RallyOn: "20% of features always/often used; 60% of features never used"

  • .@RallyOn: "Must be experimenting always to achieve Mastery"

  • .@RallyOn: "In 2004, we did Alpha, Beta, Ship; Resulted in "oh crap" moment; too many guesses, untested"

  • .@RallyOn: "Agile = Build it Right; Agile = Mindset of small Change"

  • .@RallyOn: "Agile requires the humility to understand Customer Development, validating customer needs"

  • .@RallyOn: "As a civil engineer, I was trained in my 4 years by doing over 4000 problems"

  • .@RallyOn: "First Big Guess: we understand the Customer Problem"

  • .@RallyOn: "Dave McClure: customers don't care about your solutions; they care about their problems"

  • .@RallyOn giving h/t to Blank, Reinersten; Reis

  • .@RallyOn: "A huge hurdle we overcame: creating sense of safety to do experiments and execute countermeasures"

  • .@RallyOn: "It gets dicey when experiments tell you need to delete half of your backlog, b/c you were wrong"

  • .@RallyOn: "My passion is now mastering Strategy and Execution (thru Experimentation)" (Wow. It shows)

  • .@RallyOn: "Thru Google Chat, Yammer, etc., our exec team went from weekly interactions to constant communication"

Up: Otto Hilska, Flowdock (recently acquired by Rally)

  • Hilska: "teams use Facebook, IRC chats, project mgmt, version mgmt systems, etc..."
  • Hilska: "To get more info on Flowdock, we're showing at 11am"

  • .@RallyOn: "Mastering Impact: in 2006, focused on flow; 2009: book: Getting The Right Things Done; created True North

  • .@RallyOn: "2009/2010: how do we become Disciplined Explorers; now 2013, thinking about Innovation: being brave enough to cancel backlog

  • .@RallyOn: "Our business is like an option pricing business; some things in your backlog is 100x more than others; how do you find which ones?"

  • .@RallyOn: "That becomes an exercise in PPM:

  • .@RallyOn: "Emerging org: R&D functional (nee UX); Business functional (nee Marketing); R&D strategy (nee New Initiatives)

  • .@RallyOn: "A vision: moving from 'System of Record' to 'System of Engagement'"

  • Next steps

    • Wed 8p: Rally ALM and PPM roadmap
    • Tue 945p: Enterprise Lean Startup
  • Q: "How long is your UX runway?" A: "

Planning

  • 2:15
    • Eric Willeke: Xanadu II
    • Larry Maccheone: Tent

Misc

  • @JackieRebrovic: View from mine and @hshain's room! #perks #rallyon2013 #flatirons #stjulien #boulderhttp://t.co/FpjwmnSOs7
  • @odgrzybianie: RT @RallySoftware: Video: “#RallyON2013: Leading An Agile Organization” - http://t.co/UKuuv52edb Agile Leaders in Boulder June 3-5. http://t.co/yM1WR9fYt5
  • @korbelandrew: @alshalloway Cool to see you are the SAFe team w/@Deanleffingwell...hope to see both of you at #rallyon2013 for some good conversations!
  • @JackieRebrovic: Cornfield! Stuffing bags...all in a day's work #RallyON2013 #hustle @ Rally Software Development http://t.co/cQu8IMiAR1
  • "What does your inbox or Leaning Tower of Pisa have in common?" "Neither of them are going away. Leave your inbox alone -- it'll be there in 300 years"

  • PS: I'm posting my notes from #RallyON13 user conference here: http://scribes.tweetscriber.com/RealGeneKim/143

To Quote File

  • @bjlang: RT @benbuxton: Watch the work, not the workers. #Agile #RallyON13