3/22 Kanban Day 1

by Gene Kim on

#kanban @dominicad

My introduction:

  • For 13 years, I was founder and CTO of Tripwire
  • Studied high performing IT orgs since 1999, that's what turned into the Visible Ops Handbook
  • Currently full time author, working on 3rd and 4th book -- John Willis is co-author
  • based on our reserach, it appears that tools like Kanban are important for IT Ops, and combining the work streams of Dev and Ops
  • Using kanban on daily basis to manage the two people on my team, and the three agencies helping with publication, marketing, etc.

  • visualize histogram

  • Damon and Alex Honor

  • Luke Kanies

  • Travis, CarFax project manager

  • Kelsey, program manager, Microsoft

  • Project Manaer

  • Jamie, Manager Production Control

  • John Willis, 4-5 people services, Scrum, hiring 9 people, all remote

  • Alex Honor, lean afficianado

  • Betsy, enterprise release manager, Cisco WebEx

  • Jane, program management

  • Dominica DeGrandis: @domincad, dominca@djaa.com

  • Started working with David Andersen, ex Microsoft

  • In room w/luminaries: @puppetmasterd/@damonedwards/@botchagalupe/@alexhonor at @dominicad Kanban training

  • Fundamental Kanban goal: balance capability vs. demand

    • Available options: improve capability (word harder), shape demand, eliminate causes of failure demand (interruptions)
  • Foundational principles

    • Start w/what you do now: no sweeping reengineering changes
    • It's not an SDLC: you can put it on top of scrum, XP, waterfall, etc.
    • Agree on incremental change
    • Interesting! Agile Management: applying TOC for business results (David J Andersen & Eli Schragenheim)
    • The Big J curve: performance dips after process improvement injection (often resulting in change agent getting fired) (haha)
      • function of tolerance,
    • Respect current process, roles and titles
  • 5 core practices

    • visualize
    • limit WIP
    • Manage flow
    • make process policies explicit
    • improve collaboratively (using models & scientific method)
  • Corbis example

    • build ready column
    • next/investigate/implement/test/deploy
    • A pull system cannot be overloaded if capacity (# of cards) is set correctly
    • Univ of Mich study: perf went down by 40% when going from 1 -> 2 concurrent tasks (!!)
    • set WIP limit: (3/3/3/2/infinity)
  • Too much WIP

    • Too much WIP results in unevenness in flow
    • non-instant availability of specialist skills
    • information fails to arrive before it's needed
    • hidden/implicit classes of service that cause work to be interrupt
    • variety in work (complexity & size)
    • analysis: cumulative flow (# cards vs time)
      • smooth flow vs. ragged flow
    • SPC
    • make process policies explicit
      • "only check code into the Dev2 branch"
    • TOC, Deming, Lean Economic Model, Cynefin framework, real options theory, monte carlo method
    • Kanban is not prescriptive at all (vs. Scrum, which is very prescriptive on team size, roles, rules, etc.)
  • Quotes

    • "A complex system can only be understood by interacting with it" -- David Snowden
  • Exercise 1

    • Describe customers and analyze demand
      • types of work (planned/unplanned, value demand/failure demand, emergencies/standard)
      • arrival rate (unevent, seasonal or monthly functuations)
      • expectations (SLA?)
    • describe sources of internal dissatisfaction
      • variability that randomizes the process (radical constant shifts in prioritization)
    • describe sources of customer dissatisfacation (or conflict)
      • it's always late, not what we wanted, etc.
      • "what randomizes our day to day work?"
  • types of work (planned/unplanned, value demand/failure demand, emergencies/standard)

    • Planned dev releases, infrastructure projects, audits, monthly processes (database storage manipulation, surge computation)
    • Unplanned
      • outages, audits, documentations, PMO comes in screaming (new release, new business opportunity, to save the deal we had to promise X, promotion); once a quarter report 5-50K/minutes
      • external: event driven, patch vulnerability
      • personnel changes or blockages (single specialized knowlege)
    • arrival rate (unevent, seasonal or monthly functuations)
      • sensational news: tsunami, bin Laden,
    • expectations (SLA?)
      • commitment already made by sales, marketing
      • best fit guesses by the team (not knowing complexity of task/projects)
      • abused spouse, powerless, BWAG (3 months)
    • describe sources of internal dissatisfaction
      • variability that randomizes the process (radical constant shifts in prioritization)
      • source of the McGuffins: mismatched expectations
      • crossing organizational boundaries (Speed of Trust)
        • worse with distance: Cynefin: organizational volunteer projects to encourage people to work together
      • Internal problem: source of bureacracy
      • Not knowing if you're working on the right thing
      • Person screaming loudest
      • Unintended thrash and reaching in (messing with dev schedule, changing things on the fly, 10 hours of estimation meetings)
      • Hire 3 more people
    • describe sources of customer dissatisfacation (or conflict)
      • it's always late, not what we wanted, etc.
      • "what randomizes our day to day work?"
    • lots of outages: 10 changes/day to website
  • trust: need to earn trust slowly, but one thing can destroy the trust overnight

Luke Kanies, case study

  • 75 people right now
  • total reboot of Dev team
  • product board:
  • too many boards
  • @botchagalupe: @puppetmasterd talking about #kanban at @puppetlabs 3 to 75 ppl in 3 years... Wow....
  • product board: feature, design, development, QA and release
    • now product board is the 1st thing that everyone sees when you get out of elevator: has awareness value @puppetmasterd
    • UX output card goes onto restaurant pull chain: "order up": value: see WIP amt @puppetmasterd
    • They're experimenting w/Kanbanery tool @puppetmasterd (OTOH, @hannahconconnan/me love @leankitkanban)
    • Another team is experimenting w/Trello @puppermasterd -- value of someone prioritizing junior engineer work was invaluable
    • Yet another 2 teams using manual kanban whiteboard @puppermasterd -- CEO getting involved "almost always dysfunctional" haha
    • OH: "my ambition [in previous role] wasn't to be person who always says we're going to be late [& miss project release date]"
    • kanban couldn't tell us that code reviews was largest bottleneck: showed how we approached it wrong @puppetmasterd
    • one place that displays all the red/green, showing how local decisions affected global goals was awesome @puppetmasterd
    • it's remarkably difficult to get data into one dashboard: no tools, lots of expensive consulting options @puppetmasterd
    • "When we had 25 ppl, we had 2 sysadmins; w/75 ppl, we still hv SAs; I guess we love underinvesting in ops" @puppetmasterd :)
    • @dominicad: Listening to @puppetmasterd talk on emergent #kanban property - tolerence for exploration. Cc @realGeneKim
    • "Better 3 things done well, than 10 poorly - it'll come back as rework. Showed resourcing probs" @puppetmasterd
    • "Awesome use case: Kanban for Personal family budgeting: wishlist/next/budgeting/buy" @puppetmasterd (Awesome)
    • On tolerance for exploration: "The thing I don't have tolerance for is 'lack of exploration'" @puppetmasterd
    • "Don't believe that u learn more from failure than u do from success" @puppetmasterd (recc books Switch and Adapt)
    • "Only thing you learn from success is what didn't work; w/success, you learn the important things" @puppetmasterd
    • "People who have only failed clearly don't know anything. Those who only succeed are obv lying" @puppetmasterd
    • "Lessons: visualize work, not time; value streams, not teams; someone in process has to own process" @puppetmasterd
  • Struggles

    • non-discrete columns/collaboration (design)
    • individuals with multiple responsibilities
    • parallel work items (e.g., test and docs)
      • begins at the beginning, ends at the end
    • splitting work items
    • understanding constraints with dramatic work size differences
    • our batches are still far too big
  • John: "long ago, most useful role was intern who watched over everything, and pestered people"

RT @botchagalupe: @puppetmasterd talking about #kanban at @puppetlabs 3 to 75 ppl in 3 years... Wow....
RT @dominicad: Listening to @puppetmasterd talk on emergent #kanban property - tolerence for exploration. Cc @realGeneKim

Board designs

  • Development board
    • input, analysis (in progress, done [buffer]), dev ready, development (in process, done), demo, test, release ready
    • >>> WIP limits in "Done" lanes is what prevents work from flooding into the system
    • Rows: change requests, maintenance, production defects
    • Allocation: 6% on maintenance, 90% on change requests, balance (say 4%) of defects
    • A separate done lane creates the signaling mechanism to downstream station when work is piling up
  • Help desk
    • Fast lane on top (swarm on complete)
    • Priorities pull top 5 tickets, work to completion
    • Then added "todo weekend"
  • #2
    • Firelane
    • Todo
      • Planned
      • Unplanned
    • Goal was to highlight the unplanned work
  • #3: @lordcope
    • Next (4), expedite (1), support (2), projects (3), waiting for, signoff
    • is no WIP limit in Waiting For a timebomb?
  • #4
    • Backlog, next next (10), next (4), doing (3), waiting (2), done
    • @puppetmasterd: see the card "puppet hadoop?" up there? Cool. :)
    • "next next" is the opposite of "done done"
  • #5
    • investigate/implement/test/done
    • sysadmin (2)
    • config/automate (1)
    • network/monitor (1)
    • dba (2)
    • security (1)
    • the people with avatars are allowed to float, and probably attached to the card
    • "Kanban motto: Stop starting; start finishing"
    • Tracking time of "investigate" helped us estimate our lead time
  • #6: Corvis: boards evolve over time
    • Almost all boards are over-engineered at the beginning
    • Release ready lane: people piled up the cards because of sense of accomplishment
    • blue = bugs; yellow = change requests; 80% was failure demand
    • Retrospective <=> ops review
  • #7: Dominica office
    • problem: dark matter was the weight of all the unapproved projects
    • Dark matter, approved being prioritized, dev, sqa
  • 8: two tier kanban board

    • SAP: brought out 100 people: whole floor for contractors
    • Epochs then broken down into tickets
    • SLA of 5 days
    • orange tickets: indicated specialized floats
  • Exercise 4

  • OH: "Tried using OpenOffice for 1 year. It was like using Trabant of office apps. Awful."

Lessons

  • Every team achieved their goal
  • Policies work
  • Carlos stabilized, but then variance went through the roof
  • during carlos era, what was the compensating behaviors
  • Google troy maguinness: monte carlo simulation
  • Game inventor: Russo Heeles: Australia: XP/Ruby shop
    • GetKanban.com game costs: $400