2014/07/17: Creating High Velocity Organizations, Dr. Steven Spear

by Gene Kim on

#devops

Creating High Velocity Organizations, Dr. Steven Spear

(40-60 attending)

  • Spear: "MIT goal is to solve societal problems; carved into stone ceiling in atrium
  • Spear: "all with varying missions: healtchare (provide care), resource extraction; some do better than their peers; gaps are extraordinary, unbelievable
  • Spear: "complex system: how to do you get people choreographed to solve a common problem
  • Spear: "they seem exempt from tradeoffs like cost vs. quality; high perfs do tradeoffs so favorable, defy belief, earn envy of all
  • Spear: "Performance: gap betw GREAT & TYPICAL is HUGE;
  • Spear: "Pace: great performance comes from discovery, not decisions
    • Spear: "Academics want to boil it down to a set of practices
  • Spear: "Principles; how to manage complex systems for high velocity, high endurance decisions
  • Spear: "My job is not to create lesson plans; students come to you for insights that make a difference in what they do; outsidesr often perceive a very simple value statement
  • Spear: "Ex: healthcare: they take care of you when you're not feeling well, and make you feel better; utility: take carbon rich resource and turn into another form of energy; no, they give you safe reliable power
  • Spear: "Hotels: take a behind scenes tour of major hotel; the level of detail is extraordinary to provide welcoming, comfortable accomodating area
  • Spear: "Exercise: tell other people at table what you do?"

  • Spear: "Observation: most people were quickly able to tell from customer perspective what they do

  • Spear: "when we satisfy customer needs, we earn the right to do what we do

  • Spear: "Intel plant: 40 miles west of here: throughput: 70d -> 20d (about as fast as any in world); +$10MM per month; unit cost cut 50%; everyone knew this plant couldn't compete: shutdown list: 30 people working at plast

    • especially in elec, there's huge value in getting prod into market quickly; at 20 days is incredible; now 20% free capacity; yield loss disappear; unit cost dropped by 50%
    • Intel: $120MM annual profit; only factory if its type not shut down, earned 5 years of existence
  • Pratt & Whitney: mfg jet engines not as difficult as earning right to make them; to be candidate in game for jets in $500MM proposition, in contests (F-15 fighter)

    • every card ante is billions of $$$; you lose lots of hands
    • P&W: 4-5 yr design time; can we do it faster, and get better design;
    • 4->3 years to have product ready to be evaluated; (home contractor; customer accepts first bid; change orders of 'good ideas' is what makes money); chg orders lose money
    • cut change orders by 50%; won F-35 JSF; largest weapon system ever
    • went from commercial precariousness -> annuity that will run for decades
  • Utility: Honda & Toyota devestated the rest of auto industry decades ago; "it will work for them and not for us"; examples: "we're unionized, they must use easy process, ours is diff to muster

    • Detroit Edison: not Japanese,, highly unionized, not even repeatable process
    • They do field service ops: guys w/tan coveralls in trucks; to compete, we needed to be 2x more productive; why weren't they?
    • led to back office ops; burning of coal (we've done it for 10000s of years): even using it to create steam is 100s of years
    • looked at economic stresses in Michigan: revenue will decline, so we need to cut costs; asked why it costs so much to generate power?
    • looked at chemistry of coal burning: cut opex 25%, without affecting headcount; now does it better than anyone else
  • Healthcare:

    • elimination of complications: want to provide care, comfort, cure; 1 in 200 hurt; 1 in 2000 die; safer to BASE jump; not parachuting, BASE jumping; cut accident rate by 80%
  • Social services: women's shelter in Pittsburgh

    • time to being settled: navigate 42 state and county agencies to find next place to sleep;
    • takes 4 days; what do I do in meantime? sleep in car with my kids?
    • what would great be? that night!
    • months later: people would call, say "I can't take it anymore, I have to leave husband", now answer is, "don't worry. there are 42 agencies, but we'll navigate it for you. it's 4 hours. come in right now, by tomorrow morning, you'll be safe and secure"
    • Mission: "we take people from great danger to great safety"; the problem, there's too many people still who aren't safe
    • we reach them too late; where can we get contact with them earlier: police force
      • also neighbors: who notice trash can is out, you left your lights on; they're also the people who call the police to say, "I hear screaming, I hear something breaking"
      • police show up, instead of breaking down down, separate victim & abuser: ask her, "what do you want to do next? File charge?" "how can I file charges, if he'll be free in 5 hours, so I won't file charges. So, no, I won't file charges; he loves me."
      • If we can settle people safely in 5 hours; started working with 800 Pittsburgh police; they separate them, say "we can find you safe place in 4 hours; talk to lovely lady on this phone; let me get this bum out of here."
    • think not about defect rates in manufacturing; we measure defect rates on "second black eye"
  • Alcoa: impeccable w

    • every form of energy (chemical, electrical, thermal) that can harm human beings, and put it in concentrated form; Alcoa has it.
    • hurt on job: 2 out of 100; 1 of us in this room will get hurt
    • 50% rate of getting hurt;
    • why isn't it that we can't hurt people? why can't we be perfect
    • hurt rate: 2% to 0.01% per year
    • 2%: a friend will get hurt; over course of career, you will be more likely than not of being hurt
    • at 0.01%: likely you won't know anyone who gets hurt; unlikely to ever be injured
    • $100MM recurring costs cut
    • Applied to admin processes: 11 days -> 2.5 days;
    • When O'Neill became US Secretary of Treasury: they closed US Govt books: 6 months to 3 days
      • "It takes me longer than that to balance my checkbook; only 2 people can write checks; Fed Govt has millions of people; amazing!"
    • bean counters objection response: if we can't focus on safety, we won't ask how we can perfect on yield,
  • Questions: what does "2x as good" mean to you?

    • Most important thing I do?
    • How is success measured?
      • Lead time and Book sales
    • Current measure?
      • 2-4 years: 52K books
    • Game changing score?
      • 9 months
  • defects lead to recalls: in cars: millions of $$$

  • 9 months of communciation: between R&D and packaging; small launch window

  • customers almost always only care about 2 things:

    • time (how long to design, convert silicon to chips, phone call to safety): more timely is always better
    • reliability (secdonary): product quality
    • corrollary: we know that long lead times disappoint customers
  • Pace

    • we're always captivated by altitude difference: half people/space/inventory and twice the output
    • as much as we focus on elevate performance (and vendors who promise who can help you), the data and logic is overwhelming: focus not on altitude difference, but the climb rate
    • Detroit Edison: "I want to improve productivity or op ex"; response: "that's brilliant, that's why you're the CEO and paid the big bucks"
    • in auto industry, studying Toyota, there was such a thing as a bad car; theses days, there are no Yugos
    • look at how efficient and cost; Toyota: they're performance altitude was so much better: 1985: it was; but before, it wasn't, it was lousy
    • 1958: Toyo Ped: bestseller in Japan; brought to US; bombed in marketplace
    • Spear: "we're dazzled by altitude diff betw high vs low performers; data is overwhelming: focus not on altitude, but climb rate #devops
    • 1962: not only was car lousy, but extremely lousy productivity
    • By 1973, when re-entered US market, doubled world standard (and can sell far less expensively)
    • Spear: Before 1973, could predict who owned what cars, by size gaps in the oil stains (big vs small car; size of pool)
    • 1980s: bestselling car was Ford Taurus: the problem was there was 4 yr gap between model years; what is competitive advantage of being on 2 yr cycle
    • Lexus: 1989: bestselling car 2 yrs later
    • Scion: did the same thing for younger market
    • fuel efficiency: Nissan as the Leaf (good for 2 good friends who are skinny); Chevy Volt sold 50K copies; Toyota has sold 5MM copies, on 21 diff platforms, 6th generation of technology
  • given choice between "good" and "great", why would you ever choose "good?"

    • best logic: maybe tomorrow will be a better day
    • diff between parents & grandparents
    • real reason, people don't know how to be great
    • Difference between good and great: know a lot vs. know something; ---> Discovery rate (accumulated learning)
  • Spear: "All data is that diff between hi/low performers: creating ability to discover our way to greatness"

    • played trumpet for first time: "what did you think?" "that is one great looking trumpet; is that real ivory? playing stunk!" "there was no relationship between music and what you played." "I guess I need to learn to read music"
    • what was difference: difference between music written and music played; spent time learning to play notes; notes in horn come out in same order as on page
    • still sounds mechanical; you got right notes and appoximation, but timing is all wrong; you sound like alarm clock
    • once I got to sustaining level on playing notes, then allowed me to look at looked at timing
    • next: no emotion; difference between CD and your playing; you got none of that
    • next time: he's playing at town library
    • each level allows you to solve the next problem and achieve the next level of performance
    • in our labs here at MIT, people see solutions, opening window to new problem; they'll tell everyone about the solution, introducing more subtle problems
    • same thing in astronomy: horizon used to be moon & stars; now we can see the event of creation; we always see further
    • must get into dynamic of seeing we don't know
  • Next story:

    • Mike is an adult, unafraid to show his incompetencies to everyone else; come on eover, and I'll show you how bad I am
    • all had MBAs: 600 class sessions: 50% from Harvard; 50% from Wharton
    • "I want to share everyone how I'm wrong I am, cuz I don't know what right answer is"
    • 20 ppl: no one took 600 opps to say "professor, call on me so I can learn about my wrong answer"
    • 600K opps: no positive responses:
    • the hard stuff is to show what you don't know: shows fallacy of "see, solve, spread"
  • shows fourth key attribute one: agressiveness to see something wrong, and aggressive to spread:

    • At Harvard and Wharton, it's not safe to say "I don't know what's wrong"
    • Spear: "My observation: at Harvard and Wharton MBA programs, not safe to say 'I don't know'"
  • Discovery Imperative: Leaning into the problem space

    • how do you sell more?
      • cheaper
        • use cheaper products
        • make sweeter, more addictive
      • make it taste better: sell premium version
        • offer more flavors
      • improve packaging
      • bundle it
      • ask at point of sale
    • consultants came up with these ideas: concluded wasn't that it was a good idea
    • hourly data of sales: it was odd: steady, spike was at 6am for an hour; lunchtime; 3pm; 6pm; why?
      • kids buying it; looked into outlets to see who's buying it; middle aged men at 6-7 am; not high school kids
        • asked why? this is one awesome commuter breakfast; neat and clean; can drink in car; fun to drink
      • 3-4pm: soccer moms; why? kids are hungry; filling and gives them energy;
      • evening: teenagers out on dates: why? height of romance
    • What is the problem we're trying to solve?
      • Commuter breakfast: different for breakfast (fruit smoothie: more nutritious or more breakfast item): solve breakfast problem, not thick shake problem
      • Afternoon snack: soccer mom: thick shake melts in minivan; cna't clean it; gotta burn the car; fruit with caramel dip
      • Dessert: evening: romantic date dessert problem; thick shake sharing problem; one cheescake 2 forks
  • Honda Element vs. Scion:

    • Element: tens of thousands
    • Scion: rampaging success: millions sold
    • Honda: they want active lifestyle: skiing, beach; they have lots of gear; synthetic; easy to hose down; moon roof; can change in the backseat
      • great for women to take husbands to go antiquing: rocking chairs
      • avg age: over 45 yo
    • Toyota: most young people don't have active lifestyle;
      • we don't know what we're trying to solve for
      • set up dealership in SoCal: little Corrolas; asked people "what would you want me to do to this car so you'd buy" -- had body shop, so they could rapidly prototype the car
      • couldn't figure out the pattern; imagine salaryman saing, "can't find the pattern; this is a wickedly hard generation to solve for"
      • discovering iPhone; this is my highly personalized music carrying device (they all look the same)
      • my playlist is different, despite same basic platform
    • aha moment: personalization, not features
    • expresses your individuality
    • Often difficult to ask "what is the real problem?"
  • Question:

    • what is your problem space? (instead of analyzing your processes)
      • instead of focusing on growth rate or market share; instead of improving thick shakes, they ended up doing products that had nothing to with thick shakes (breakfast burritos)
      • ethnographic studies
      • Pfizer: heart condition study: Viagra was different was repurposed
  • Video: we teach kids w/positive and negative: US history and civics;

    • George Washington and cherry tree
    • Benedict Arnold story: traitor who gave West Point plans to British army
    • contemporaries; both generals; Washington becomes figure of incredible historical significance; Arnold was insignificant/inconsequential;
    • you typically need pairs: Cain/Abel; Washington/Arnold
    • 2000: show people in healtchare industry that they're wicked dangerous
    • Emergency Caesarian problem
    • What did we hear that captured attention?
      • lots of fingerpointing
      • cutbacks exacerbating problem
    • What did we hear?
      • Remorse: I'll live with this for the rest of my life, not sure if I can do this ever again
      • Don't blame me: I did everything right
      • Blame someone else: I told him... Code team wwas too slow; wasn't job for a medical student
    • Predictable pattern of behavior
      • Oil spill: it's terrible, wasn't me, it's George's fault
    • Common theme: you can scrub the pelicans all day, it won't solve the problems all day
    • What went wrong and where could we have had an opportunity for a deflection? Prevention is key
    • Exercise: one thing that goes wrong, and who does it
      • observation was squashed; doctor allowed himself to be overruled
      • spinal wasn't working; doctor pushed; command and control, pathological (chief surgeon was real bully)
      • anesthiologist was a wimp
        • "I need a few more minutes" "you can't have it"
        • countermeasure: "until we get XXX, we can't do YYY"
        • actually, no way; we get a good outcome
        • two men having a fight, not about a sport; one wants to slow down, the other wants to speed up; baby vs. mother
        • role predicts the side they're on:
          • anesthesiologist has proximate and intimiate relationship with woman
          • surgeon is separated by screen; monitors he hears is for the baby
          • guy with the kinife makes the decision
      • staff couldn't find c-section room
      • drugs in close proximity
      • pediatrician too busy; rushing case
      • nurse gave wrong medication (instead of vitamin K, got meds for mothers to stop bleeding):
      • how did methergen end up in the Vitamin K box? (she pulled from right location): human factors issue
    • this is a story about oxygen
      • intubation
      • medication of infant
    • pre-ordained bad outcomes
      • roles of anesthesiologist vs. surgeon
      • pediatrician focusing on baby here or babies otherrwhere
      • rushed, so it's delayed for baby, but can't be delayed for peditrician
    • how do we relieve them of failure?
    • Must go back further in time
  • 10:30pm: 90 minutes earlier: video

    • Nurse says to Dr. Feldman: I need you to look at Ms. Romanov
    • if anasthesiologist saw her then, wouldn't have siad, "why is this caught so late? another rush job"
    • what goes wrong
      • delegated to med student
      • overloaded
      • priorities and distraction
      • squeaky wheel: nurse was squeaky wheel: raises sense of emotional urgency (with no urgency)
      • heightened emotion
      • med student:
      • Real issue
        • urgent: room 3: urgent birth imminent: clear information
        • room 4: heightened pressure, "I think you should take a look"; very vague, so send a med student, so can you get me more information, given indication of distress
    • Deflection
      • no deflection in operating room
      • how about here? 10:30pm is too late, too. Because Dr. Feldman is getting bad information
  • 8:40pm: Earlier

    • Mother is at 42 weeks, instead of 40 week normal
    • Normal: Head not crowning, Cervix not dilated, labor, pain, show
    • Anomalous: Decels, decreased movement
    • Any surgeon would have deflected; but didn't get to Dr. Feldman;
    • only information presented was Decels
    • Video: what goes wrong?
      • Unit nurse: Betty: Triage Nurse
      • Handoff
      • all info presented
      • Why was info presented difficult to comprehend
        • Rushed, verbal, other conversations and distractions
        • Sequence,
        • acknowledgement, confirmation
      • Better passdown procedures
        • Written, predictable structure, calm, uninterrupted (when interrupted, they start over from scratch), confirmation, explicit recommendation ("she should be examined quickly")
          • early onset Alzheimer/dementia: it went away by 30s; why? kids demanding attention all the time; kids stop demanding attention made Alzeimer go away
        • counting out pills: when standing in this tape, don't interrupt me
    • it turns out that they're not even having the same conversation
    • what if Ms. Romonov had bannerr w/danger on it, as she wheeled into the ER
    • spread too thin, or overburdened by their own faults
    • what would have helped?
      • flag for triage nurse for Ms. Romonov
      • passdown in template
      • calm box
      • data + interpretation + recommendation
      • confirmation
    • Only point of deflection: not in ER (conflicting goals), not at 10:30pm (bad information) -- had to go back three hours
    • why passdown? we start work, continue work, stop work, passdown work
    • firefighting; stimulus/response
    • cheap, 10 minutes to generate;
  • why didn't they intervene here?

    • 12 deliveries on shift; 12 passdowns
    • 60 passdowns from triage into nursing
    • 3000 per year
    • time delay of consequences
    • distance and time and people
    • urgent: wee can see the consequences
    • if we can't see the consequence
    • happens all the time, but we can't anticipate the consequence
    • b/c we will never here, "this one passdown we must get right"
    • what is the principle?
      • pervasive process that happens all the time?
      • is it something about passdowns in general that have this property?
      • is it that this activity that where consequences are distant in time, space and people?
      • only point where we have control?
      • protected the value in the assessment? information was destroyed?
      • all processes should be standardized?
      • all of the above?
      • none?
    • >>> All work has to be done so that we can see a problem
      • the problem was lack of alarm:
    • Difference between standard and alarm: when pilot says, "we have turbulence, put on your seatbelt"
      • it's the only time that we have an alarm where we can't see the consequence
      • like mike playing trumpet; alarms are subtle, do we respond
      • Challenger: shuttle disaster: rubber, cold, cpressure: we had data thhat hot gas exhausted; they didn't intervene; decision couldn't have foreseen
      • deviance got normalized
      • Columbia: orange foam broke shuttle tiles; in orbit, maybe can intervene; on reentry: no way; when do we have to interene, before it happened
      • missing alarm: foam fell off in every flight, hit space shuttle; caused crack every time; big cracks; 1st flight: concerns; by 20th flight, no big deal, we'll fix it
        • only place to save columbia: on enterprise
  • opposite of not ffixing isn't "not good to great," it's disaster

  • video: june 18, 23, 25

    • Ms. Romonov: shows up on july 7th
    • what goes wrong?
      • "come back on june 25th"
      • OB clinic call center: overburden: earliest appointment is july 12" (resulted in protest)
      • interruption
      • no voicemail; got busy signal
    • intervention could have happened here:
      • electronic messaging, voicemail,
    • do they respond to this alarm?
  • How Systems Fail

    • Spear: "The alternative to great is not good, or even mediocre; it's setting ourselves up for conditions of total disaster"
    • System metaphors: weak link in chain; assumes linearity
    • spider web strands: our systems today are complex
    • only strategy that spider can take: fix every strand, one by one; or build new one, if we can't keep up; especially when it's cheap to fix the strand, and we don't know what happens when the strand breaks
  • Story: young engineer going to meeting at 2pm;

    • idiot, fool or moron; be prepared to learn; come with a list of questions and objections
    • wihtout it, you haven't set an alarm that you're ignorant
    • Rickover and Tim Rockwell
    • Before engine test: Rickover: testing emiisions: what are we really testing:
    • not only is 11 bad, but 9 is just as bad, too. "where diid the 1 go"
    • we have to predict the reading
    • perfect safety rating since 1949: Nautilus
    • Soviet experience: fires, sinkings, contamination
    • 60-70 yrs of perfect vs. chronic catastrophe

Paper airplane exercise

Reflections

  • Reference: chapter 6, The High Velocity Edge; table 6-1
  • some of the decisions that doom failure; no simple way to make decisions
  • having a methodical approach
  • arbitrary choices; you make angle, I make straight; b/c it's in front of me
  • not have built in habitual methodology, like asking customer
  • what is the actual work content: rip apart the prototypes
    • suggestion: take picture after each step of disasembly
    • take pictures as we take it apart
  • System
    • what is the output: if we don't ask customer, we're firing blind
    • architecture of system: pathways (verbs) in what order, performed by whom
    • connection: content, format, timing, location of exchanges (nouns)
    • method: work content, timing, location, sequence
  • now that we have a system, we can constant check if system is working
  • "do you know how much time we took to create standards, so we can see if can even see if something is broken? We constantly do experiments"
  • paper airplane
    • design standard
    • test standard
    • run standard
    • improve standard
  • Alarms
    • the secret airplane:
    • did someone ever name the paper airplane #
    • everyone watching the test, and seeing someone not be able to build one step
  • Simple principles for designing systems
    • why were Italian architecture were so repetitive?
      • standards?
      • why? it can't fall down. 1400s; what did they know about why buildings stand up or fall down? they probably didn't know
      • risk averse: build it like one that didn't fall down
      • b/c no one knew how buildings fell down, they imitate those that stay up
  • Newton: F=ma

    • predicting the future: just add up all forces
    • you don't need precedent to predict the future
    • next breakthrough: computers; torque calculations
    • Frank Gehry used computers; Eiffel towers needed hand calculations; few calcs per minute; billions of calcs per min
    • how did contractor have enough confidence that it will stand up; how did they get zoning? b/c computers confirmed it
    • In Italy, they could only replicate the past
    • "MIT is right place for me, b/c I'm a flipping nerd"
  • keep it simple

    • mechanics
    • economics
      • people respond to forces
    • finance
      • value affected by payment timing, decision timing, risk diversification
    • systems design and operations
      • learning speed for complex, dynamic systems
      • build in feedback to trigger: control, improvement, knowledge generation and spread
      • having simple rules that we can apply at speed creates incredible results
  • systems fail if we have ambiguous design on inputs and outputs, and no specificity of how work is performed

    • underdesigned:
    • think about what systems in our org look like Round 1
    • approach: "what is frustrating to you?" -- test your assertion
      • ill defined connections, methods, outputs
      • Intel: preventive maintenance was frustrated
      • you look frustrated, why?
  • imagine doctor had to ask same (or worse) different questions to each patient? hard to imagine that you can get good care that way

  • @botchagalupe: Target is serious about #devops - gold sponsor at #devopsdays - 50 attendees - an awesome story to tell

  • how to start in software

    • in autos: 10K parts and 3K people: lots of detail
    • at top level: coil of steel, stamp, weld, then paint, and then put other parts in it
    • 4 steps: then put powertrain in (that's 5 steps)
    • chasis to paint: worrry about baton pass
    • The big handoffs enable decomposition to the smaller process
    • how does body shop decompose: receiving -> fixturing -> welding ->
    • Welding decomposes into 5-6 steps
    • Isaac Newton: mechanical things: we want plane to fly (point object); how much does it weigh? dictates lift and thrust
      • then fuselage and wing; then look at beams in the wing
    • how did you know which of the 5 to go to?
      • we're suffering; diagnosis; who is generating output
      • fine in body, paint; but sticks in Assembly
      • Everything is fine until we attach the wheels
    • identify something that you can latch; then dig further

Misc

  • @potto007: RT @AndiMann: "The course we're running now isn't doing #DevOps any favors" - great piece on defintions & consenus http://t.co/upWze3qDTr v…
  • not enough resources to even help with RFP and RFI: een to hire people
  • @releaseteam: Union Bank Puts Its Money on DevOps Baseline http://t.co/03QeuLXSTv #devops #ibmpartner
  • @Lanooba: RT @jeffsussna: .@sascha_d key #devops principles: empathy, trust, reflection. And the hardest one: inclusiveness. #devopsdays
  • @beerops: RT @jeffsussna: .@sascha_d key #devops principles: empathy, trust, reflection. And the hardest one: inclusiveness. #devopsdays
  • @Lanooba: RT @jeffsussna: .@sascha_d key #devops principles: empathy, trust, reflection. And the hardest one: inclusiveness. #devopsdays
  • @CloudBees: What are the best resources for learning about #DevOps? This list is a really good starting point - http://t.co/Zo6t1GrL7p

RT @Lanooba: RT @jeffsussna: .@sascha_d key #devops principles: empathy, trust, reflection. And the hardest one: inclusiveness. #devopsdays
RT @botchagalupe: Target is serious about #devops - gold sponsor at #devopsdays - 50 attendees - an awesome story to tell