Starting DevOps

by Gene Kim on

@botchagalupe

  • DevOps is a culture problem
    • Proof that DevOps is primarily a human problem: in 10 day deployment, 8 is spent waiting in Dev/Ops handoffs
    • When you do acts repeatedly, it becomes habit. (Like Talent Code, Rother Improvement)
    • The tale of two wiring closets: one is hodgepodge of cables, or mandatory wiring
    • How to make habits: Practicing 15m/day is far better than practicing 2 hours/week: that's culture
    • Must integrate "improvement of daily work" into your daily work. Otherwise, you don't have the necessary culture
    • If you interview w/company "doing DevOps," & you see messy data center wiring, that's a red flag of screwed up priorities
    • Like someone who dresses like a slob vs. someone who dresses neatly
    • Appearances matter, because it's the most visible manifestation of culture
    • Thin-slicing
    • Culture is visible: you can tell
  • Good habits
    • Trust
    • respect
      • Then golden rule, like kindergarten
      • It's like a cancer: many organizations are disrepectful
      • Because DevOps & transformation are so difficult, trust/respect/open communications are prerequisites
      • Anti-trust: "I'd never allow those kunckle-heads to deploy that code in my production environment"
      • "Managers need to demand good habits of respect/trust in every interaction: within & across Dev/Ops"
    • communication
    • no victims
      • "Jody Mulkey: when we go into the war room, the problem is the enemy. There are no victims."
      • "No sense of powerlessness"
      • "We're not warring tribes: where someone loses and somebody ones"
    • Smell Test
      • Chris Read: "the smell test red flag: no none knows what Joe does; he doesn't do anything" that's an org/mgmt prob
      • "The fact that someone is question Joe's value or competence indicates some org failure "
    • slay the dragon
      • "in a 10 person startup, everyone knows who the enemy is: slay the dragon; but when orgs grow, tribal warfare starts"
      • This is the classic tribal warfare anti-pattern
      • Organizational pressure and challenges
      • Culture of insitiutional paranoia: chaos monkey, coaching kata
      • 'As a habit, are we spending enough time culturally asking "why are we here?"'; helps keep tribal warfare away
      • "Are we spending time to encouraging respect, trust, and slaying the dragon? If not doing this, not ready for DevOps"
      • (PS: you're all witnessing @botchagalupe/me are working on #devopscookbook, organizing cultures vs. habits vs. steps :)
    • fearless behavior
      • "if your org isn't fostering fearless behavior, you'll never be able to install/run chaos monkey" cc @adrianco
      • "prerequisite: w/o firefighter mentality of preparedness & rehearsing responses, chaosmonkey will just cause chaos"
      • .@builddoctor: "if everyone working recklessly, deploying code at 4pm then going hm, cont delivery will never work" -- instead, need trust respect, and dpeloying code at 2pm
    • piercing the veil
      • "we can't do this because..."
      • "we can't do cloud here." "why?" "PCI"
      • why?
      • People to hold up initiatives, say "stop, don't go further"
      • often impediments to process improvement
    • shaman
      • Chris Reed: the shaman knows the backstories, the people who can communicate between groups, someone who knows where the bodies are buried, the connector, boundary spanner,
      • Shaman: chief pilot: serves as insitutational and tribal memory
      • The Shaman: Chris Reed: We can't do CentOS 4.1: b/c we had this outage one time, but we've never really did a real RCA
      • They make
    • no arrogance
  • Leadership defines culture: DevOps requires 'line of sight leadership'
    • Is this true? "On leadership: If person at top of org never responds to emails, then likely their managers will, too"
      • Necessary prerequisites: leadership matters: need more than just tone at the top: need active support from management to succeed
  • 7 sins of devops
    • overproduction, delay, transporting, over-processing, inventory, motion, defects
    • isomorphic mapping between IT Ops and plant:
  • @botchagalupe blames tech debt on Larry Wall: ops laziness, impatience: the goal is not to read books, but be more productive (laziness isn't a really a virtual, but automation is)
    • 20% off kilter: has had grave impacts on culture of system administration
    • impatience: get off my office, go read the manual: it's efficient but it's not systems thinking
    • Chris Reed: 2x sessions with junior sessions: because he gets how important it is
    • Perl programmer is the uber sysadmin: Larry Wall laid down the law
    • hubris: sowed early seeds of destruction
  • John Willis: 3 great virtues: automation, mentoring, fearlessness
  • DevOps strategies
    • start small: hack culture at high level: repitition creates habits; focus on improvement of daily work instead of big ass project
    • DevOps strategy: create champions: @jesserobbins: "I'll take all the blame, you get all the credit": before you trust the message, you must first trust the messenger
    • Anti-pattern of start small: Ripping out the email: the best you can get is match the old; $3-4MM training for McKinsey for sysadmin org; wanted to do Lotus Notes infrastructure first; Tivoli rollout took down email for 1.5 days;
    • Who can be my champion: what is the relationship I need, trusted relationship; it's a great opp, greap opp for the project and for you.
    • Your goal is to create winners -- don't need credit
    • Metrics for confidence: we use metrics to prove your point: we don't collect metrics or just collect them because... show that we're doing what we promised
    • not deploys/day; Stephen Nelson Smith: soft metrics
    • Faces board: what does everyone's face look like at end of day
    • B/c so many DevOps success factors are human-related & culture, success metrics should be 30% non-technical
      • HELP! What do you call the agile technique of collecting # happy/sad faces each day? Need screenshot!
    • The flakiness of AWS is forcing a degree of app fault-resilience that Rugged/Ops/Infosec have wanted for decades
    • As opposed to hw vendors you can hold accountable, AWS basically says, "tough sh*t: not our problem. that's our SLA" Haha
    • For ages, we paid our hw vendors provided our availability. Now, AWS expects sw to provide availability. Or else.
    • Maybe this is the true spirit of @adrianco: it's your job to provide availability, not AWS's. You can do it, despite AWS
    • Jesse: negotiated with Lloyds a contingency/liability plan for how many Amazon data centers before they get reimbursed for nukes
    • @lordcope: do you have pic of daily happy/frowny face scorecard? Many thanks! Talking w/
    • Celebrate success: SREs have cool jackets, having parties
      • Intuit story
      • "You can take the girl out of the gutter, but you can't take gutter out of the girl." (harhar. we're joking about IT Ops)
      • @allspaw tweeting like he's frustrated about the growth and complexity
      • Like Six Sigma black belts: show status and respect
    • Exploit compelling events
      • cloud computing, cmdb, big outage, game day/firedrills, look for big change (e.g., an acquisition, taking over someone else's web property, licensing issues AIX, AIX -> RedHat transition)
      • Be the guy in the back of the room with a possible solution
      • Cultural permission to change: change is not event: it happens all the time
      • White paper: Collab.net
  • Deck 2
    • Aha -> kaching
    • "if you started car company today, you don't buy raw materials first: you build assy line 1st. why do we buy hw first?"
    • Jesse Robbins IT Ops as strategic weapon
    • Zynga case study of cloud story goes wrong
    • diagram came from DTO engagement
    • day of recknoing was when Google was going to use them as gaming platform as choice

For Flynn
* after tweeting, focus needs to go back to main window -- currently disappears to who knows where
* when in bullets, pressing CR results in endless stream of bullets -- want two CRs to result in normal text
* Flipping back between sessions always moves focus to top of screen: rather we retain focus
*
* Shift-Tab scrolls to top!
*
*