2013/03/14 DevOpsDays London

by Gene Kim on

#devopsdays

DevOps for Dinosaurs

Niek Bartholomeus

  • Buy before build
  • Heterogeneous enevironments
  • version control, release manaement, testing
  • dinosaur not likeamodern enterprises
  • first create config library, then fully documenting the release instructions
  • Oh, no! Why didn't they automate deployments? infosec said 'no.' Something about app privilege levels
  • Roadmap
    • automate deployments and environment creation; increase frequency of releases; continuous delivery (maybe)

Checking DevOps’ vital signs

John Clapham & Paul Swartout

  • @HarmB: #devopsdays Very nice talk on measuring devops culture @johnCbristol
  • @SquireMatt: Getting enthused about understanding how healthy our DevOps culture is. Great talk by @johnCbristol at #DevOpsDays
  • @marygrace: listening to @johnCbristol at #DevOpsDays London. talking #devops #culture, and how to check the vital… http://t.co/BycrGtGqHC
  • @mark_mann: #Monitoring culture at #devopsdays - McKinsey Org Health? #agile /scrum checklists? No.. We use Net Promoter Scores #NPS
  • @PierG: RT @themetriost: culture telemetry from nokia at
  • @freistil: #devopsdays Interesting talk by John Clapham about cultural vital signs in IT organizations. ^jl
  • @matthewpskelton: Great example of ECG machine (heart monitor) interface as a UI pattern for monitoring culture from @johnC_bristol

StartOps: Growing an ops team from 1 founder:

David Mytton

  • "Bootstrapping is opposite of raising money: means lots of deferring to last minute. There was only 2 of us; 1 for ops"
  • "First beta was 4/2009 on Slicehost: surprise: people used our svc, set up 2nd server just in case"
  • "Switched to mongodb: 50GB/month: even low volumes, had problems with drivers; easy to deploy"
  • "Taking shortcuts & deferring work: resizing disks meant hours of downtime" (showing 'df' output: / at 100%.)" Hahaha.
  • "Finding quickest way to do things: b/c scarce time/$$; network copy too slow; soln? UPS truck. Bandwidth = 11.22MB/sec" :)
  • "Even traveling was expensive; especially roaming"
  • "Don't update while traveling"
  • "Let other people help: Rackspace staff helped manage hosts, reboot, etc, when I was unable to do work, in car, etc"
  • "Engrs often bad at valuing their own time; outsource things like service list access, PCI certification"
  • (infosec people are fainting in aisles here :)
  • @milosgajdos: David Mytton from @serverdensity interesting talk on growing ops team
  • RT @milosgajdos: David Mytton from @serverdensity interesting talk on growing ops team
  • "Dealing w/people: automate as possible; ensure people only do things once: automating config mgmt has huge payoff"
  • "Siloed information is bad; Docs must be up to date; doc failure scenarios, vendor contacts, team contacts
  • "2010: staff grows from 3 to 12; we finally have on call rotations"
  • "On call: first line: 1 ops engr; second line: all engineers; third line: ops engineers"
  • "On call CEO: i get push notifications of all incidents and all outages; weekly internal report on uptime; we use google+, enables team discussions"
  • "Citing Amazon, Heroku as exemplars of public discussion of downtime issues"
  • "Outages test your vendor escalation process; citing problems w/Terremark 1st line support during 2am outage"
  • "When hosting providers have large scale outages, they're impossible to get ahold of: Rackspace phone systems broke"
  • "Must simulate failures: can systems restart, can failovers handle load?"
  • "Ideally, you want your own team, unlike vendors, outsourcers, your own team cares the most"
  • "We use pagerduty, bamboo, travis, papertrail, puppet; atlassian confluence, g+, hipchat, jira (kanban board), github"

  • @BMC_DevOps: Gene Kim and Patrick Debois at #devopsdays London @RealGeneKim @patrickdebois http://t.co/uDVHuCw2S9

Adding Business Metrics

Deri Jones

  • "No further competitive advantage w/Google AdWords; enter Facebook and social network"
  • Citing @lordcope: "why are we not monitoring business metrics alongside technical metrics?"
  • "What data does business want? sales metrics, but we don't want to dupe data they already have" (a good point)
  • "Example: lost sales is an easy one: plot delta of sales numbers vs. historical averages"
  • Multiply by your outages: in statsd/collector or in graphs
  • "Experience from 40 orgs: most biz teams don't know hour by hour typical sales; may not want to tell tech folks sales nmbers
  • "Trying out 'unbuyable products': customers can be found, but not bought: greyed out, empty shopping basket
  • "Reasons for this was biz trying to understand revenue misses due to web team or merchandising team"
  • @BMCDevOps: RT @HarmB: #devopsdays use metrics to calculate lost sales. @scivisum_UK http://t.co/tG22QR5JF8
  • @HarmB: #devopsdays use metrics to calculate lost sales. @scivisumUK http://t.co/tG22QR5JF8
  • @renierbotha: Quoting Deri Jones:"... unreliable websites is a repetitional risk..." It can be a reputation risk as well?!
  • @DevOpsGuys: Can't believe how many interpretations of "DevOps" I'm seeing.Used in diff context repeatedly! http://t.co/ez6x7c1lR9 @#DevOpsDays #devops

Vendor pitches

  • there's something really nice about these vendor pitches: treats us as intelligent humans and fellow practitioners
  • Enstratus, Riak

Ignite talks

  • Up: "Threshold Voting Workshops" by Sarah
    • Argh. Did anyone get that URL?
    • (Suggestion for speakers: instead of putting URLs in slides or reading them aloud, tweet them. :)
  • Up: "On Imperative (fabric) vs Declarative (puppet/chef)"
    • Haha. That was a joke. It's actually about "NoSQL should really be called Schema-less" -- unless he's pulling our leg again
    • "all automation should be idempotent; schema migrations are the hardest part of DevOps" (or deployment)
    • @kief: RT @pikesley: “DevOps is like violence, if it’s not working you’re not using enough of it” #DevOpsDays
    • "you can automate, or do what we all do: hire good DBAs & make them work at night" (hahaha)
    • "Use NoSQL; it's a great fit for continuous delivery, b/c changes are restricted to code"
    • "Scary stuff: no referential integrity"
    • @petemounce: @simonvc death to schemas!
    • @beddari: RT @garethr: OH: You'll spend more time building schema migration tools than shipping your product
    • Great talk on schemas by @simonvc!!!
  • Up: "OpenStack"
    • Cloud: $240 billion by 2020?
    • OpenStack: used by RackSpace, IBM, RedHat
    • @mary_grace: the cloud is becoming as essential a tool for technology as a power source for your computer. #ignite
  • Up: "What is Config Management?" by @patrickdebois

    • (@patrickdebois Ignites are allowed to go as long as he wants, and no auto slide advance. :)
  • Funny stuff on Ignites

    • @melaniersumner: RT @RealGeneKim: (Suggestion for speakers: instead of putting URLs in slides or reading them aloud, tweet them. :)
    • @thattommyhall: RT @garethr: Why ignites are ace in one moment. "What you see on this graph here is..." then the slide advances
    • @garethr: Why ignites are ace in one moment. "What you see on this graph here is..." then the slide advances
    • @beddari: Sadly the #devopsdays livestream never materialized. We are 17 people still watching the blank canvas, though

DevOps In A Traditional Enterprise

  • Up: "DevOps In A Traditional Enterprise": Thomas Falkenberg
  • "Payback is a customer loyalty program, one of the most popular brands in Germany"
  • "pre-2007: picture book waterfall: long release cycles; 2007: Agile but still long releases due to scrumfall"
  • "2011: first steps to continuous delivery and DevOps: inspired by DevOpsDays; difficulties encountered in enterprise"
  • "issues diff from startups: multidimensional env; complex tech stack; differnt depts; multiple VMs/datacenters"
  • @mary_grace: RT @bruntonspall: Interesting term. "Devops promoter". Someone who promotes devops within the organisation.
  • Nice! They're boundary spanners. RT @mary_grace/@bruntonspall: "Devops promoter", who promotes devops within the org
  • "Cause/effect diagrams: countermeasures: ops joins dev standups; config in svn; no manual chgs allowed; private repo for bins
  • "repo incl binaries: httpd2, tomcat, weblogic, etc; qa/dev does deplys for ops; fully automated deploys"
  • "tools: everyone loves tools, but don't solve problems; processes do; common tools across org & lifecycle ease communication
  • "tools: confluence, jira, graphite, jenkins, deployment tools
  • "tools sound easy but in practice, took much longer to get running; graphite took 2 days to get working b/c dependencies
  • "now tracking 14K metrics:
  • @rposbo: Biggest takeaway from #devopsdays so far; everyone should learn to use #logstash and #graphite well
  • @rposbo: Biggest takeaway from #devopsdays so far; everyone should learn to use #logstash and #graphite well
  • RT @rposbo: Biggest takeaway from #devopsdays so far; everyone should learn to use #logstash and #graphite well
  • "Before: deploys used to take 8 hours of downtime; now we're down to 1 hour, reproducible, automated, no snowflakes"
  • @matthewpskelton: RT @stack72: Good to hear that payback use logstash in qa as well as production to make sure logfiles are used in qa
  • "shared goals are important: dev cant' be measured by story points/features delivered; needs dev/qa/ops to work together
  • "devops requires patience: we started ~2 years ago; we feel we're at 50% complete; no one knows what 100% looks like"
  • @al_maisan: Interesting terms picked up at #devopsdays: snowflake server, scrumfall :)
  • @scoobiedoobie: RT @stack72: Good to hear that payback use logstash in qa as well as production to make sure logfiles are used in qa
  • @rakh1: There it is - snowflake server - everyone drink! #DevOpsDays London
  • @tomduckering: #devopsdays "Jenkins as a SPOF" - make sure your local build is the same as in CI so be careful what plugins you rely on.
  • @rakh1: EVERYONE is hiring, common theme.

How Can We Sell DevOps

Gene Kim

  • Q: "how did you get infosec onboard?" A: "create trust; took away root access for deploys; took away internet access"
  • @kief: Some popular topics at this #DevOpsDays: Security, #BigCompanyDevOps, #InfrAsCode. Less monitoring-heavy than Rome
  • "I was a perf engineer, always stuck between Dev & Ops; ops has much less pain in their daily work"
  • @stack72: RT @jonathan_thorpe: The Phoenix Project is one of the most important books I have read. Business owners must read. @RealGeneKim
  • @stack72: It took until 1237 on the second day of #DevOpsDays to get a Deming quote. @botchagalupe would not be happy
  • @GigaManz: RT @HowardvRooijen: Burnout higher in InfoSec than in first responders and military
  • @stack72: RT @jonathan_thorpe: The Phoenix Project is one of the most important books I have read. Business owners must read. @RealGeneKim
  • @stack72: It took until 1237 on the second day of #DevOpsDays to get a Deming quote. @botchagalupe would not be happy
  • @GigaManz: RT @HowardvRooijen: Burnout higher in InfoSec than in first responders and military
  • @OortIO: RT @markrendle: FYI, deploying https://t.co/fA2KpGbpsT to 6 datacenters is a 6 line bash script. #devopsdays :-P
  • @fmo1973: Great talk from @RealGeneKim gutted I didn't a copy of his book yesterday :-(
  • @actionjack: RT @adamhuffman: #devops as modern incarnation of Anarcho-Syndicalism, according to @thesamoth #DevOpsDays
  • @markrendle: FYI, deploying https://t.co/fA2KpGbpsT to 6 datacenters is a 6 line bash script. #devopsdays :-P
  • @gdb_: RT @garethr: OH: "cynicism is our core competency"
  • @RealGeneKim with a fun but important/useful talk at
  • @rakh1: Some of the best DevOps guys come from an architecture background. #DevOpsDays
  • @jonthan: Looks like something I'll have to read! #ThePhoenixProject #devopsdays ~ http://t.co/BKYKRxEDrW http://t.co/wR4X8DFL68
  • @KrisBuytaert: The people in this room are the 1% , we live in a paralellel fantasy world of how things should be . @realgenekim
  • @daveavent: RT @SamNixJobs: Loving this talk from Sam Eaton - It was only a matter of time before @DEVOPS_BORAT showed up... #devopsdays http://t.co/AfFTMqdP62
  • @jonathan_thorpe: The Phoenix Project is one of the most important books I have read. Business owners must read. @RealGeneKim
  • @alister_b: RT @garethr: OH: "cynicism is our core competency" @RealGeneKim with a fun but important/useful talk at
  • @lusis: "Cynicism is our core competency" @RealGeneKim
  • @BMC_DevOps: RT @jamesbetteley: "It's not an Operations problem, it's a business problem" Gene Kim on "ahaa moments"
  • @lusis: "Cynicism is our core competency" @RealGeneKim
  • @scoobiedoobie: RT @jamesbetteley: "It's not an Operations problem, it's a business problem" Gene Kim on "ahaa moments"
  • @ElMundio87: RT @jamesbetteley: "It's not an Operations problem, it's a business problem" Gene Kim on "ahaa moments"
  • @KrisBuytaert: RT @garethr: OH: "cynicism is our core competency" @RealGeneKim with a fun but important/useful talk at
  • @BoobBoo: 3 signs of burn out - 1. Fatigue, 2. Cynicism, 3. Illusion of self efficacy - sounds like most consultants I know #devopsdays #devops
  • @unixdaemon: RT @ripienaar: @TheSamoth that was an awesome #devopsdays talk
  • @paradoxbound: @RealGeneKim the Visible Ops book has helped me to change company culture. Your talk @ #devopsdays shows you aren't a 1 hit wonder. ;)
  • @BMC_DevOps: How to get buy in and DevOps pitch to infosec http://t.co/PfI95aPnD7 #devopsdays @RealGeneKim
  • @turowicz: RT @rposbo: "Show me a developer who isn't causing an outage, I'll show you a developer who's on holiday" Thanks, @realgenekim..
  • @HowardvRooijen: The Phoenix Project: A Novel about IT, DevOps and helping Your Business Win" http://t.co/os9Ltt7M1N
  • @HowardvRooijen: The Phoenix Project: A Novel about IT, DevOps and helping Your Business Win" http://t.co/os9Ltt7M1N
  • @jonathan_thorpe: Awesome, @RealGeneKim set me up for the blame if his session sucks! Do I get credit if it is awesome ;)
  • @thorrsson: RT @BoobBoo: BOFH is supposed to be comedy not a manual for human interaction
  • @rjw1: RT @MatStace: “Snowflake” #TakeADrink #DevOpsDays
  • @ddelmoli: RT @rposbo: "Show me a developer who isn't causing an outage, I'll show you a developer who's on holiday" Thanks, @realgenekim..
  • @HowardvRooijen: Gene has posted the slides for this talk on slideshhare: http://t.co/CXBWf4AAOB really good slide on the downward spiral
  • @BMC_DevOps: RT @chrisflink: "How many people recognize any of these stories?" - @RealGeneKim => everybody raises hands #DevOpsDays :-)
  • @HowardvRooijen: "small changes can have awesome outcomes"
  • @matthewpskelton: RT @withnale_: #devopsdays gene kim brings the lolz.. show me a dev who doesnt cause an outage i'll show one on vacation
  • @BMC_DevOps: RT @HowardvRooijen: DevOps is about trying to crack the downward spiral to failure
  • @HowardvRooijen: If you're firefighting concerns like InfoSec can hardly be considered as there are so few free cycles to deal with it
  • @fmo1973: The talk from Payback was really great too, I could really relate to it
  • @HowardvRooijen: Deployments get bloated - goes from slick, fast process to taking hours, days to complete
  • @garethr: RT @adamhuffman: #devops as modern incarnation of Anarcho-Syndicalism, according to @thesamoth #DevOpsDays
  • @chrisflink: Looking forward to this tak by @realgenekim #DevOpsDays http://t.co/rucMhtxRgU
  • @al_maisan: ops view on devs :) #devopsdays http://t.co/xJbpRzbUTs
  • @fmo1973: Couldn't agree more! "@ripienaar: @TheSamoth that was an awesome #devopsdays talk" #DevOpsDays http://t.co/sUQOE7hJlb
  • @HowardvRooijen: Product Managers focus on features but don't know what technology do. Developers focus on delivery which incurs technical debt

*@BoobBoo: RT @marygrace: in this case, these aren't 7 Deadly Sins, but _Useful. it pays to have envious, gluttonness ppl on team.

Ignite talks