DevOps for Dinosaurs
Niek Bartholomeus
Buy before build
Heterogeneous enevironments
version control, release manaement, testing
dinosaur not likeamodern enterprises
first create config library, then fully documenting the release instructions
Oh, no! Why didn't they automate deployments? infosec said 'no.' Something about app privilege levels
Roadmap
automate deployments and environment creation; increase frequency of releases; continuous delivery (maybe)
Checking DevOps’ vital signs
John Clapham & Paul Swartout
@Harm B: #devopsdays Very nice talk on measuring devops culture @johnC bristol
@Squire Matt: Getting enthused about understanding how healthy our DevOps culture is. Great talk by @johnC bristol at #DevOpsDays @mary grace: listening to @johnC bristol at #DevOpsDays London. talking #devops #culture , and how to check the vital… http://t.co/BycrGtGqHC @mark_mann : #Monitoring culture at #devopsdays - McKinsey Org Health? #agile /scrum checklists? No.. We use Net Promoter Scores #NPS @PierG : RT @themetriost : culture telemetry from nokia at@freistil : #devopsdays Interesting talk by John Clapham about cultural vital signs in IT organizations. ^jl@matthewpskelton : Great example of ECG machine (heart monitor) interface as a UI pattern for monitoring culture from @johnC_bristol
StartOps: Growing an ops team from 1 founder:
David Mytton
"Bootstrapping is opposite of raising money: means lots of deferring to last minute. There was only 2 of us; 1 for ops"
"First beta was 4/2009 on Slicehost: surprise: people used our svc, set up 2nd server just in case"
"Switched to mongodb: 50GB/month: even low volumes, had problems with drivers; easy to deploy"
"Taking shortcuts & deferring work: resizing disks meant hours of downtime" (showing 'df' output: / at 100%.)" Hahaha.
"Finding quickest way to do things: b/c scarce time/$$; network copy too slow; soln? UPS truck. Bandwidth = 11.22MB/sec" :)
"Even traveling was expensive; especially roaming"
"Don't update while traveling"
"Let other people help: Rackspace staff helped manage hosts, reboot, etc, when I was unable to do work, in car, etc"
"Engrs often bad at valuing their own time; outsource things like service list access, PCI certification"
(infosec people are fainting in aisles here :)
@milosgajdos : David Mytton from @serverdensity interesting talk on growing ops teamRT @milosgajdos : David Mytton from @serverdensity interesting talk on growing ops team
"Dealing w/people: automate as possible; ensure people only do things once: automating config mgmt has huge payoff"
"Siloed information is bad; Docs must be up to date; doc failure scenarios, vendor contacts, team contacts
"2010: staff grows from 3 to 12; we finally have on call rotations"
"On call: first line: 1 ops engr; second line: all engineers; third line: ops engineers"
"On call CEO: i get push notifications of all incidents and all outages; weekly internal report on uptime; we use google+, enables team discussions"
"Citing Amazon, Heroku as exemplars of public discussion of downtime issues"
"Outages test your vendor escalation process; citing problems w/Terremark 1st line support during 2am outage"
"When hosting providers have large scale outages, they're impossible to get ahold of: Rackspace phone systems broke"
"Must simulate failures: can systems restart, can failovers handle load?"
"Ideally, you want your own team, unlike vendors, outsourcers, your own team cares the most"
"We use pagerduty, bamboo, travis, papertrail, puppet; atlassian confluence, g+, hipchat, jira (kanban board), github"
@BMC_DevOps : Gene Kim and Patrick Debois at #devopsdays London @RealGeneKim @patrickdebois http://t.co/uDVHuCw2S9
Adding Business Metrics
Deri Jones
"No further competitive advantage w/Google AdWords; enter Facebook and social network"
Citing @lordcope : "why are we not monitoring business metrics alongside technical metrics?"
"What data does business want? sales metrics, but we don't want to dupe data they already have" (a good point)
"Example: lost sales is an easy one: plot delta of sales numbers vs. historical averages"
Multiply by your outages: in statsd/collector or in graphs
"Experience from 40 orgs: most biz teams don't know hour by hour typical sales; may not want to tell tech folks sales nmbers
"Trying out 'unbuyable products': customers can be found, but not bought: greyed out, empty shopping basket
"Reasons for this was biz trying to understand revenue misses due to web team or merchandising team"
@BMC DevOps: RT @ HarmB : #devopsdays use metrics to calculate lost sales. @scivisum_UK http://t.co/tG22QR5JF8 @Harm B: #devopsdays use metrics to calculate lost sales. @scivisum UK http://t.co/tG22QR5JF8
@renierbotha : Quoting Deri Jones:"... unreliable websites is a repetitional risk..." It can be a reputation risk as well?! @DevOpsGuys : Can't believe how many interpretations of "DevOps" I'm seeing.Used in diff context repeatedly! http://t.co/ez6x7c1lR9 @#DevOpsDays #devops
Vendor pitches
there's something really nice about these vendor pitches: treats us as intelligent humans and fellow practitioners
Enstratus, Riak
Ignite talks
DevOps In A Traditional Enterprise
Up: "DevOps In A Traditional Enterprise": Thomas Falkenberg
"Payback is a customer loyalty program, one of the most popular brands in Germany"
"pre-2007: picture book waterfall: long release cycles; 2007: Agile but still long releases due to scrumfall"
"2011: first steps to continuous delivery and DevOps: inspired by DevOpsDays; difficulties encountered in enterprise"
"issues diff from startups: multidimensional env; complex tech stack; differnt depts; multiple VMs/datacenters"
@mary_grace : RT @bruntonspall : Interesting term. "Devops promoter". Someone who promotes devops within the organisation.Nice! They're boundary spanners. RT @mary_grace /@bruntonspall : "Devops promoter", who promotes devops within the org
"Cause/effect diagrams: countermeasures: ops joins dev standups; config in svn; no manual chgs allowed; private repo for bins
"repo incl binaries: httpd2, tomcat, weblogic, etc; qa/dev does deplys for ops; fully automated deploys"
"tools: everyone loves tools, but don't solve problems; processes do; common tools across org & lifecycle ease communication
"tools: confluence, jira, graphite, jenkins, deployment tools
"tools sound easy but in practice, took much longer to get running; graphite took 2 days to get working b/c dependencies
"now tracking 14K metrics:
@rposbo : Biggest takeaway from #devopsdays so far; everyone should learn to use #logstash and #graphite well@rposbo : Biggest takeaway from #devopsdays so far; everyone should learn to use #logstash and #graphite wellRT @rposbo : Biggest takeaway from #devopsdays so far; everyone should learn to use #logstash and #graphite well
"Before: deploys used to take 8 hours of downtime; now we're down to 1 hour, reproducible, automated, no snowflakes"
@matthewpskelton : RT @stack72 : Good to hear that payback use logstash in qa as well as production to make sure logfiles are used in qa"shared goals are important: dev cant' be measured by story points/features delivered; needs dev/qa/ops to work together
"devops requires patience: we started ~2 years ago; we feel we're at 50% complete; no one knows what 100% looks like"
@al_maisan : Interesting terms picked up at #devopsdays : snowflake server, scrumfall :)@scoobiedoobie : RT @stack72 : Good to hear that payback use logstash in qa as well as production to make sure logfiles are used in qa@rakh1 : There it is - snowflake server - everyone drink! #DevOpsDays London@tomduckering : #devopsdays "Jenkins as a SPOF" - make sure your local build is the same as in CI so be careful what plugins you rely on.@rakh1 : EVERYONE is hiring, common theme.
How Can We Sell DevOps
Gene Kim
Q: "how did you get infosec onboard?" A: "create trust; took away root access for deploys; took away internet access"
@kief : Some popular topics at this #DevOpsDays : Security, #BigCompanyDevOps , #InfrAsCode . Less monitoring-heavy than Rome"I was a perf engineer, always stuck between Dev & Ops; ops has much less pain in their daily work"
@stack72 : RT @jonathan_thorpe : The Phoenix Project is one of the most important books I have read. Business owners must read. @RealGeneKim @stack72 : It took until 1237 on the second day of #DevOpsDays to get a Deming quote. @botchagalupe would not be happy@GigaManz : RT @HowardvRooijen : Burnout higher in InfoSec than in first responders and military@stack72 : RT @jonathan_thorpe : The Phoenix Project is one of the most important books I have read. Business owners must read. @RealGeneKim @stack72 : It took until 1237 on the second day of #DevOpsDays to get a Deming quote. @botchagalupe would not be happy@GigaManz : RT @HowardvRooijen : Burnout higher in InfoSec than in first responders and military@OortIO : RT @markrendle : FYI, deploying https://t.co/fA2KpGbpsT to 6 datacenters is a 6 line bash script. #devopsdays :-P@fmo1973 : Great talk from @RealGeneKim gutted I didn't a copy of his book yesterday :-( @actionjack : RT @adamhuffman : #devops as modern incarnation of Anarcho-Syndicalism, according to @thesamoth #DevOpsDays @markrendle : FYI, deploying https://t.co/fA2KpGbpsT to 6 datacenters is a 6 line bash script. #devopsdays :-P@gdb_ : RT @garethr : OH: "cynicism is our core competency" @RealGeneKim with a fun but important/useful talk at@rakh1 : Some of the best DevOps guys come from an architecture background. #DevOpsDays @jon than : Looks like something I'll have to read! #ThePhoenixProject #devopsdays ~ http://t.co/BKYKRxEDrW http://t.co/wR4X8DFL68 @KrisBuytaert : The people in this room are the 1% , we live in a paralellel fantasy world of how things should be . @realgenekim @daveavent : RT @SamNixJobs : Loving this talk from Sam Eaton - It was only a matter of time before @DEVOPS_BORAT showed up... #devopsdays http://t.co/AfFTMqdP62 @jonathan_thorpe : The Phoenix Project is one of the most important books I have read. Business owners must read. @RealGeneKim @alister_b : RT @garethr : OH: "cynicism is our core competency" @RealGeneKim with a fun but important/useful talk at@lusis : "Cynicism is our core competency" @RealGeneKim @BMC_DevOps : RT @jamesbetteley : "It's not an Operations problem, it's a business problem" Gene Kim on "ahaa moments"@lusis : "Cynicism is our core competency" @RealGeneKim @scoobiedoobie : RT @jamesbetteley : "It's not an Operations problem, it's a business problem" Gene Kim on "ahaa moments"@ElMundio87 : RT @jamesbetteley : "It's not an Operations problem, it's a business problem" Gene Kim on "ahaa moments"@KrisBuytaert : RT @garethr : OH: "cynicism is our core competency" @RealGeneKim with a fun but important/useful talk at@BoobBoo : 3 signs of burn out - 1. Fatigue, 2. Cynicism, 3. Illusion of self efficacy - sounds like most consultants I know #devopsdays #devops @unixdaemon : RT @ripienaar : @TheSamoth that was an awesome #devopsdays talk@paradoxbound : @RealGeneKim the Visible Ops book has helped me to change company culture. Your talk @ #devopsdays shows you aren't a 1 hit wonder. ;)@BMC_DevOps : How to get buy in and DevOps pitch to infosec http://t.co/PfI95aPnD7 #devopsdays @RealGeneKim @turowicz : RT @rposbo : "Show me a developer who isn't causing an outage, I'll show you a developer who's on holiday" Thanks, @realgenekim ..@HowardvRooijen : The Phoenix Project: A Novel about IT, DevOps and helping Your Business Win" http://t.co/os9Ltt7M1N @HowardvRooijen : The Phoenix Project: A Novel about IT, DevOps and helping Your Business Win" http://t.co/os9Ltt7M1N @jonathan_thorpe : Awesome, @RealGeneKim set me up for the blame if his session sucks! Do I get credit if it is awesome ;)@thorrsson : RT @BoobBoo : BOFH is supposed to be comedy not a manual for human interaction@rjw1 : RT @MatStace : “Snowflake” #TakeADrink #DevOpsDays @ddelmoli : RT @rposbo : "Show me a developer who isn't causing an outage, I'll show you a developer who's on holiday" Thanks, @realgenekim ..@HowardvRooijen : Gene has posted the slides for this talk on slideshhare: http://t.co/CXBWf4AAOB really good slide on the downward spiral@BMC_DevOps : RT @chrisflink : "How many people recognize any of these stories?" - @RealGeneKim => everybody raises hands #DevOpsDays :-)@HowardvRooijen : "small changes can have awesome outcomes"@matthewpskelton : RT @withnale_ : #devopsdays gene kim brings the lolz.. show me a dev who doesnt cause an outage i'll show one on vacation@BMC_DevOps : RT @HowardvRooijen : DevOps is about trying to crack the downward spiral to failure@HowardvRooijen : If you're firefighting concerns like InfoSec can hardly be considered as there are so few free cycles to deal with it@fmo1973 : The talk from Payback was really great too, I could really relate to it@HowardvRooijen : Deployments get bloated - goes from slick, fast process to taking hours, days to complete@garethr : RT @adamhuffman : #devops as modern incarnation of Anarcho-Syndicalism, according to @thesamoth #DevOpsDays @chrisflink : Looking forward to this tak by @realgenekim #DevOpsDays http://t.co/rucMhtxRgU @al_maisan : ops view on devs :) #devopsdays http://t.co/xJbpRzbUTs @fmo1973 : Couldn't agree more! "@ripienaar : @TheSamoth that was an awesome #devopsdays talk"
#DevOpsDays http://t.co/sUQOE7hJlb @HowardvRooijen : Product Managers focus on features but don't know what technology do. Developers focus on delivery which incurs technical debt
*@BoobBoo : RT @mary grace: in this case, these aren't 7 Deadly Sins, but _Useful . it pays to have envious, gluttonness ppl on team.
Ignite talks
Please enable JavaScript to view the comments powered by Disqus.