2013/03/31 DevOpsDays Austin Day 1

by Gene Kim on

#devopsdays

Up: John Willis: The State of DevOps: (@botchagalupe)

  • .@botchagalupe: "Drucker: 'Culture eats strategy'"
  • .@botchagalupe repeating one of funniest lines I've heard; "Culture is dead. I'm taking the 'C' out of 'CAMS'" (haha)
  • .@botchagalupe: "Culture helps you win the hiring war. How many people here are hiring?" (30% of hands went up. Wow)
  • .@botchagalupe: "The greatest culture book ever written: Gerstner: 'Who Says Elephants Can't Dance?" on IBM turnaround
  • .@botchagalupe: "3 orgs that seem to nail culture: Netflix, Etsy, Github" (can't help but how smart these ppl are!)
  • .@botchagalupe: "From Netflix culture: 'adequate performance gets you a generous severance package'" (haha!)
  • .@botchagalupe: "Github culture: no managers at github -- everyone is a manager; structure of github is unstructure"
  • .@botchagalupe: "Most of you will agree that DevOps is probably one of most interesting things happening in our industry;
  • .@botchagalupe: "Github could be greatest social experiment going on right now; counterintuitive & seems to working"
  • .@botchagalupe: "Who's using github now? law firms, realtors; allows mass collaboration, not just source code anymore
  • .@botchagalupe: "Etsy: code as craft; enable ppl to make living making things; customers make weird lampshades, they're making weird software
  • .@botchagalupe: "If you're doing something interesting in culture, tell me about it. I want to talk with you"
  • @tboubez: #devopsdays @botchagalupe having spent many years at IBM starting in 94, I can attest to the effects of culture!! Amazing turnaround!

  • 2 min vendor pitches

    • Thx to @bmcdevops!!! I'll be signing copies of Phoenix Project at lunch today! See you there! (There's 100 copies)@scottsbaldwin: Thanks @BMCDevOps and @puppetlabs for helping sponsor #devopsdays!
  • Sponsors: 2 minute pitches

    • Zero Turnaround: jrebel reduces turnaround time of code deploys, to make java developers happy; liverebel helps make everyone happy, for jenkins/bamboo/etc into prod
    • UC4: company is 25 years old

Up: Pete Cheslock, Dyn: How to retain your hardest to hire positions (DevOps, System Automation, Release Engineering)

  • .@petecheslock: "I'm Director of DevTools at Dyn; I don't like titles; I build tools for Dev, automation engineer, etc."
  • .@petecheslock: "I like making Dev happy"
  • .@petecheslock: "Retention is difficult when you're not Etsy/Netflix/Github, esp for smaller players"
  • .@petecheslock: "Losing people sucks; it's even more painful when you realize you could have avoided it"
  • @realpdm: finding the deeper meaning of #devops at #devopsdays Austin http://t.co/akKxovAINm
  • @robynbergeron: Watching @petecheslock at #devopsdays - "director of get shit done" ... excellent.
  • .@petecheslock: "As manager: 2 most important things: recruitment & retention" (exactly mission of higher education too)
  • .@petecheslock: "Hiring and interviewing sucks; it takes so muchtime"
  • .@petecheslock: "Why is hiring diff? 2 wks notice; 2 wks "intro to company"; 3 mo rampup; Total: 6 months"
  • (Blushing a bit as @petecheslock does h/t to #phoenixproject. Golly.)
  • .@petecheslock: "Hour lost on bottleneck is hr lost in entire system hour gained on non-bottleneck is illusion"
  • .@petecheslock: "What happens if Brent quits?" (Utter, total disaster; no redundant keymasters. Hahaha)
  • .@petecheslock: "Impact: project slows down; days get longer; more stress; more people leave" (nice)
  • @calebgroom: Yes. Read The Goal. Awesome business fiction to understand constraints.
  • @byron_miller: #devopsdays book #3 The phoenix project
  • .@petecheslock: "Money/pay package: STPR: the 'shit to pay ratio'" haha
  • .@petecheslock: citing Dan Pink: "Drive: The Surprising Truth Of What Motivates Us:" Autonomy, Mastery, Purpose
  • .@petecheslock: "I tell my daughter what to do; I don't tell my engineers what to do"
  • @wickett: The root of the problem is not money but you should pay market or slightly above market to avoid making ppl swap because of $
  • @wickett: The root of the problem is not money but you should pay market or slightly above market to avoid making ppl swap because of $
  • @dwinter33: RT @robynbergeron: .@petecheslock "pay people enough so they can do their job - monetary adjustments are just short term fixes"
  • .@petecheslock: "On work assignment: align work w/owner, back off & let them work; offer support; filter distraction
  • .@petecheslock: "Often my role is 'shit filter' to protect engr from upper mgmt, or play goalie between Dev & Ops"
  • @austinmills: "The Shit-to-Pay Ratio: How much you have to be paid in order to put up with the annoying parts of your job." @petecheslock
  • RT @wickett: The root of the problem is not money: pay market or slightly above market to avoid making ppl swap because of $
  • RT @austinmills: "The Shit-to-Pay Ratio: How much you have to be paid in order to put up w/annoying parts of your job
  • .@petecheslock: "Anti-definition of Purpose:'I want to spend all my time writing code that no one will ever use'"
  • .@petecheslock: "Story: every evening at 7pm, Nagios would dump alerts to all developers; 1 day, someone says, 'I think there's a better way'
  • .@petecheslock: "1 week to POC; 1 month to MVP; example of Hate Driven Development"
  • .@petecheslock: "Told CEO: great idea, I want to open source it; replaced Nagios within 2 weeks; most satisfying point in my career"
  • .@petecheslock: "after 1 mo skunk project, replaced Nagios within 2 wks; most satisfying point in my career; birth of Sensu"
  • .@petecheslock: "I couldn't keep awesome folks w/o enabling them to create/work on awesome OSS projs like Sensu"
  • .@petecheslock: "Remote employees; only way to get the talent you need these days"
  • .@petecheslock: "Unscheduled time: time to go off roadmap, some of our best features came from here" (slack, hackdays, etc)
  • @byron_miller: RT @filler: “You want to scale in this economy? Talent first, location second.” @petecheslock
  • .@petecheslock: "Challenging, meaningful work; employee ownership; high levels of trust"
  • .@petecheslock: "Distilled even further: treat everyone like adults"
  • @petecheslock: Here is the slidedeck my #DevOpsDays Austin talk. "How to Keep the People You Need" http://t.co/VTAvhf9mnd.
  • @jzb: RT @zehicle: RT @petecheslock: #DevOpsDays Austin talk. "How to Keep the People You Need" http://t.co/LtFhDsspE2. < plugs Project Phoenix & the Goal

Up: Lee Thompson, Its not me, its you - your dysfunctional relationship with script code

  • .@stagr_lee: "10 yrs of industrial automation and control; I was guy at eTrade during 1998; went from 50K trades to ???
  • .@stagr_lee: "Increasing trend: common repos shared between Dev & Ops; diff toolchains cause communication probs
  • .@stagr_lee: "When both sides can see the same thing, you start to change the culture"
  • .@stagr_lee: "Most of us are building something like a Service Delivery Platform (SDP): but can take yrs to build
  • .@stagr_lee: "Unbudgeted improvement projects can take years; so, use MVP-style to create small improvements
  • .@stagr_lee: "Who uses bash in their infrastructure?" (almost everyone) "Umm, puppet/chef?" (60%)
  • @filler: “It really helps your culture when your teams can see the same thing, have visibility.”
  • .@stagr_lee: "Typical languages/syntaxes used orgs (eg, bash, ruby, puppet/chef, C++, etc)? 45. That's a problem"
  • .@stagr_lee: "It's diff to be proficient in 45 languages: can u understand, test, correct & release someone else's code
  • @robynbergeron: .@stagr_lee syntax proficiency - i can understand, test, fix, deploy someone else's code
  • @filler: “Ok, so who doesn’t use bash in their infrastructure?” lone hand goes up “Powershell?” #devopsdays (haha)
  • .@stagr_lee: "Good architecture eliminates complexity"
  • .@stagr_lee: "Bash is bad; not object-oriented, not modular, rarely tested"
  • .@stagr_lee is demoing rerun: http://dev2ops.org/2012/12/rerun-making-shell-scripts-even-more-useful-and-a-bit-cool-again/
  • .@stagr_lee: "Training everyone on ruby is great; but Rerun may be easier, more MVP, b/c it reduces complexity
  • .@stagr_lee: "You can embed rerun into packages, like Tomcat"
  • .@stagr_lee: "Rerun is very similar to powershell; easy to go from rerun to powershell and back"
  • (I got powershell demo from @stack72 last week. Blew my mind. It's come a long way. Can even impl DSLs!)

Up: Chris McClimans: Sharing a Culture of Automation - Instant Infrastructure in Developing Nations

  • @Brett2point0: Phoenix Project fans: do you work w/ a Bill? Know a Chris? Have colleagues who remind you of Patty? John? Erik? cc: @RealGeneKim
  • @RealGeneKim: Which one are you? http://t.co/Vhpse5lMeR@Brett2point0: Phoenix Project fans: are you Bill, Chris, Patty, or John? @petecheslock: Wow - Reading Rainbow, Bob Ross, Julia Child and Mr. Rogers in an amazing talk by @hippiehacker at #Devopsdays
  • @filler: “knife create hospital” @hippiehacker
  • @hunter_jared: .@hippiehacker "It's ok to ask for help and it's ok to make mistakes" Still great insight when things don't as planned.
  • @petecheslock: Wow - you can feel the passion from @hippiehacker giving his talk at #DevOpsDays So much heart and positive energy.
  • @newgoliath: Weirdos Outperform Norms http://t.co/7PWXeiq4NN #hippiehacker "if you're not doing it out of love, you're just making noise."
  • @petecheslock: “Code can do some amazing things, but you need to use your imagination”. @hippiehacker at #DevOpsDays
  • Wow. Showing Mr. Rogers testifying before skeptical Congress, lobbying for continuing PBS during Nixon years.
  • Mr. Rogers: "Lots of scary things in the world; My mom told me to look for the people in the world who are willing to help"
  • "It will take chefs and puppets"
  • @barton808: @hippiehacker connecting dots and showing grace under pressure amid tech difficulties #devopsdays http://t.co/25yJJH775H
  • @dwinter33: just had to explain to wife why I had a still shot of #mrrogers on my TV in office.
  • "Give 5% of your time to something you're passionate about"

Up: Dominica DeGrandis, Who is the Enemy - Us or Motionlessness?

  • .@domincad: "Story about smartest, most thoughtful build engineer being accosted by mgr (CTO?), told, 'stop goofing off!'
  • .@domincad: "Invisible work is like dark matter:
  • (Oh, no. This is going to hurt; @dominicad talking about harms of WIP, work sitting idle, bad multitasking... Argh...)
  • .@domincad talking about managing tasks, PMO inserting contingency buffers, how it opens up doors for new invisible work.
  • .@domincad: "Contingency buffers make things worse: workers in highest demand take longer to complete needed work"
  • .@domincad: "Knowledge work is perishable; delays cause people to pester, 'are you done? are you done yet?"
  • .@domincad: On hospitals and bad queues: "They don't call it a waiting room for nothing."
  • .@domincad: "How do we convince everyone for the need for slack time? Managers, customers, etc." (Wait time kills)
  • .@domincad: "Single largest predictor of queue time is capacity utilization" (Love this.)
    https://pbs.twimg.com/media/BJHXSFrCQAA7hAb.jpg
  • @petecheslock: Brent is the most popular person at #devopsdays Austin. For good reason! /cc @RealGeneKim
  • @jonathan_thorpe: Feel sorry for that Brent guy! He works so hard but is a key part of the problem @RealGeneKim
  • .@domincad showing when arrival rate significantly lagging completion rate
  • .@domincad showing red flag: arrival rate significantly lagging completion rate via Cumulative Flow Diagram #devopsdays
  • .@domincad: "@zehicle: RT @petecheslock: I want more people to read "The Goal" so I can make Herbie references.
  • @jonathan_thorpe: 100% capacity utilization doesn't work. Bad for employees, morale, productivity b/c the unexpected happens with no buffer
  • (I love listening to @dominicad talk about flow and restricting WIP)
  • .@domincad: "As much as kanban-lovers love Pull, Ops is often Push; outages can't wait"
  • .@domincad: "1) kill zombie projects to limit # of projs - can't kill it? at least push to backlog, to clear work pipeline
  • .@domincad: "2) allocate capacity for work categories (43% of work is expedited? create Expedite class for 43% of work)
  • .@domincad: "4) create aging pipeline: if it's stuck in WIP for 90 days, it's probably not that important
  • .@domincad: "5) find regular cadence: reduces transactional overhead (eg, finding conference rooms, etc)
  • .@dominicad: "6) create explicit policies for escalations (to mitigate using political pressure, reduce thrash)
  • (Oh crap. I've been misspelling @dominicad's twitter handle. Sorry!)
  • .@dominicad: "@zehicle: I recommend considering #kanban as an alternative for #agile. Both #lean! Avoids wagilefall.
  • @jheintz: @dominicad talk at #devopsdays reminded me of Deming quote "Don't just do something, stand there"
  • @RealGeneKim: .@dominicad shows red flag: arrival rate significantly lagging completion rate in Cumulative Flow Diagram #devopsdays http://t.co/a7fPfSCffI
  • @hunter_jared: Taking home ageing reports! Great way to bring visibility into those slow tasks on our wall. Great talk @dominicad
  • @dominicad: Slide deck from my talk at #devopsdays up at http://t.co/VgAhfpqU5Q #flow
  • @dominicad: Aack! Forgot major point at #devopsdays talk - "We don't let our servers get to 100% capacity utilization. Why do we let our people?"
  • .@ernestmueller: "If you don't like the Ignite talk, just hang in there. It'll be over in less than 5 minutes." (Haha!)

James Wickett (Ignite): Be mean to your code with Gauntlt

  • .@wickett: "we built gauntlt: generic garmr, dirb, sslyze, curl, etc..." (to be mean to your code)
  • .@wickett: "everyone type 'gem install gauntlt'": github/gauntlt
  • .@wickett: showing how gauntlt utilizes gurken to test security assertions, just like unit tests. nice)

Ernest Mueller (Ignite): Speeding Up Releases (how we reduced release cycle time from 10w to 1w)

  • .@ernestmueller: "Dev went Agile, tried it, and things went bad. very bad. Ernest, get us to bimonthly releases. Go!
  • .@ernestmueller: "Bad habits: checkins right before release; no testing; took 3 weeks off; stopped 5 year long test cycle
  • .@ernestmueller: "switched to trunk/branch: close branches within 2 weeks; moved to features to do dark launches
  • .@ernestmueller: "we couldn't reindex SOLR index in 2 hours; did our launch on 3/1; managers said 'no go'. right decision
  • .@ernestmueller: "released on 3/6: 5 days late, 2 cust issues; kept reducing lateness and # of issues
  • .@ernestmueller: "summer was ok; added one more control point; went to weekly releaes in 2 weeks
  • .@ernestmueller: "9/27: avged 0 cust impacting issues on weekly releases
  • .@ernestmueller: "our goal for 2013: continuous delivery! read more at AgileAdmin blog"
  • (Whew. I loved that @ernestmueller talk!)

Nick Silkey (Ignite): Level Up: From Ops to Engineers

  • Nick: "if release code is difficult, you will delay releasing"
  • Nick: "give Dev prod parity; they do want a copy of Prod outside of production!)" (Nice)
  • Nick: "sufficiently tested & accurate releases creates trust"
  • Nick: "identify systems: automate 1 simple function well"
  • Nick: "rebuild systems well"

Paul Voccio (Ignite): Deploying 10,000 nodes simultaneously (@paulvx)

  • .@paulvx: "Haiku: writing code is hard; if you can't deploy it; it doesn't matter" (haha)
  • .@paulvx: "Rackspace: largest OpenStack infrastructure:
  • .@paulvx: "not deploy behind load balancer; we do puppet deploys
  • .@paulvx: "Packaging is not OS indepndent; desinging for 10x currrent scale
  • .@paulvx: "everyone should be able to deploy; admins, mgmt, everyone!"
  • .@paulvx: "we use jenkins"
  • @filler: “Writing code is hard. If you cannot deploy it. It’s not production.” @paulvx’s awesome team haiku
  • @RyanVanderwerf: Very cool, @rackspace uses bit torrents to help deploy infrastructure to avoid network saturation.
  • @NGalbreath: haiku by @paulvx of @Rackspace "writing code is hard; if you cannot deploy; it does not matter" #devopsdays http://t.co/B72cRFmAQV

Ben Whaley (Ignite): Monitor the customer - Feed the business

  • Often customer and business not represented
  • Whaley: "you want to know when customer signups exceeding demand; you don't want to find out via twitter" (haha)
  • @NicholasMistry: Notifications: "if you are hearing about it on twitter, you are probably doing it wrong"
  • @zznate: Representing @Apigee with 'monitor the customer' by @iAmTheWhaley at #devopsdays http://t.co/VO7Qj2E8DK
  • @zznate: Representing @Apigee with 'monitor the customer' by @iAmTheWhaley at #devopsdays http://t.co/VO7Qj2E8DK
  • @dwinter33: RT @filler: “If you track, store, graph metrics but don’t act, it’s as if you never collected the metrics at all.” @iAmTheWhaley
  • @dwinter33: RT @filler: “If you track, store, graph metrics but don’t act, it’s as if you never collected the metrics at all.” @iAmTheWhaley

Up: Matthew Hooker (Ignite): ZooKeeper discovery with Exhibitor

  • Hooker: "On chef 10: it uses solr; impedance mismatch: we want key/value stores, not search
  • @blkintechnology: RT @wickett: My slides from DevOps Days Austin http://t.co/08h4RvnXu0

Up: Cindy VanEpps (Ignite): The DevOps Dunking Booth: Continuous Consumption

  • VanEpps: "Improving the cycle time of oops and loops doesn't always prevent accidents"
  • VanEpps: "architect for consumablility; feature for consumability; assess capacity to consume;
  • VanEpps: "Must build features that are worth the feature release pain for users"
  • VanEpps: "
  • @mwhooker: Cindy VanEpps is doing an awesome spoken word ignite at
  • @wickett: RT @mwhooker: Cindy VanEpps is doing an awesome spoken word ignite at
  • VanEpps: "We now release to testing every 2 wks to UAT; every 6 wks to users" (Yes, decouple deploy from release!)
  • @filler: “Make change flexible. Use change champions.”
  • VanEpps: "We want to talk more about the consumption side of continuous delivery"

  • All, I'll be signing Phoenix Project books at BMC table! 100 free books to first people there. Thx! cc @bmc_devops

Misc

  • RT @tboubez: #devopsdays @botchagalupe Spent many years at IBM starting in 94, I can attest to effects of culture! Amazing turnaround!
  • RT @scottsbaldwin: Thanks @BMC_DevOps and @puppetlabs for helping sponsor #devopsdays!
  • @petecheslock Loved it, Pete! Thx for shout out -- let's grab a drink tonight?
  • Nice! RT @petecheslock: Slidedeck for my #DevOpsDays Austin talk. "How to Keep the People You Need" http://t.co/VTAvhf9mnd.
  • @BMC_DevOps: #devopsdays attendees happy to be in Austin 2013 http://t.co/HBDnintMOS
  • RT @BMC_DevOps: #devopsdays attendees happy to be in Austin 2013 http://t.co/HBDnintMOS
  • @Brett2point0: Phoenix Project fans: do you work w/ a Bill? Know a Chris? Have colleagues who remind you of Patty? John? Erik? cc:
  • @adrianco: Thx you for agreeing to keynote FlowCon! Very excited!!@jezhumble & I want to create forum for elephants, not unicorns. :)

Haha. We've all been Brent. :) RT @petecheslock: Brent is most popular person at #devopsdays Austin. For good reason! /cc @RealGeneKim
Awesome! RT @dominicad: Slide deck from my talk at #devopsdays up at http://t.co/VgAhfpqU5Q #flow
RT @dominicad: Forgot major point at #devopsdays talk: 'We don't let our servers get to 100% capacity utilization. Why do we let our people?
RT @zznate: Representing @Apigee with 'monitor the customer' by @iAmTheWhaley at #devopsdays http://t.co/VO7Qj2E8DK
RT @filler: “If you track, store, graph metrics but don’t act, it’s as if you never collected the metrics at all.” @iAmTheWhaley
RT @wickett: My slides from DevOps Days Austin http://t.co/08h4RvnXu0
Yes! RT @mwhooker: Cindy VanEpps is doing an awesome spoken word ignite at