2012/9/27 PuppetConf Day 1

by Gene Kim on

#puppetconf

Luke

  • @nrcantor: P
  • upp
  • et
  • Forge has 500 modules. I'd be more impressed if it was 5
  • 00 work
  • ing-but-for-configur
  • ation modules.
  • @ahpook: 750 people at #puppetconf holy crap. Come see my talk on Puppet 3 at 2:45 in fisher ..
  • Agreed. I'm up in 40m. Hope those espressos didn't have caffiene... RT @ahpook: 750 ppl at #puppetconf holy crap. My talk: Puppet 3 @ 2:45
  • @PuppetConf: From @puppetmasterd 's keynote: RT @larsfronius: Devops "No-one knows what the hack that means"
  • @puppetlabs: RT @ShadowSoftNews: Luke Kanies (@puppetmasterd), CEO @puppetlabs kicks off #PuppetConf. "A Software-Defined Infrastructure is the future" http://t.co/HyvpxyR4
  • @thetomfox: "Managing with some Perl app written in 1999 and been applying spackle and duct tape to ever since." SOOO TRUE #puppetconf genius
  • @thetomfox: "Managing with some Perl app written in 1999 and been applying spackle and duct tape to ever since." SOOO TRUE #puppetconf genius
  • RT @thetomfox: "Managing with a Perl app written in 1999 and been applying spackle and duct tape to ever since." SOOO TRUE
  • Haha. Love it. @puppetmasterd: "I'd far rather you think I'm dumb than evil"
  • @dhgwilliam: RT @nrcantor: PuppetDB is one of the world's largest Clojure apps.
  • RT @dhgwilliam: RT @nrcantor: PuppetDB is one of the world's largest Clojure apps.
  • @0xEFF: "My favorite razor is Hanlon's razor." Puppet Labs Test Pilot program at #PuppetConf http://t.co/bbwy8V4a
  • RT @0xEFF: "My favorite razor is Hanlon's razor." "Never attribute to maliciousnes what can be attributed to stupidity"

Up next:
9:30 Keynote in Robertson – Paul Strong, VMware

  • Up next: Paul Strong, VMware: "I've managed VAX clusters": "Virtual servers to software defined data centers"
  • @paxindustria: watching #puppetconf live via http://t.co/5z79xmMs so far pretty awesome!
  • Live streaming links here: RT @paxindustria: watching #puppetconf live via http://t.co/5z79xmMs so far pretty awesome!
  • Paul Strong: "Virtualization layer becoming the new meta-OS layer: abstracts everything beneath it"@0xEFF: "We have to virtualized the network as well as the computer." #puppetconf http://t.co/kGlE2emj
  • RT @0xEFF: "We have to virtualize the network as well as the computer." #puppetconf http://t.co/kGlE2emj
  • Paul Strong: "The new container isn't the OS anymore: it's collected at the virtualization layer"

My keynote

  • @kentbye: #PuppetConf talk from @RealGeneKim was fast & furious. These notes will make more sense once he posts his slides: http://t.co/i2hkz0bo
  • @jeffersongirao: RT @beauchristensen: Great talks today, streamed live at #puppetconf. Thank you @puppetlabs for making this available to all of us http://t.co/boLshFth
  • @jason_southgate: RT @ahpook: Damn, @realgenekim throwing down the knowledge at #puppetconf ...
  • @kentbye: Liveblog notes from #PuppetConf Opening Keynote from Paul Strong, CTO of @VMware http://t.co/xVTagmVY
  • @unixgeekem: The Chaos Monkey. Dreaded or loved? Gene Kim speaks at #PuppetConf. http://t.co/AhNi2O9R
  • @SamuelBrownIV: Great presentation by @RealGeneKim at #PuppetConf. Very interesting and inspiring!
  • @SamuelBrownIV: Great presentation by @RealGeneKim at #PuppetConf. Very interesting and inspiring!
  • @joshroppo: @RealGeneKim Thanks for the great #puppetconf talk! Any chance your slides will be posted online?
  • @unixgeekem: The Chaos Monkey. Dreaded or loved? Gene Kim speaks at #PuppetConf. http://t.co/AhNi2O9R
  • @SamuelBrownIV: Great presentation by @RealGeneKim at #PuppetConf. Very interesting and inspiring!
  • @joshroppo: @RealGeneKim Thanks for the great #puppetconf talk! Any chance your slides will be posted online?
  • @patrickdebois: I don't mind if someone brings me a #puppetconf T-shirt - Size L :)
  • @pvdissel: RT @phantom: "The only way to avoid failures, is to fail always" #PuppetConf
  • @yumano: 超受ける。LOL>If you don't choose Chaos monkey, Chaos monkey will choose you.!!
  • @jordansissel: Always love presentations by @realgenekim
  • @nrcantor: Easily the best keynote at #puppetconf is @RealGeneKim's. Awesome. I want a copy of the talk.
  • @ryanj: RT @kad: "If you don't allocate 20% of your cycles to technical debt reduction, you will spend 100% of your cycles"
  • @mochicrunk: RT @phantom: "The only way to avoid failures, is to fail always" #PuppetConf
  • @kad: This is probably one of the best "soft skills" conference in an operations-centric conf I've been to, thanks @RealGeneKim
  • @tophlammiepie: Gene Kim++ manage IT in ways that makes your life as a human being better - #puppetconf makes you happy
  • @kcampos: RT @kad: "If you don't allocate 20% of your cycles to technical debt reduction, you will spend 100% of your cycles" #puppetconf #li
  • @GBTimko: Automation makes IT better #devops #puppetconf #cloudops
  • @ahpook: Damn, @realgenekim throwing down the knowledge at #puppetconf ...
  • @phantom: "The only way to avoid failures, is to fail always" #PuppetConf
  • @beauchristensen: Great talks today, streamed live at #puppetconf. Thank you @puppetlabs for making this available to all of us http://t.co/boLshFth
  • @sfrench: RT @kad: "If you don't allocate 20% of your cycles to technical debt reduction, you will spend 100% of your cycles"
  • @Mikey_Regan: @RealGeneKim I enjoyed your keynote at #puppetconf, I filled my kindle with all the books you mentioned.

END

  • @miiiiiche: .@sonofhans is gathering user data in his talk. Genius. #UX
  • RT @miiiiiche: .@sonofhans is gathering user data in his talk. Genius. #UX

Puppet at GitHub, Jesse Newland

  • Great stuff in "Puppet at GitHub" by Jesse Newland! Awesome seeing behind the curtains of awesome svc like github!
  • Newland: "We use hubot daily with jenkins, graphme, capistrano"
  • Newland, Github: "Why is hubot (our stupid chat bot) so useful? Everyone sees everything that happens from Day 1"
  • @nullfinch: RT @dhgwilliam: ChatOps++ @github
  • RT @nullfinch: RT @dhgwilliam: ChatOps++ @github
  • Newland: "Things u learn: what broke, how to fix it, who to apologie to, how to rollback, how to celebrate (clear status pg)
  • Newland: "ChatOps: building tools to operate your infrastructure by chatting, instead of command line or Chrome
  • Newland: "By placing chatops in middle of ops, you pair everyone, teach by doing"
  • Newland: "It triggers what we don't have docs for, creates backlog for automation creation; Communicate by doing
  • Newland: "I haven't ask: 'how's the deploy going?' 'has someone cleared that alert?' All automatic & visible
  • Newland: "Github ops is very distributed: no two ops people in the same city (!?!)" (Wow)
  • Newland: "@alexanderfortin: ChatOps @Github, i.e. IRC bot awesomeness live on #puppetconf /cc @roccozanni
  • RT @alexanderfortin: ChatOps @Github, i.e. IRC bot awesomeness live on #puppetconf /cc @roccozanni
  • Newland: "Hubot makes muting nagios alerts very easy, so people actually do it@GBTimko: RT @eshamow: I think @jnewland's #chatops deserves its own hashtag. #puppetconf #devops
  • Agreed! RT @GBTimko: RT @eshamow: I think @jnewland's #chatops deserves its own hashtag. #puppetconf #devops
  • Haha. Awesome. Newland: "Hubot makes on-call duty so much less shitty."
  • Does anyone have Newland's twitter handle? @complex: hubot & the chat room, 21st century equiv of hallway meets, reintroducing workplace serendipity. cc: @jnewland @github #puppetconf #chatops
  • RT @complex: hubot: the chat room, 21st century equiv of hallway meets, workplace serendipity. cc: @jnewland @github #puppetconf #chatops
  • Amazing #chatops session by @jnewland in main room. His kung fu rocks. :)
  • @bindr: RT @0xEFF: Novel idea #ChatOps is all about placing tools (like hubot) in the middle of the conversation. @jnewland #puppetconf http://t.co/fxQeDFld
  • RT @bindr/@0xEFF: #ChatOps is places tools (like hubot) in the middle of the conversation. @jnewland #puppetconf http://t.co/fxQeDFld
  • @jnewland: "Github ops grown from 3 to 8; training is new challenge; hubot is easier than writing documentation@ruthlinehan: Puppet at GitHub and ChatOps. So cool! @github #puppetconf http://t.co/BdedJv4V
  • @nullfinch: @rodjek did @jnewland's presentation go to your head yet? :)
  • @jnewland: "Q: how do handle alerts? A: alerts go to Notices room; stuff that goes into Ops room, only critical alerts
  • @jnewland: "When github goes down, alerts disappear; to handle signal/noise, several Campfire rooms resv for experiments
  • PS: Thanks for @jnewland's twitter handle, all! (Giving up trying to replying to all; damn Twitter client. :)
  • @jnewland: "Github drinkup! Joing him tomorrow 830pm @ Zeke's: 3rd and Brannan"
  • @nrcantor: RT @eshamow: Anyone who missed @jnewland's #chatops talk at #puppetconf just missed a seminal moment in #devops. Catch the recording when it comes out.
  • RT @nrcantor/@eshamow: Anyone who missed @jnewland's #chatops talk at #puppetconf just missed a seminal moment in #devops. Catch recording!
  • @jeffsussna: RT @puppetlabs: RT @jnewland: Slides from my #puppetconf talk http://t.co/SlEcvcp7 << created a new pinboard tag for this #chatops
  • RT @jeffsussna/@puppetlabs/@jnewland: My #puppetconf slides http://t.co/SlEcvcp7: created a new pinboard tag for this #chatops
  • @benjamin_ds: RT @puppetlabs: RT @unixgeekem: The Ops bottleneck illustrated at Wikimedia (at #PuppetConf) http://t.co/1KvNxYXO
  • RT @benjamin_ds/@puppetlabs/@unixgeekem: The Ops bottleneck illustrated at Wikimedia (at #PuppetConf) http://t.co/1KvNxYXO
  • @maddox: RT @jnewland: Slides from my #puppetconf talk on #chatops https://t.co/vCZ8RwkV
  • @eshamow: RT @jnewland: Slides from my #puppetconf talk on #chatops http://t.co/TJF3gLZ4 < READ THIS if you read nothing else today.
  • @poohneat: number of star trek references at #puppetconf is now at 6, data was crying
  • Haha. ST:TOS forever. :) RT @poohneat: number of star trek references at #puppetconf is now at 6, data was crying
  • @jetdillo: #hubot looks pretty freakin' cool: http://t.co/S8xxr2Wd
  • Fascinating. Wikimedia funnels alerts into RSS feeds (read with Google Reader); Github puts them into chatrooms...
  • @PuppetConf: RT @nrcantor: Puppet progression: Hard Coded -> Parameterized Classes -> Hiera & ENC -> ?
  • @PuppetConf: great tweets! where are you sitting? totally want to show you @tweetscriber. You'll never go back to the old way. :)
  • Ryan Lane,Wikimedia : "All puppet modules/configs used forWikipedia & Wikimedia properties available for world to use"

Ryan Park, Head of Pinterest Ops

  • Next up: Ryan Park, Head of Pinterest Ops: Highest AWS growth rate
  • Park: Pinterest based on python, entirely in EC2; his team did a great talk at #velocityconf this year.
  • Park: "before puppet, we'd use EC2 AMI images; 150 virtual servers, 12 images; cut -f 1 ~/.ssh/known_hosts"
  • Park: "in old days, we'd manually change the .bashrc to change host names; made for lots of unique snowflakes
  • Park: "problems: difficult to even find an authoratitive list of active server instances
  • Park: "our 'source of truth' is puppet dashboard; not puppet manifests
  • Nice. Park: "Want to use our Puppet modules/configs? Goto http://github.com/pinterest"
  • Nice. Park: uses data from puppet to generate /etc/hosts, push hostnames to Amazon Route53, etc.
  • Park: On puppet and EC2: "we have 1 AMI for all instances; basic Ubunti AMI w/facter, puppet & ec2-api-tools; 20m to launch
  • Park: "problem: too slow to launch (20m): soln: we pre install 60 Debian packages"
  • @jburrows: Ryan Park showing off how Pinterest uses Puppet. #PuppetConf http://t.co/h4BfWXvS
  • @eshamow: Watching @StanfordRyan's @Pinterest talk at #puppetconf - doing some awesome stuff with REST interface and custom clients.
  • .@StanfordRyan: "Prob:
  • @RealGeneKim: "@StanfordRyan: My #PuppetConf talk on Puppet at @Pinterest begins at 2:00! Slides and code samples are on GitHub. https://t.co/wFizkrsU"
  • .@StanfordRyan: "@eshamow: "By using autoscaling we have cut average server deployment by 40%…server usage is now directly tied to demand" @StanfordRyan at
  • Park: "RT @eshamow: "By using autoscaling, cut avg deploy time by 40%. server usage now directly tied to demand" @StanfordRyan at
  • @dberkholz: Pinterest cut EC2 instance use by 40% by switching to autoscaling.
  • .@StanfordRyan: "We're hiring! http://pinterest.com/about/careers! We're doubling IT Ops staff!"@eshamow: "A single source of truth usable both by humans and servers" < EXACTLY where most orgs aren't and need to be. @StanfordRyan at
  • RT @eshamow: "A single source of truth usable both by humans and servers" < EXACTLY every org needs this @StanfordRyan at
  • @botchagalupe: #puppetconf is awesome. Freaking loved @jnewland 's @github #chatops presentation - Culture Lives! ...
  • Agreed. RT @botchagalupe: #puppetconf is awesome. Freaking loved @jnewland 's @github #chatops presentation - Culture Lives! ...
  • .@StanfordRyan: "AWS spot instances half the cost of standard instances; can get shut down, but still highly recommended"
  • .@StanfordRyan: "Very difficult to move running instances to VPC; if starting out, start with VPC"
  • .@StanfordRyan: "
  • .@StanfordRyan: on
  • @PuppetConf: @mestery @lynxbat Don't worry! You can still catch the conference stream online! http://t.co/nHYwGVi9#puppetize
  • @PuppetConf: RT @botchagalupe: RT @botchagalupe: #puppetconf Sitting next to the world famous @mitchellh waiting for his and also looking forward to @jezhumble 's upco ...
  • .@jezhumble is up at 3:50pm. RT @botchagalupe #puppetconf Sitting next to world famous @mitchellh before his talk, @jezhumble up after that
  • @rylon: RT @eshamow: "20 minutes is too slow" -- we are in a different universe now re server deployment. @StanfordRyan at
  • Indeed. RT @rylon/@eshamow: "20 minutes is too slow" -- we are in a different universe now re server deployments @StanfordRyan
  • @rylon: RT @eshamow: "20 minutes is too slow" -- we are in a different universe n@PuppetConf: RT @saasoperator: RT @saasoperator: Watching all the great tweets coming out of #puppetconf today. Going to have to attend next time around!
  • ow re server deployment. @StanfordRyan at
  • Altho, focus needs to shift to keeping deployment pipeline going, which is bottleneck now, cuz deploys r so fast @rylan/@eshamow/@jezhumble
  • @PuppetConf: 10 minutes until @mitchellh's talk on advanced use cases of Vagrant in Puppet environments. Check it in Robertson at 2.45 PM!
  • RT @PuppetConf: 10m until @mitchellh's talk on advanced use cases of Vagrant in Puppet env. Robertson at 2.45 PM!

@kentbye: How Pinterest Uses Amazon EC2's Auto-Scaling Features with #Puppet http://t.co/EGFdudpG #PuppetConf @StanfordRyan

Mitchell Hasimoto: @mitchellh: Vagrant in Puppet

  • .@mitchellh: "
  • @Pintsider: RT @GBTimko: RT @GBTimko: Pinterest uses puppet in ec2 to auto scale and cut cost at night scale high during day #puppetconf #pinterest great pres @s ...
  • RT @GBTimko: Pinterest uses puppet in ec2 to auto scale and cut cost at night scale high during day #puppetconf #pinterest great pres

  • .@mitchellh: "I'm an automation freak, as I'll prove in my talk; desires: mainfest dev, both simple & not simple; repeatable

  • .@mitchellh: "Fast feedback; confidence; used to be chef users for 3yrs; now puppet"

  • .@mitchellh: "current state of vagrant+puppet: basic manifest dev/test; but that's it... we can do much better than that

  • .@mitchellh: "MY state of vagrant+puppet: fully automated puppet master setup, in one cmd; common deploy for vagrant & EC2

  • .@mitchellh: "@0xEFF: Puppet releases over time. Ten times more puppet in 2010. #puppetconf http://t.co/ij1MGFdN

  • .@mitchellh: "result: production ready puppet master on your own machine in 10m"

  • .@mitchellh: "Q: how do ppl test more than the most basic puppet module? A: suspect they don't test modules adquately. (NO!!)"

  • .@mitchellh: "solution: automated puppet master + multi-vm

  • .@mitchellh: "puppetdb, hiera, etc.."

  • .@mitchellh: "multi-vm: enables vagrant to manage a cluster of machines that can communicate

  • .@mitchellh: "Prob: How do you test Hiera? not hiera code, cuz puppetlabs test that; how do u test configs, loading right seq

  • .@mitchellh: "We do Dev in VirtualBox; Staging is in EC2; Production is in EC2; same workflow"

  • .@mitchellh: "Because vagrant rake cmds are same in dev/staging/production, you learn one, you've learned them all"

  • .@mitchellh: "Q: do you tie into jen

@PuppetConf: RT @evilensky: RT @evilensky: #puppetconf is informative, engaging, and fun. The "Gartner industry trends" sessions are at a nice minimum.
@PuppetConf: RT @evilensky: RT @evilensky: #puppetconf is informative, engaging, and fun. The "Gartner industry trends" sessions are at a nice minimum.

@courtneynash: RT @nullfinch: RT @nullfinch: Does eating the Google ice cream at #puppetconf mean our intestines will be indexed and put on Street View?
@courtneynash: RT @nullfinch: RT @nullfinch: Does eating the Google ice cream at #puppetconf mean our intestines will be indexed and put on Street View?
Yep. It's live! RT @courtneynash/@nullfinch/@nullfinch: Eating Google icecream @ #puppetconf. Wiil r intestines be indexed & on Street View?

Jez Humble: The Deployment Pipeline: from check to release"

  • Next up: legendary Jez Humble: "The Deployment Pipeline: From checkin to release"
  • .@jezhumble: "birth of CD: provisioning process: perl to J2EE: wrote on Windows, deployed on Solaris; what could go wrong?"
  • .@jezhumble: "painful to test (2 weeks to deploy); 8K lines of ANT code to build"
  • @kad: "... and it was written in java, so what could possibly go wrong, right?"
  • .@jezhumble: "We got it to build and deploy within an hour, and flip back/forth between N and N-1; and no more weekend work
  • .@jezhumble: "This was in 2005; we got so sick of explaining that there was a better way, so we wrote "Continuous Delivery"
  • .@jezhumble: "30% of attendees here work for > 1000 employees; so many of you got Agile training, but nothing gets deployed
  • .@jezhumble: "Dev Complete to Service Live is most painful part: 'the last mile': it's actually the first mile..."
  • .@jezhumble: "50% of you are having this problem; which sucks, because it's 2012"
  • .@jezhumble: "2009: flickr: last deployed 12h ago, 9 changes by 5 people; last wk: 54 deploys of 636 chgs by 23 ppl"
  • .@jezhumble: "@allspaw: found that Flickr had better uptime than any other Yahoo! property; CD can incr stability"
  • .@jezhumble: "Goal: build the right thing: 'more than 50% of features are never or rarely used'. biggest src of waste"
  • .@jezhumble: "Least efficient way of testing: build the feature & see whether people use; best way: minimum viable product"
  • .@jezhumble: "
  • @clstokes: . @jezhumble on Agile, "Now everyone is taking orders standing up rather than sitting down." #puppetconf < LOL
  • .@jezhumble: "Eric Reis: IMVU startup: no one clicked past landing page; so MVP is landing page w/404 page for every link"
  • .@jezhumble: "In enterprises, success is proj completed, not whether svc used or achieved ROI; & poor bastard has to maintain
  • .@jezhumble: "How long for ur org to deploy a change that involved one line of code? Is it repeatable/reliable (ie not manual)
  • .@jezhumble: "More freq releases decreases risk; 3 months cycle time is lots of changes; almost impossible to rollback
  • .@jezhumble: "Small batch sizes means small changes, easy to find and fix. And lots of practice, which leads to mastery
  • .@jezhumble: "Flickr got awesome performance, because they optimized for Mean Time to Restore Svc; small chgs + monitoring"
  • .@jezhumble: "Video available of Canadian Army disassembling and reassumbling Jeep in less than 2m! Cheap/easy to repair"
  • .@jezhumble: "Satellites (and medical equipment) cares about MTBF; rest of us, optimize for MTRS"
  • .@jezhumble: "Continuous delivery also allows real measurement of project: code commit is not done; deployed is done"
  • .@jezhumble: "Y distance: batch size; X distance is lead time"
    https://pbs.twimg.com/media/A31cIMmCEAA4Szy.jpg
  • .@jezhumble: "Quote from Agile Manifesto: 'highest priority: satisfy customer thru eary and continuous delivery"
  • .@jezhumble: "Goal: Fast automated feedback on prod readiness of ur apps for every chg: code, schema, infrastructure, config"
  • PSA: Translation: when @jezhumble says "rooter", he means "router"
  • .@jezhumble: "Must prioritize keeping system working over 'doing work.' Must always maintain production readiness"
  • .@jezhumble: "Now releases tied to business needs, not operational constraints (vs always waiting for IT)"
  • .@jezhumble: "Now IT is no longer the constraint; creating business requirements is the constraint"
  • .@jezhumble: "When I say 'config mgmt,' I mean one-cmd-line to go from new box to new instance spun up"
  • .@jezhumble: "Every checkin must go into trunk, and is tested"
  • .@jezhumble: "If u're working on feature branch, u're not doing continuous integration; check into trunk at least once/day"
  • .@jezhumble: "Deming: Cease dependence on mass inspection to achieve quality; Improve process & build quality into product"
  • .@jezhumble: "Quality is the responsbility of everyone"
  • .@jezhumble: "QA is everyone's job: Role of testing is to make quality transparent, so team can decide what to do about it"
  • .@jezhumble: "
  • @botchagalupe: #puppetconf .. Oh shit I better not suck tomorrow... both @jezhumble and The @RealGeneKim both did a shout out to my presentation tomorrow.
  • Haha. @jezhumble: "At night when humans to to sleep, computers laugh at us: nowhere is this more true than manual testing"
  • .@jezhumble: "
  • @eshamow: .@jezhumble "When human beings do the work computers should be doing, the computers get together at night and laugh at us."
  • RT @eshamow/@jezhumble "When human beings do work computers should be doing, computers get together at night and laugh at us."
  • .@jezhumble: "The goal & aspiration of every build is to make it into production; not all make it; like Hindu concept of (XX)"
  • .@jezhumble: "Puppet workflow in NBN:
  • @nullfinch: RT @eshamow: RT @eshamow: .@jezhumble "The cheapest way to fix a bug is to not check it in"
  • RT @nullfinch: RT @eshamow: RT @eshamow: .@jezhumble "The cheapest way to fix a bug is to not check it in"
  • .@jezhumble: "CI and puppet would generate report with 'proposed' chgs, which can be reviewed by chg mgmt, to satisfy reg req"
  • .@jezhumble: "My colleague Andrew insisted that I say, 'Vagrant is awesome.' Critical for this project I described" @mitchellh
  • @botchagalupe: @dr138 750 Attending and 2k+ viewing online... @puppetmasterd has come a long way... ( http://t.co/b6ElyQaZ )
  • RT @botchagalupe: @dr138 750 Attending and 2k+ viewing online... @puppetmasterd has come a long way... ( http://t.co/b6ElyQaZ )

Ruaidhri Power, Demonware: "We do

  • Ruaidhri Power, Demonware: "We do Lobby services for matchmaking, leaderboards, stats, messaging for XBox Live, Call of Duty"
  • Power: "in 2007: 20K concurrent users was a big deal; Now: 11 dev, 3 ops, 3 managers; recently acq by Activision"
  • Power: "90+ games in total now; Now: 280+ million games supported; 2.4M concurrent users"
  • Power: "Puppet initiative in 2009 as skunkworks project: we did lots of bad things (eg: passwords in code, hosts in spreadsht"
  • Power: "Because passwords in puppet code, we couldn't share puppet code with developers" (Nice)
  • Power: "We used noop in production; saved us from catastrophe, but really slowed us down"
  • Power: "2010: Moved from Ubuntu to CentOS: opportunity to do puppet right"
  • Power: "MySQL users (users, passwords, grants); MySQL databases, sysctls"
  • Power: "To avoid passwords in puppet: we wrote function to lookup pws in puppetmaster; can share modules w/o sharing secrets"
  • Power: "Internal django app; management app that does everything; game developer interface: for game dev, not hosters"
  • Power: "ENC script: python script connected to our custom inventroy databse: django"
  • Power: "bdpuppetconfig: bitdaemon:

@preston4tw: RT @stahnma: RT @stahnma: Comedy gold from @agentdero #puppetconf http://t.co/ArrilPa8
@courtneynash: Ending day 1 of #puppetconf with some ops anti-patterns and humor with @agentdero. So far he's dropped "schaudenfreude" and "shenanigans."

AgentDero, Ops Anti-Pattern

  • @nullfinch: RT @stahnma: RT @stahnma: "You should probably not take ambien while you're on-call" -- @agentdero #puppetconf #opslife
  • RT @nullfinch: RT @stahnma: RT @stahnma: "You should probably not take ambien while you're on-call" -- @agentdero #puppetconf #opslife
  • @Aethylred: RT @Bastux: RT @Bastux: "We designed the stuff around a single point of failure and never thought about this" #puppetconf #operations http://t.co/8o ...
  • .@agentdero: "kernel was so old that patches weren't getting backported; didn't get leap year bug fix: ruined new years for everyone"
  • .@agentdero: "all work went into JIRA: file JIRAs to multiple people, amortizing risk; spamming JIRA" (everyone in org is person)
  • .@agentdero: "dev must beg ops for everything, if ops is locked down: i've heard lots of things about mongodb; putting puppet manifests and code in the same repo: because everyone can run in blessed env"
  • .@agentdero: "Dev can get so frustrated w/Ops that for new project, they totally bypass them; outsourced; 1y later, Ops discovered"
  • .@agentdero: "
  • @langdonwhite: DOS'ing ticketing systems as a method for organizational change! thanks @agentdero
  • JIRA spamming. Awesome. RT @langdonwhite: DOS'ing ticketing systems as a method for organizational change! thanks @agentdero
  • .@agentdero: "There's a time to just get the product out, to see whether this crazy business idea will actually work"
  • .@agentdero: "Anti-pattern: MySQL expert compares CouchDB hyped pos vs. MySQL neg; 1 yr later, migrated back. Painful"
  • .@agentdero: "Anti-pattern: NIH: interesting b/c it's universal to ops, QA, dev all say, 'I can do that...'"
  • These @agentdero stories are awesome. Can't compact them into 140c. I give up. :)
  • @solarce: RT @stahnma: RT @stahnma: Comedy gold from @agentdero #puppetconf http://t.co/ArrilPa8
  • @jetdillo: "There's a fine line...and it's probably 6 months behind you"
  • .@agentdero: "Netboot all things: doesn't scale; everyone wants to be Facebook; pre Razor, NetApp, PIXI book; easy, until central storage dies, and entire site dies
  • .@agentdero: "These days, it's easy to do local storage
  • .@agentdero: "Logstash or /dev/null: everything is just garbage: DEBUG flag
  • .@agentdero: "Test your backups"
  • .@agentdero: "@stahnma: "Dont' delete production data in order to test backups" -- @agentdero #puppetconf #opsantipatterns
  • @stahnma: "Dont' delete production data in order to test backups" -- @agentdero #puppetconf #opsantipatterns
  • Everyone groans/cringes. :) RT @stahnma: "Dont' delete production data in order to test backups" -- @agentdero #puppetconf #opsantipatterns
  • .@agentdero: "Multi-site: ops bravado: 'I'm in six sites globally'; bought two of everything, never updated apps, testing, DR
  • .@agentdero: "
  • @nullfinch: Quoth @agentdero "We run in 6 data centers globally bla bla bla bler hickory dickory" - on pointless multisite.
  • .@agentdero: "Continuous deployment: let any deploy, especially me from my laptop: must be from central source of truth. I've done it, and will again"