2013/08/23 PuppetConf Day 2

by Gene Kim on

#puppetconf

Up: Jez Humble, ThoughtWorks Studios: "Stop Hiring DevOps Experts (And Start Growing Them) @jezhumble

  • .@jezhumble: Michael Sahota: 4 culture quadrants: "collaboration (UL), control (UR), cultivation (LL), competence (LR)"
  • .@jezhumble: "Ops tends to be Control quadrant; noting that so many Ops characters are ex-military in #phoenixproject"
  • .@jezhumble: "Dev community in 1960s also were in Control quadrant; conferences sponsored by NATO; extensive forward planning
  • .@jezhumble: "During 70s/80s, Dev migrated to agile-like, smaller iterations away from Control; Ops going thru same transition
  • .@jezhumble: "Often missing from cultures are Cultivating: growing people" (like servant leadership, I think)
  • .@jezhumble: "My thesis: you can't cultivate people by hiring them; you can't hire in culture change: oft disruptive/painful
  • .@jezhumble: "New person brought in faced with hostility; often miserable for that person; ability to chg culture very diff
  • .@jezhumble: "Organizations reject change in general
  • .@jezhumble: "Cultivation culture: output is not just systems we build, but also knwldge; Ex: 3M policy of 20% time innovation
  • .@jezhumble: "It's not enough to have 20% slack time for innovation; ability to execute is just as important; Ex: Kodak
  • .@jezhumble: "XP/pair programming: 'this will never work'; when joined ThoughtWorks, got paired, learned TDD, awesomeness
  • .@jezhumble: "Only by pairing with really smart people is only way to create visceral understanding; not training/books
  • .@jezhumble: "Experimentation/innovation vs. risk mgmt theatre; mgmt is not about cmd/control, instead to foster innovtn
  • .@jezhumble: "Key mgmt KPI should be 'how good were you at growing your people?' as part of review process
  • @binford2k: The role of management is to create an environment in which it's safe to learn.
  • RT @binford2k: The role of management is to create an environment in which it's safe to learn.
  • @stack72: Organisations can encourage improvisation but they don't follow it through. People still get in troublem for failing
  • .@jezhumble: Citing book Toyota Kata by Mike Rother. Awesome and amazing book: hugely influenced #phoenixproject
  • .@jezhumble: "Rother: you can only be a mgr in Toyota plant by being on plant floor for 7 yrs (? did I get this right?)
  • @chorankates: 'the way you become a manager is by knowing how to do the work' @jezhumble
  • .@jezhumble: "Training is actually not that effective; u need to know why it's important & build muscle memory thru practice
  • .@jezhumble: "Managers must make it safe and easy to learn
  • .@jezhumble: "Tools are not an effective way to change." (Huh. #puppetconf live stream just went done. Hahaha :)
  • @kylemurley: #puppetconf Training != Tools != Change
  • @JonHarrisNM: "Training not the answer, hands on learning is"
  • RT @kylemurley: #puppetconf Training != Tools != Change
  • RT @JonHarrisNM: "Training not the answer, hands on learning is"
  • .@jezhumble: "If you create a Cultivation culture, suddenly you may no longer have a hiring problem!" (Indeed)
  • .@jezhumble:
  • @madgreek65: Create an organization where people value learning and you won't have a hiring problem
  • @plarivee: Arny at #puppetconf ! http://t.co/nTo3HYbaWf
  • .@jezhumble: "Game Days is a great anti-fragile pattern; that which doesn't kill you makes you stronger"
  • .@jezhumble: "It's like backing up but not practicing restores; Simian Army; Google exercises"
  • .@jezhumble: "Dave Mangot: 'are u devops? I don't know; do you care about flow, communications vs. silo; like 'are you Agile?
  • @kylemurley: #puppetconf MT @davemangot "Are you a DevOps? Of course not, are you AN AGILE?"
  • @madgreek65: Are you a DevOps? Of course not are you an agile?
  • @chorankates: @davemangot getting some love from @jezhumble at #puppetconf keynote -- "are you an agile?"
  • @s0enke: .@netflixoss everywhere
  • @randomfrequency: “@ineverthink: @jezhumble “Creating a new silo to solve a silo problem is an ironic way to solve the problem” #puppetconf” so much
  • @Aethylred: Netflix Simian Army: automated failure to continuously test resiliancy.
  • .@jezhumble: "Microsoft is likely in the Competence corner of quadrant" (fielding Ballmer retirement question)
  • .@jezhumble: "And ThoughtWorks is hiring. Umm, ironic, I know..." (hahahaha)
  • @petersjolin: RT @madgreek65: Are you a DevOps? Of course not are you an agile?
  • RT @madgreek65: Are you a DevOps? Of course not are you an agile?
  • @binford2k: Cultivate, don't control. @jezhumble #puppetconf http://t.co/KhlNPnbAxU
  • @SoberBuildEng: "It might be a few years before the results of your bad HR policies become obvious."-@jezhumble (And then it's super obvious!)
  • @kylemurley: #puppetconf The trick is creating a crisis before there actually is one and then know how to deal with it
  • @tonycollander: A school system which treats ppl like empty bank accounts to fill w/ knowledge and then having companies emptying them is wrong.
  • @stacy_brock: .@rquelle keynote: "You call that a knife? This is a knife." There's no one tool to rule them all.

Up: Reinhardt Quelle, Cloud Services Architect, Cisco/WebEx: "Puppet For Production in WebEx"

  • Quelle: "WebEx: Millions of mtgs, 10s of millions of users, billions of minutes each month; no scheduled downtime
  • Quelle: "Only 7K hosts, compared to Google; 8 data centers, 12 iPOPs;
  • Quelle: "Spent time as cabinet maker, remodeling contractor; lots of simple parts (carcasses);
  • Quelle: "How do we upgrade pkgs? Some are easy b/c they can be replaced; but things like db svrs must be upgraded inplace
  • Quelle: "Every upgrade must be done in a specific sequence; most horrendous upgrades have lots of dependencies
  • Quelle: "We never run one of anything; cluster mgmt lives above puppet, like cabinet maker working w/general contractor
  • Quelle: "Experiments w/cluster definition: Many clusters have JSON description, new open stds (tosca), CloudFormation, etc
  • Quelle: holding up Leatherman: "it's tempting to use this well-integrated, tool that does lots of things for everything
  • Quelle: holding up huge saw: "for certain tasks, you need something like this, instead of Leatherman" (haha)
  • @madgreek65: RT @binford2k: It's very tempting to try to force a tool to do all the things. Better is to find the right tool for the job.
  • Quelle: "@walidshaari: under the belt attack for Fabric, Saltstack and Ansible in #puppetconf assuming that you can orchestration only not real configuration mgmt
  • @VMTrooper: #VagrantUp also getting some love as a Test\Dev Environment Tool for Cisco WebEx #puppetconf /cc @mitchellh
  • @binford2k: Don't look at your infrastructure as one big problem; think of it as many small problems and tackle them one at a time.
  • Quelle: "We want to reduce the 'bus factor'; you know, the problem of when one person gets hit by bus, can we continue svc
  • @madgreek65: RT @kylemurley: #puppetconf #DevOps elephant eating: "solve little problems then build out"
  • Quelle: "Design pattern: cooperative modules; loosely coupled modules that can stand on their own
  • Quelle: "Design pattern: assembling configurations: puppet-library
  • Quelle: "modules are designed for transparency: we have '4am-proofing': must be understandable even at 4am
  • Quelle: "In other words, 4am-proofing says it's okay to repeat yourself (vs DRY) if it makes it easier to fix at 4am" (Nice)
  • Quelle: "All Ops hotfixes MUST flow all the way back to Dev" (must keep all value stream synced)
  • Quelle: "By the way, Cisco/WebEx is hiring" (Haha. It's true, everyone is hiring)

Up: Stan Hsu, Sr Dev Mgr, PayPal: "Puppet At Scale: Case Study of PayPal's Learning"

  • Fascinating how many Devs are presenting at #puppetconf; It's so #devops
  • Hsu: "132MM registered users; 25 currencies in 193 markets; $43B payments in Q2; 7.6MM transactions/day; that's $5K/sec!
  • Hsu: "Years ago, toured new multi-hundred million high density data; center; QA lives in VMs; bad things happen w/that much stuff crammed into VMs
  • Hsu: "thousands of staged envs in QA; 3K developers; 100 GB of code! 4K packages; every release has new 20-50 pkgs
  • Hsu: "Worse, every checkin introduces more dependencies, req puppet chgs;
  • @kylemurley: #puppetconf Paypal counts code in GBs, not lines!
  • Hsu: "Entire PayPal svc provides REST API; openstack, heat; heira; Dev enters: {label, application, size} in cloud portal
  • Hsu: "When Dev hits Go: heat creates vm, registers w/load balancer, adds to DNS, register in heira, register w/puppet; fully automated
  • @Aethylred: PayPal is storing hiera data in a MongoDB. Interesting....
  • Hsu: "Project Velocity; Dev is deployed in minutes;
  • Hsu: "Challenge: trad 1 app per module doesn't work; how do you introduce new pkgs, sunset old pkgs, supporting 3K devs
  • Hsu: "Soln was to train all puppet skills to Dev; can we create system where puppet coding is not required?
  • Hsu: "That's why we wrote the Ninja engine: takes list of apps you want to install, and it will assemble list, autodiscover dependencies, generate puppet resources from dependency graph, and then execute; we then cache it
  • @stack72: Paypal store their hiera data in MongoDB
  • @garethr: Automated puppet code generation is a pretty interesting topic. Would make for a good talk topic
  • PayPal: desc problem of load balancer killing long-lived connections; betw ActiveMQ cluster and clients w/mcollective
  • PayPal: we heavily depend on mcollective: "if you're not using it, revisit it; it's helped us a lot; replaced all ssh scripts
  • @Aethylred: RT @rothgar: "When having trouble scaling MCollective we did what any good dev team does, we blamed someone else" #PuppetConf
  • @stack72: How on earth do Paypal have 100gb of code in their web application? That must be a lot of lines of code
  • @Aethylred: ActiveMQ clusters and Mcollective don't get along with load balancers. Allow direct connections and manage with puppet.
  • @ryanycoleman: Hey #PuppetConf folks. PayPal agrees, MCollective rocks. Did you know it comes ready to rock with Puppet Enterprise? http://t.co/pOvNe7bX1O
  • @plarivee: Their puppet progress modules looks nice #paypal

Up: amazing Nick Weaver, Cloud Automation Arch, Hybrid Cloud Svc, VMware: "VMware vCHS, Puppet & Project Zombie" @lynxbat

  • .@lynxbat: "...oh, and please contact me if you want to work at VMware" (haha)
  • .@lynxbat: "My job is automation; it isn't dumb robot (Bender); or evil robot (Terminator), or push a license (Oracle)" HA!
  • .@lynxbat: "My definition of 'automation' is Effort Evolution; makes me more powerful as technologist
  • @kylemurley: #puppetconf @lynxbat "Automation is effort evolution" tech by itself brings no value, BE THE CHANGE
  • @madgreek65: Larry Ellison just got bitch slapped at
  • .@lynxbat: "There's a difference between 'eating our own dogfood' and just 'wearing a dog suit'
  • @VMTrooper: "vCHS needs adapt to the rate of change as VMware releases new features to the Enterprise" @lynxbat
  • .@lynxbat: "We called it Project Zombie because it's something that you cannot kill" (Haha. ie, resilience)
  • .@lynxbat: "We chose puppet, cassandra, jruby" (why? because ops use ruby, architects live in code; ruby is good middle ground)
  • .@lynxbat: "We choose RabbitMQ
  • @stack72: "You can never make the assumption that everything stays up" WRT to ProjectZombie by @lynxbat
  • @VMTrooper: "Expect things to fail, accept failure in your system design,Project Zombie: an automation system that you cannot kill" @lynxbat
  • @VMTrooper: "Expect things to fail, accept failure in your system design,Project Zombie: an automation system that you cannot kill" @lynxbat
  • .@lynxbat: "Res is our refrigerator; track all our resrouces: how much storage? what is the LUN settings? ownership
  • .@lynxbat: "ZED is the Zombie Engine DSL: ZED allows concurrent launching of tasks for ops (parallelize 48 chgs * 2 min)
  • .@lynxbat: "@BaconIsKing: All operations in Rez happen through REST API #vchs #PuppetConf
  • @VMTrooper: #ProjectZombie Rez - All REST everything @lynxbat
  • Fascinating talk on Project Zombie by @lynxbat: This is automation of standardized ops work; love it.
  • @behrisms: god @lynxbat Zed (Project Zombie) is freaking cool. What a great talk Nick Weaver is giving at
  • @PuppetConf: Liked @lynxbat's keynote? Check out his upcoming #Automation Generation talk in the Venetian room. http://t.co/r1IRPXLsCu
  • Project Zombie handles queueing, job definition, scheduling, sequencing, abstraction... This is so @lynxbat...
  • @madgreek65: VMWare using 47 community Puppet modules in addition to their own.
  • @plarivee: I have been waiting for this one #logstash #puppetconf #ballroom http://t.co/sCpO1lcegt
  • @georgemcintosh: Zombie is a Puppet sandwich - oddest sentence ever
  • .@lynxbat: "Ex: bring up cloud build: 72 man-hours; lead time: 6 days; max ops per week: 2"
  • .@lynxbat: "After Zombie: 1.5 man-hours; lead time: 1 hour; max ops per week: ___?" Wow.
  • .@lynxbat: "WBS for that operation: 120 tasks; 2700 config points; 1400 managed resources;
  • @puppetlabs: "We turned 4–5 days of human effort—mistake-prone, difficult to do—to 1–2 hours, hands-off." @lynxbat talking #automation at
  • @nodoubleg: I wonder how much suck Project Zombie removes from the EMC VNX.
  • @BaconIsKing: ZED enabled VMware to go from 6 days to 1.5 hours for cloud deployment #PuppetConf
  • .@lynxbat: "Project Zombie helped VMware get to market quicker to take on entrenched competitors"

Misc

  • @kartar: RT @ruthlinehan: Awesome to meet so many amazing women of the Puppet community at the #puppetconf women’s breakfast. Thanks @FeyNudibranch
  • @Aethylred: I wonder if @rquelle is aware of the http://t.co/uBLX8MZfFv project to train researchers devops-like programming skills?
  • Stunning. RT @kylemurley: #puppetconf Paypal counts code in GBs, not lines!
  • HAHAHA! RT @rothgar: "When having trouble scaling MCollective we did what any good dev team does, we blamed someone else" #PuppetConf
  • RT @kylemurley: #puppetconf @lynxbat "Automation is effort evolution" tech by itself brings no value, BE THE CHANGE
  • That was funny. Cited as "automation is pushing a license" :) RT @madgreek65: Larry Ellison just got bitch slapped at
  • RT @stack72: "You can never make the assumption that everything stays up" WRT to ProjectZombie by @lynxbat