10/3 Velocity Day 3

by Gene Kim on

#velocityconf

  • @redturtleltd: RT @stack72: RT @stack72: "show me a developer not causing an outtage and ill show you a developer on vacation" @RealGeneKim #VelocityConf
  • @devpg: RT @mpaluchowski: RT @mpaluchowski: "It's OK for us to break sh** as long as we can fix it very quickly."
  • @rlengwinat: RT @stack72: RT @stack72: devs need to take ownership of instrumenting their code. enable self service metric creation - key culture shift #VelocityConf
  • @CommunityLP: RT @devpg: Create a culture of continuous experimentation! (by @mjenno and @davenolan from @lonelyplanet) #velocityconf http://t.co/SGxyF4jL
  • @AndrewBrockway: RT @lozzd: RT @lozzd: Hilton room 1-6 packed for DevOps Pattern Distilled #velocityconf #floorroomonly http://t.co/86TA2eSl
  • @mark_barger: RT @devpg: RT @devpg: Create a culture of continuous experimentation! (by @mjenno and @davenolan from @lonelyplanet) #velocityconf http://t.co/qvK ...
  • @stack72: fantastic talk on devops patterns - this has really motivated me to get this culture shift #VelocityConf
  • @Bruvva74: Brilliant!! #inspiring DevOps Patterns Distilled at #velocityconf EU. http://t.co/42bmBIyL
  • @kief: @RealGeneKim said something about how the authors are "showing the sausage" - the sausage looks great so far
  • @unixdaemon: @botchagalupe is on roaring form.
  • @unixdaemon: "@botchagalupe says showing the sausage is very difficult" - @RealGeneKim #velocityconf #iam7
  • @mrmanc_tech: You don't choose chaos monkey, chaos monkey chooses you. #VelocityConf http://t.co/b6aUzz42
  • @abratland: RT @stack72 noops - one engineering team that emcompasses dev and ops. this os my goal to build! #VelocityConf //cc @noopman
  • @mudassarmian: you don't chose chaos monkey. Chaos monkey chooses you! #velocityconf #devops
  • @stack72: . @cread is getting some love at #VelocityConf
  • @Hiroyuki_Ohno: RT @unixdaemon: RT @unixdaemon: Anti pattern 1 - config management is devops - @patrickdebois
  • @mpaluchowski: Continuous experimentation - extremely important and missing in sooo many companies.
  • @neilch: RT @mrmanctech: RT @mrmanc_tech: "When we woke up developers at 2am, defects got fixed quicker" - Gene Kim #VelocityConf
  • @stack72: design for operations. make logs easier for then to read and config files easier to find etc. #VelocityConf
  • @hsiboy: Toyota, "khan barn" and "stopping the line" getting a mention.. Design for operations, make your logs readable!!@velocityconf: Meet with @ashcairo, Adam Cheng, & @alq at #velocityconf EU Office Hours, 13:15, by the O'Reilly booth. http://t.co/r9d7M3kd
  • @mark_barger: RT @pingdom: RT @pingdom: To stay fast, “make it trivial to measure anything” @davenolan
  • @stack72: noops - one engineering team that emcompasses dev and ops. this os my goal to build! #VelocityConf
  • @redturtleltd: @RealGeneKim loved the Dev Ops Patterns Distilled talk at #velocityconf. Enlightening and 100% relevant to me as a software developer.
  • @redturtleltd: @RealGeneKim any ideas when the video will be available?
  • @redturtleltd: @RealGeneKim loved the Dev Ops Patterns Distilled talk at #velocityconf. Enlightening and 100% relevant to me as a software developer.
  • @stack72: @RealGeneKim @damonedwards @redturtleltd the devops pattersn talk was recorded?
  • @ITSMNinja: RT @tomtaylor: RT @tomtaylor: New dev ops book out next year! Looking good! Dev Ops Cook Book! Big mention to #ChrisReed #VelocityConf http://t.co/IRu ...
  • Don't forget our beloved colleague "patrick debraw" :) @botchagalupe @patrickdebois @hsiboy
  • We tease who we love, right? :) @botchagalupe @hsiboy

    • @hsiboy: Toyota, "khan barn" and "stopping the line" getting a mention.. Design for operations, make your logs readable!!@velocityconf:
  • @rlengwinat: RT @stack72: RT @stack72: devs need to take ownership of instrumenting their code. enable self service metric creation - key culture shift #VelocityConf

  • @KojiISHIMOTO: RT @bluesmoon: RT @bluesmoon: if you missed my late night tweet, the summary and slides of our #velocityconf talk on #webperf are online: http://t.co/N ...

  • Yo, u're at wrong conference, mate. Java One isn't until next month! Haha. Thx for kind words! (From fellow (ex) developer) :) @redturtleltd

  • @hsiboy: @RealGeneKim It was a great talk, packed out and i spotted @nasrat in the crowd too. lolz taken in good jest

Michael Rembetsy, Director Ops Engineering (@mrembetsy), Patrick McDonnell, Senior Ops Engr (@mcdonnps)

  • Next up: Michael Rembetsy, Director Ops Engineering (@mrembetsy), Patrick McDonnell, Senior Ops Engr (@mcdonnps)

  • .@mrembetsy/@mcdonnps: "Here's how we've scaled our culture at Etsy. 125 engrs; 12 ops people"

  • .@mrembetsy/@mcdonnps: "sprouter was the 'middleware of distrust', initially designed to give Dev access ot the databse

  • .@mrembetsy/@mcdonnps: "

  • @cmsj: Etsy has 12 ops folks, 125 engineers, 350 employees total.

  • .@mrembetsy/@mcdonnps: "2008 was the year of pain; deploys took hours, w/ops at helm, code didn't work; no communication

  • .@mrembetsy/@mcdonnps: "Pushes failed, but couldn't restart easily, causing 500 errors across the site."

  • .@mrembetsy/@mcdonnps: "After a deployment day, I'd be completely spent. That's when we created internal blog fix.etsy.com"

  • .@mrembetsy: "We realized we had to fix tech debt; can't keep living in sea of engineering filth" (hahaha)

  • .@mrembetsy: "The day before Cyber Monday (largest ecommerce day), thought 'WTF did I get myself into?'

  • .@mrembetsy: "2008 stats: $87.3M; 163M visits; made decision to switch to CDN

  • @mrmanc_tech: Etsy 2008: 250 servers in two DCs, deploy take hours, and complicated process. Rollback similarly complicated. Sound familiar? #VelocityConf

  • @stack72: Etsy post outtage updates on their blog. Total transparency

  • .@mrembetsy: "Sea change was 2009; brought people back inside to Brooklyn [less remote]; 1st step was to manage banner

  • .@mrembetsy: "Breakthrough: could change banner w/o pushing code; then built Deployinator; infrastructure overhaul

  • .@mrembetsy: "Moved to hiring people in Brooklyn, moving offices to DUMBO; standup mtgs start to improve communication

  • @botchagalupe: etsy / deployinator http://t.co/0iI1tW69

  • .@mrembetsy: "2009: the year stability arrives; mgmt stops saying 'go do this.' Ppl happy to come to work; dev helping rack

  • .@mrembetsy: "Ended scheduled downtime; site remains up as much as possible; Master db purchased as capacity stopgap

  • .@mrembetsy: "2009 takeaways: beginning of DevOps culture; sales grow 102% that year

  • .@mrembetsy: "2009 action items: stabilize most painful part in org; hire staff that make diff; pick projects that matter

  • .@mrembetsy: "Just ship it

  • @_neckbeard: RT @mpaluchowski: RT @mpaluchowski: The plece you work in must fit with culture. Can't have lean, creative, agile in a plain, dull office.

  • .@mrembetsy: "2010: renewed energy; Kellen comes in as VP Engr; @allspaw comes in as VP Ops (was on etsy advisory board)

  • .@mrembetsy: "Started Code As Craft.etsy.com; created continuous integration team at end 2009; started standardizing on PHP

  • .@mrembetsy: "Use it or nothing else: benefits: everyone could read/rewrite your code; MySQL migration begins from Postgres

  • .@mrembetsy: "If it moves, graph it; tools: ganglia, graphite, built 'incomparable line technology' to correlate w/deploys

  • .@mrembetsy: "Nagios: 7000 checks for 700 hosts: we used to have lots more; pulled chks for unimportant 3am wakeups

  • .@mrembetsy: "Management ideals: accept failures but don't lower stds; blameless post-mortems; career planning

  • .@mrembetsy: "Happy company = happy community

  • @scoobiedoobie: Have a blameless postmortem if you have a failure

  • @mpaluchowski: "We're being woken up at 3AM for stupid things. Why?" Find out and fix.

  • @lozzd: I want to expand on my "If it moves, graph it" comment: Even if it doesn't move, you should graph it, because it might move!

  • .@mrembetsy: "

RT @cmsj: Etsy has 12 ops folks, 125 engineers, 350 employees total.
RT @lozzd: Expanding on my "If it moves, graph it" comment: if it doesn't move, you should graph it, because it might move!
@itarchitectkev: Etsy chose PHP and MySQL so Dev and Ops could understand the stack and everyone can contribute if they wanted to. #VelocityConf

  • .@mrembetsy: "Developers become on call: 'why should I be the only people woken up at 3am?'; code issue? call Dev 911"
  • .@mrembetsy: "Constant A/B testing to get feedback on feature effectiveness; feature flags & ramp-ups; Schema Chg Thursdays
  • .@mrembetsy: "
  • @kief: @etsy deserve props; v.few companies whose business is not selling tech share a fraction as much of their tech lessons
  • .@mrembetsy: "We stopped changing schemas whenever we feel like it; now once/week, we batch on Schema Chg Thu"
  • .@mrembetsy: "2010 Takeaways: decr tech used by Dev; Incr tech visibility thru org; Dev takes resp for code release
  • Jeez, anyone else getting teary eyed of the Hero's Journey being told by @mrembetsy/@mcdonnps? Jeez.
  • .@mrembetsy: "2010 Action Items: don't guess what's wrong: graph it; keep human mgmt a priority; document stds/practices@botchagalupe: @etsy giving customers a way to go into a feedback mode on the web site... #brilliant
  • RT @botchagalupe: @etsy giving customers a way to go into a feedback mode on the web site... #brilliant
  • @stack72: Developers take responsibility for rollbacks when things go wrong with their features at etsy
  • RT @stack72: Developers take responsibility for rollbacks when things go wrong with their features at etsy
  • .@mrembetsy: "2010: The Reaping: The day non-std tech died; Monogo/scala/coffeescript killed; sprouter killed; python dies
  • .@mrembetsy: "# of sprouter calls to zero by opportunistically removing it, bit by bit. Finally, it dies, too"
  • .@mrembetsy: "2010 becomes a tech driven company; tech executives promoted"@stack72: Etsy killed mongo on their dev stack in 2011. Also killed scala and coffeescript too #VelocityConf
  • @scoobiedoobie: RT @mpaluchowski: RT @mpaluchowski: If you deployed it, you're responsible for it.
  • .@mrembetsy: "Tech contribution: Deployinator released, used by Rackspace; statsd, logster
  • .@mrembetsy: "Every engineer responsible each yr: speak at conf, post blog, release code" (check: mission accomplished 2012)
  • .@mrembetsy: "Config mgmt: finally xforms use: signif increased agility by incr conformity; allowed growth of team"
  • .@mrembetsy: "We move from svn to git; took one wkend where everyone worked on transition; important prod gets done (srch)
  • .@mrembetsy: "Increase signal/noise; write Schemanator to automate schema chgs to reduce risks; infosec (SCRAM team)
  • .@mrembetsy: "Shoutout to @ngalbreath on DevOpsSec (and his DevOpsDays Austin) talk"
  • .@mrembetsy: "2010 Holidays: graphs: GMS/sec, Naglight; 1st game day to test failure before happen; financial reports to all
  • .@mrembetsy: "2011 Takeaways: year of the tools; DevTools adds 3 engrs; Prepped for PCI audit in 6 wks: $526M revenue"
  • .@mrembetsy: "Senior mgmt at tech company should be tech focused; config mgmt even if u only hv 2 severs
  • .@mrembetsy: "Don't let #pci change the culture of the company; separation of duty reqs can still be fulfilled"
  • .@mrembetsy: "massive hiring/growth; Incr focus on community; uniq engr challenges force non-std tech: redis, virual ci test
  • @stack72: "do not let technical challenges & compliance (pci, sox) change your culture" #VelocityConf
  • .@mrembetsy: "Invite other teams to office to chat: tumblr, etc; ops becomes more involved in external info exchgs"
  • .@mrembetsy: "Hack weeks rock: we built 'Guess That Admin': we keep score of who recognizes most people"
  • .@mrembetsy: "Guess That Admin: randomly select LDAP record; Eatsy: mandatory communal gathering"
  • @stack72: "You should be doing hack weeks" I'm sure this goes for ops and not just dev #VelocityConf
  • RT @stack72: "You should be doing hack weeks" I'm sure this goes for ops and not just dev #VelocityConf
  • @balepcdev: Funny to see how Ops advocating their laziness :) #VelocityConf
  • RT @balepcdev: Funny to see how Ops advocating their laziness :) #VelocityConf
  • @mpaluchowski: Love the idea of mini-conferences with befriended teams from other companies.
  • Like Toyota plant tours. Awesome RT @mpaluchowski: Love the idea of mini-conferences w/befriended teams from other companies.
  • .@mrembetsy: "Work in progress: dev boredom curbed by allowing inter-team xfers; international stuff; open src all things"
  • .@mrembetsy: "by planning for open source, we proactively architect/santize before we write tools" Nice
  • .@mrembetsy: "2012 action items: Know when not to try something; focus on perf early"
  • .@mrembetsy: "Some Predictions Overhd: everything will fall aprt at 500 employees. didn't happen: theyve been proactive"
  • .@mrembetsy: "Will create better/innovative ways of communicating; building tools is Etsy way; sharing everything"
  • .@mrembetsy: "Etsy is now B Corp, certified to follow sustainabililty, etc.; Belief: happy = successful"

  • Kudos to @mrembetsy/@mcdonnps. Fantastic talk!!! cc @stack72: Hearing how etsy built a culture is amazing. Lots to learn here. Kevin Costner was right, build it and they will come #VelocityConf

  • @allspaw+100 RT @stack72: Hearing how etsy built culture is amazing. Lots to learn here. Kevin Costner right, build it & they came #VelocityConf

  • Amazed anyone found this room! Excited! RT @botchagalupe: An englishman talking about queuing "iLoviT" .. @ph

  • Pattern: @unixdaemon: "job queues allow you to do slow work outside of the http request" @ph

  • RT @unixdaemon: "job queues allow you to do slow work outside of the http request" @ph

  • .@ph: "Anything not shown on page load: no one will notice: email, tweets, external apis, webhooks: even if it's only couple of secs"

  • .@ph: "Use job queues for: email, tweets, external apis, webhooks: even if it's only couple of secs"

  • .@ph: "User exp: 500-200ms threshold for "wtf? click reload"; follower: 1-10s; stranger: 1-2m old data is just fine"@scoobiedoobie: RT @itarchitectkev: RT @itarchitectkev: Nice hearing about developing the right culture anf giving back to the community. @etsy rocks. #VelocityConf

  • Astonishing. RT @itarchitectkev: Nice hearing about developing right culture & giving back to community. @etsy rocks. #VelocityConf

  • @stack72: Service = code + infrastructure #VelocityConf

  • .@ph: "Problem: if you're outside HTTP request [like in job queue], can't take adv of 500 error handling"; drop/requeue?

  • .@ph: "Resilient system soln: reserve_job(), remove() upon complete (prob: can't guarantee job only run once; bad for email)

  • .@ph: "Choice: run job 0 or 1; or run job 1 or many"

  • .@ph: "Choice: slow reliable queue AND Fast unreliable queue"

  • .@ph: "Idempotent: ok to run twice: resizing photo; Not: sending emails/tweets; Almost external API"

  • .@ph: "pattern: update canonical source in request: queue idempotent job to update denormalized copies"

  • .@ph: "#4: jobs don't run in order: no shared state between workers: update user 20; NOT reindex user 20 w/these attributes"

  • .@ph: "jobs may create jobs when finished"

  • .@ph: "#5: lock contention hurts; queue churn when users does lots of ops and leaves"

  • .@ph: "#6: alerting is hard

  • .@ph: Despite not wanting to talk tools, here's the list of tools @ph mentioned. Helpful!

    https://pbs.twimg.com/media/A4XdD9RCMAAbrOv.jpg

  • @phrawzty: @alq breaking down the webops cycle. #velocityconf http://t.co/mlQy8I27

  • @allspaw: “@AndrewBrockway: SPOF-O-Matic - superb tool for finding 3rd party blocking scripts. #velocityconf http://t.co/LxmiqeQy” /cc @sethwalker

  • @stack72: RT @mcdonnps: RT @mcdonnps: Slides are up from @mrembetsy and my's presentation on Continuously Deploying Culture at @Etsy #velocityconf #devops http: ...
    http://www.slideshare.net/mcdonnps/continuously-deploying-culture-scaling-culture-at-etsy

  • https://twitter.com/mcdonnps/status/253848570216726530
    RT @mcdonnps: Slides from Continuously Deploying Culture at @Etsy #velocityconf
    http://www.slideshare.net/mcdonnps/continuously-deploying-culture-scaling-culture-at-etsy

  • At risk of looking stupid: I've heard mainframes kicked ass at queueing. How do modern queues compare to mainframes? @ph