2017/01/05 US DHS Day with Jason Cox and Topo Pal 2

by Gene Kim on

#devops

2017/01/05 US DHS Day with Jason Cox and Topo Pal 2

Jason Cox

  • 3 cabs per week to deploy to Dev deployments!
    • 50 people in each CAB meeting: expensive meeting
    • just for a deploy to Dev environment
    • people scurried to look in CMDB -- "how about just in the last 5 years, just to constrain the search?"
    • "I move that we disband this CAB meeting."
  • one of my people asked at the CAB meeting, when was the last time we said no?
  • no one could remember?
  • Weeks to deploy, weeks to keep it running
  • join the rebellion -- not to external , but ourselves
  • 2007: "operations specialists" inside of Disney Online to "systems engineers"
  • hundreds to thousands of servers: pets to cattle
  • "hey we have production issue" "hey jason, it works on my laptop; I don't understand; it works on my workstation"
  • containers helped us enforced 12-factor app practices, where we externalize all our endpoint information -- now we can make environment completely configuration and generated by deployment pipeline
  • "I have to challenge our teams to always look ahead; what are you experimenting with to replace this. Where do you want your vendors to go?"
  • story: 40TB storage for Disney Story app (2014)
  • story: 2016: save the children campaign with Avengers
  • Marvel Studios on Thu: "hey listen, we just got go ahead to start dev code; but we need to launch by Tue, because we'll be on Good Morning America"
  • this isn't wordpress: it's got RabbitMQ, SQL database, redis
  • every post: $5 to donate the children: help hundreds of thousands of kids
  • r&d lab used to steal laptops: they'd wire it up: we created R&D Cloud to enable hem to rapidly prototype
  • ops helped dev tune: 2.5K events/sec to 23K events/sec
  • complex Global Registration System: deploys went from 1 day to 1 min: everyone scared;

Topo Pal

  • our lead time was 6 weeks or 6 months — depends on who you are. (Haha)
  • I'm in highly regulated bank -- thanked to other banks, were going to be more regulated. Haha
  • Fallacy of SOD: we have developers and ops people who have worked together 15 years; one bad joke that offends both of them and they can collude
  • 2009: technology became core: became on of the three pillars: Creativity, Technology, and Globalization

  • Regulated: realty only career that people don't do things that are stupid or malicious

  • classical types of pipelines: "parallel pipeline that never meet, complex pipelines that no one knows the stages; and the clogged pipeline -- QA reassigned, bad performance test environment, all work stops after deployment"

  • "can't go faster: CABs, approvals; like hitting all the red lights on way home form work"