5/2: Nick Galbreath

by Gene Kim on

#devops @ngalbreath

This is an interview that @myownroom did with Nick Galbreath ([http://twitter.com/ngalbreath](@ngalbreath)].

He is the directors

of engineering at Etsy, managing a group of engineers charge of information security (authentication, fraud).

  • Listening to @ngalbreath interview, done by @myownroom at DevOpsDays.
  • Video will be posted on IT Revolution blog, but in meantime: http://m.youtube.com/watch?v=r53ErYe9Uhs

  • He did a talk at DevOpsDays Austin on DevOpsSec: applying DevOps principles to infosec (lots in common w/@wickett)

  • Underlying principle: trust but verify (works on authentication, fraud, monitoring, compliance)

  • "Get used to constant probing, automatically correlate

  • login failures correlated to IP addresses

  • "goal: make things visible, help train Dev on infosec, so infosec not only people who have to say 'no'"

  • "trust but verify: people, machines, processes, continuous integration, metrics like MTTR"

  • "more threats: phishing attacks on Etsy staff (corporate & home systems)"

  • "fast DevOps deployment cadences helps infosec, too"

  • "undergrad CS degrees focused only on dev; missing all the ops & infosec aspects"

  • "message to Dev: there are only 2 types of code: code in prod & code not in prod; it's not an abstraction"

  • "my vision of DevOps in 2-3 yrs? DevOps disappears, b/c it's standard practice"

  • "right now, #devops has such a huge emphasis on tools; tools not the goal, though. it's end to means"

  • (BTW, can't wait to see his talk: obviously amazing synthesis of #devops & #infosec) cc @mortman/@joshcorman/@wickett

  • "DevOps movement borrows heavily from kanban, Toyota mfg metaphors, etc. Loved book Toyota Way (by Jeffrey LIker)"

  • (PS: Jeffrey Liker was one of first Shingo Prize winners; DevOpsCookbook coauthor @mikeorzenleanit latest winner)

  • Watching @nbalbreath's #devopsdays Austin talk at http://www.ustream.tv/recorded/21568549
  • @earnestmueller mc'ing
  • Uh, that's @ngalbreath...
  • "When I asked @allspaw, 'what is DevOps?' answer was, 'oh, it's all that stuff we did for years at RightMedia'"
  • "Driver there was that we had a very fragile app, so we had to do lots of very small changes"
  • "My primary DevOps Principle: Trust But Verify; yet acknowledging that we're working in complex system"
  • "Failure happens even when no one does nothing wrong; & yet how can we increase rewards we get while reducing risk?"
  • (BTW, thanks to @BMC_DevOps to recording the entire conference on Ustream!)
  • He's describing again the spirit of @allspaw's blameless post-mortems (vs. mean time to blame/claim innocence)
  • "Even tho infosec is different, incredible similarities to Ops: Unreviewed Code going out, Untrusted Data coming in"
  • "This makes stability/responsibility complicated, even more so if there's walls between silos"
  • "It's amazing things work at all when sites are connected to Internet; Ops is difficult for similar reasons"
  • (Even tho it's poor form to complain about things when they're free, holy cow, Ustream is slow. 10m waiting...)
  • Infosec like ops: "We both have latent problems that we don't know about: it just hasn't been exploited yet"
  • Infosec like ops: "Both disciplines are prone to be risk averse: easy to say 'no' b/c of consequences of failures"
  • Infosec like Ops: "Both are service organizations" (although, I'd argue, in the ideal, both are part of the team)
  • "Best infosec capability: ability to quickly deploy/rebuild infrastructure" (applications, schema, data, restore)
  • "Altho, @Etsy talks alot about their deploy rates; for infosec, ability to rebuild is critical capability"
  • "Bad when IT staff sez, 'not sure if we need to apply this patch": systems r so complicated, not sure what it'll do
  • "When you can quickly rebuild, you can just do the patch on fraction of systems & see what it does" cc @joshcorman
  • "New fave infosec question to org: how quickly can you deploy and rebuild your systems?"
  • "High deployment rates force automation of configuration; forces manual, tedious steps out of system"
  • "Some say: bullshit: how can rapid rate of change result in better infosec?"
  • "It's better than this: 'we'll rush out that infosec fix in 6 wks': that vendor is no longer at Etsy"
  • "It's ok if we have few extra firemen; I'm more concerned that we won't know about fire until house burnt down"
  • "We look for segfaults on servers: ask why things are crashing: may be bad code or successful attack probes"
  • "We look for production server 500 errs, db syntax errors: catches serious problems that could lead to SQL Injection
  • @RealGeneKim correction: log all "UNION ALL" in user input from query string, post vars etc to find who is poking you with SQLi attempts
  • @NGalbreath @RealGeneKim a much better way to detect SQLi forthcoming but this gets you started to make security a visible non binary event
  • "Our crappiest simple test that works: find and log all instances of UNION ALL in PHP code":
  • "Looking for UNION ALL in user input will undercount SQL Inj by 20x, but has low false positive rate"
  • "Having your site attacked all the time is a gift: it makes risk visible, great for educating Dev, etc."
  • "Use all the logged attacks to start shaping your test plan: focus on what attack surface is being poked"
  • "Assert your site using puppet/chef/etc: this page always should be SSL, this port never should be open, etc."
  • "Integrate into your continuous integration/test environment: eg, run ClamAV to assert malware not in our PDF files
  • "Integrate into post-mortems: security issues are always P1 and P2"
  • "Product Security is run as an inhouse consultancy: anyone interested, we'll send to training"
  • "SQL Injection is 20-30 yrs old; it's a culture & knowledge prob; we'll train any developer who's interested"
  • *To expand perimeter [& elevate avg level of hygiene], we hold Tuneup Days: ppl bring home computers & we fix them"
  • "Our post-mortem process: blameless, any interested party, create timeline, how detected, publish countermeasure"
    TODO:

  • Look for twitter status: posted ustream video link