2014/05/06: Monitorama Day 2

by Gene Kim on

#monitorama

Next: "Auditing all the things: The future of smarter monitoring and detection", Jen Andre (@jenandre)

  • @TerribleDev: "Auditing all the things": The future of smarter monitoring and detection #monitorama talk starting soon....
  • .@jenandre: "If u're logging everything & get hacked, u don't need to pay someone $600/hr for forensics [to get a clue]" #devops
  • @ashedryden: Anytime a company says “we found no evidence […]” they are bad at the job of finding evidence or are motivated to not find it.
  • Ha! RT @ashedryden: Anytime company says “we found no evidence [of being hacked]”: r they bad at job, or not motivated to find?.#monitorama
  • @randomfrequency: “Rainbows! Ponies! Cloud!”
  • @TerribleDev: Should you care if you get hacked? Depends on your product... #monitorama pet Snapchat... Probably not http://t.co/Ii6APYtNN3
  • I'm so excited to see #devops/#infosec boundary-spanners like @jenandre. /cc @joshcorman/@ngalbreath/@wickett
  • .@jenandre dropping words like active defense, access ctrls, linux auditing...; Am I at right conference? hahaha. Awesome.
  • @benzobot: Security monitoring: it’s not if you get hacked, but when, and how big is the threat?
  • @TerribleDev: "if someone get credentials to your s3 bucket, can you detect if someone else is looking at that stuff"
  • RT @TerribleDev: "if someone get creds to your s3 bucket, can you detect if someone else is looking at that stuff" #monitorama @jenandre
  • @bridgetkromhout: Terrifying: "a PHP developer with ssh keys for prod who uses the same passwds everywhere." @fun_cuddles #monitorama http://t.co/8ygf0BY3Oc
  • RT @bridgetkromhout: Terrifying: "a PHP dev w/prod ssh keys uses same passwds everywhere." @fun_cuddles #monitorama http://t.co/8ygf0BY3Oc
  • https://pbs.twimg.com/media/Bm97-UBCMAAO-D5.jpg
  • RT @bridgetkromhout: Terrifying: "a PHP dev w/prod ssh keys uses same passwds everywhere." @fun_cuddles #monitorama http://t.co/8ygf0BY3Oc
  • Oops. Presenter Jen Andre is @fun_cuddles... not @jenandre. Sorry!
  • @rberger: @fun_cuddles talk on Audit All Things talking about all the things I should do for core security auditing #Monitorama
  • @benzobot: 10 minutes into @fun_cuddles talk and I too want to AUDIT ALL THE THINGS
  • Win. :) @joshcorman/@wickett/@ngalbreath RT @benzobot: 10 min into @fun_cuddles talk & I too want to AUDIT ALL THE THINGS
  • @mary_grace: #DevOps Against Humanity cards at #monitorama #womenintech bfast this morning. thanks… http://t.co/RBHpha8s5E
  • Go @bridgetkromhout! RT @mary_grace: #DevOps Against Humanity cards at #monitorama #womenintech bfast. Thx! http://t.co/RBHpha8s5E
  • @Xorlev: Redhat auditd: "occasionally wtf-y" #monitorama http://t.co/gL9Q5hwa5x
  • Haha. :) RT @Xorlev: Redhat auditd: "occasionally wtf-y" #monitorama http://t.co/gL9Q5hwa5x
  • https://pbs.twimg.com/media/Bm99BagCEAAj3bl.jpg
  • @benzobot: Kernel logging without rate limiting can crash your box…yep, been there before…
  • @danslimmon: Thanks to @fun_cuddles I now know how to track what somebody's doing after they "su - root"
  • @TerribleDev: use pam_loginuid to setup session tracking on unix logins
  • @rberger: @funcuddles pamloginid will add a session id to all executed commands so you can link real user to sudo’d commands #Monitorama
  • @danslimmon: Thanks to @fun_cuddles I now know how to track what somebody's doing after they "su - root"
  • @rberger: RT @benzobot: Yes! Logging sanity with JSON. #monitorama @fun_cuddles
  • @Xorlev: Awesome audit & security information from @fun_cuddles, super knowledgeable.
  • .@fun_cuddles: "Most orgs take yrs to discover they're hacked, sending data to Syria...Constantly look for signs of compromise
  • .@fun_cuddles: "Once attacker has creds, diff to detect; look for anomalies: running gcc, copy bins to /lib, weird traffic?
  • @benzobot: Again - collect data, then analyze it for abnormalities and patterns
  • @TerribleDev: "as an ops person you should have unique insight into what should be going on in the system"
  • RT @TerribleDev: "as an ops person you should have unique insight into what should be going on in the system"
  • Dear world: anyone saying #devops is bad for #infosec needs to wake up & watch @fun_cuddles talking abt prevent/detect/correct
  • @benzobot: I really need to get going on rebuilding our realtime system so that I can throw ALL THE DATA at it
  • Although, any #devops team w/o someone w/#infosec expertise may be dangerously asleep at the wheel, too... @betabit: "I don't see hacking, as a trend, going down" #monitorama via @fun_cuddles
  • @betabit: "I don't see hacking, as a trend, going down" #monitorama via @fun_cuddles
  • Trivia: According to Verizon PCI Data Breach studies, most orgs leaked cardholder data for qtrs/yrs before finally detected
  • Trivia (contd): Why so long to detect cardholder data breaches? No one looked at security monitoring logs.
  • @hertling: Notes from Jen Andre (@fun_cuddles) on auditing http://t.co/pZWuzvsBpI
  • RT @hertling: Notes from Jen Andre (@fun_cuddles) on auditing http://t.co/pZWuzvsBpI

Next: "Is There An Echo In Here?: Applying Audio DSP algorithms to monitoring", Noah Kantrowitz

  • @abestanway: Oh, man. @kantrn talking about DSP algorithms for infra monitoring: #monitorama http://t.co/hMVkz9KfKy
  • https://pbs.twimg.com/media/Bm-BWUwCMAAe4GM.jpg
  • RT @abestanway: Oh, man. @kantrn talking about DSP algorithms for infra monitoring: #monitorama http://t.co/hMVkz9KfKy
  • .@kantrn asks "How many ppl noticed the 2Hz signal in this graph?" Haha. It's like I'm in conference-chat-roulette. Awesome
  • @curiousbiped: Those little infinities are not really helpful for computers. - @kantrn
  • @TerribleDev: Getting schooled using DSP with monitoring #monitorama http://t.co/qyCNKFOYPW
  • @drdabbles: That infinity right there? I wouldn't worry about that little infinity.
  • @adrianco: RT @aneel: using FFT/IFT to break out signal components in monitoring.. awesome
  • @adrianco: RT @danslimmon: OH MAN! @kantrn is talking about using Fourier transforms in ops! Strapping myself in for some badass math
  • @danslimmon: I would never have thought of using a low-pass filter & FFT to fight noise in ops. @kantrn is smart an #monitorama is inspiring.
  • @robertolupi: You can find traces of self-organized criticality, for example you can hear echos of cascading failures: http://t.co/7UwVP0hqhE
  • .@kantrn: "7 Hz sine wave, exactly what we want to see; see spectral leakage creeping in, primary signal, attenuated (which means reduced)" Hahaha
  • @drdabbles: All of this brings me back to working on radio-based Internet services. A dark time in my life.
  • @Opsworthy: Try not to pass out as your mind expands at this one bespoke, artisian monitoring conference #opsworthy
  • Yes, that. RT @Opsworthy: Try not to pass out as your mind expands at this one bespoke, artisian monitoring conference
  • Words from @kantrn: "7 Hz sine wave, yes, exactly what we want; spectral leakage creeps in, signal attenuated (which means reduced)
  • @robertolupi: DCT, wavelets for TSS storage at #monitorama Well, think about the GB/GFLOPS ratio now and in the future. Higher level formats are coming!
  • @danslimmon: Nagios's flapping feature is the worst low-pass filter ever, because they didn't know it was a low-pass filter - @kantrn
  • @drawks: Thinking of PID before the slide that said "PID" is the smartest I will feel all week. #monitorama #mathmindblown #dontworryaboutinfinity
  • .@kantrn: "DSP typically process 10^6 signals/sec; In Ops, we're typically sampling ~4 times/sec"
  • @adrianco: Great talk by @kantrn on signal processing at #monitorama - see Netflix Scryer for some real use cases.
  • @adrianco: RT @kantrn: Slides for my @Monitorama talk on signal analysis for ops are up at https://t.co/PA5LnzPYQc

Sponsor talks

  • Trivia fact: @nigelkersten was 1st Google SRE hired to cover internal Mac OS X security, fleet mgmt, LDAP, etc... (mispelled!)
  • (Holy crap. Huge number of @PuppetLabs job openings -- three screenfuls long....)
  • @RealGeneKim: “@nigelkersten: There were some good MacOps SRE folks there before I arrived :) /cc @salajander @curiousbiped @andybohm
  • @nigelkersten @RealGeneKim @salajander @curiousbiped One of the best teams I've ever worked with, hands down.

Next: A Melange of Methods for Manipulating Monitored Data, Dr Neil J. Gunther (@DrQz)

  • .@DrQz: "It's not about the math, tools; the data are trying to tell you something"
  • .@DrQz: "Ways to ask is Metric 1 and Metric 2 related in any way? Least Squares Fit Regression;
  • .@DrQz: "Most important scatter plot in hist: Hubble 1929: age of universe: further stars > faster"
    https://pbs.twimg.com/media/Bm-ImmICAAE76bW.jpg
  • @miah_: RT @ashedryden: Forgot to mention yesterday: I’m working on a book about increasing diversity on technical teams: http://t.co/zC5sZp9qXD #m
  • @miah_: RT @ashedryden: Forgot to mention yesterday: I’m working on a book about increasing diversity on technical teams: http://t.co/zC5sZp9qXD #m
  • @benzobot: “Treating data as something divine is a sin”
  • @danslimmon: Crucial difference that @DrQz is talking about but hasn't mentioned specifically: accuracy and precision are different things!
  • @Xorlev: RT @mary_grace: RT @danslimmon Man, @obfuscurity and the #monitorama team really nailed the program. Every talk has been excellent. >> +1. …
  • @xaprb: RT @danslimmon: As @DrQz points out, all measurements are wrong. A measurement is a reduction in uncertainty. That's all it is.
  • @nromdotcom: Really need to pee, but @DrQz is far too interesting for me to get up.

Misc

  • @dshack: RT @auxesis: Video for my #monitorama talk on the psychology of alert design is up: https://t.co/k5VfRHJnx7 slides are here too: https://t.…
  • RT @adrianco: Great talk by @kantrn on signal processing at #monitorama - see Netflix Scryer for some real use cases.
  • @TheSamoth: RT @bridgetkromhout: Slides from my "From Zero To Visibility" talk at #monitorama are here: http://t.co/LMe0IF8EEe; thanks for the great fe…
  • RT @ashedryden: Forgot to mention yesterday: I’m working on book on incr diversity on tech teams: http://t.co/zC5sZp9qXD
  • @ashedryden: Releasing this summer, and covers all the incremental steps needed for lasting change to increase diversity in tech companies.