2019/06/04: Monitorama Day 2d

by Gene Kim on


2019/06/04: Monitorama Day 2d

OMG. Lost all my data! Starting over

  • @ondayd: RT RealGeneKim "#Monitorama sfedov Monitoring networks and user monitoring at Netflix https://t.co/9gZoM2bpsm"


  • @stephenash: Go to a tech conference and get an art history lesson along the way. Thanks @lukedemi for the great talk.

  • "..and now I'm going to terminate all the EC2 instances in our Ohio WaveFront AZ. [click click click] After all, if your monitoring system going down, how will you fix what's broken?" (gasps heard from around the auditorium. Nice demo!!)

Up: @DaveJohnson:

  • @crayzeigh: Next up’s a story from @DaveJosephsen talking about Schema on Read for logging systems. #monitorama
  • @jesusrafael: RT @LibbyMClark: Engineering doesn’t just mean writing software. It’s the art of solving problems. We can’t keep rewarding complexity. Reward people for solving problems. @lizthegrey #Monitorama https://t.co/4cqrlo4AyC
  • @crayzeigh: --- @DaveJosephsen System logs are a running narrative of what happened in a system. But a superior story is understanding “how it is” rather than “what happened’ @DaveJosephsen #monitorama
  • @lizthegrey: [ed: this is an old military yarn that I'm going to not livetweet and just let folks watch if they're interested in it] #monitorama
  • @crayzeigh: --- @DaveJosephsen A story is told by @DaveJosephsen about a particular marching cadence that circles through anglican armies and comes from an old english rhyme. Infantry privates want “Beer! Beer! Beer!”… it tells how it is. #monitorama
  • @PuckPuck: I got to be the @WavefrontHQ chaos monkey today live on stage at #monitorama ... most fun I ever had in a live demo. #BeachOps
  • @wiredferret: #monitorama @DaveJosephsen: I am HERE for any talk that bothers to define email for The Youth.
  • @julian_dunn: These nerds are my nerds #Monitorama https://t.co/jQt2CJh2J6
  • @crayzeigh: RT @petecheslock: Yay. Love the shoutout to https://t.co/USeU3a1uF9 in @DaveJosephsen’s y’all about how to get data from your logs. #monitorama https://t.co/QfYkrRxOYN
  • @spazm: #monitorama : depth and humor, juxtaposed. https://t.co/c1KQgpJEO9

  • SparkPost: order of magnitude MTA bounces

  • @crayzeigh: --- @DaveJosephsen All of the logging and log storage tooling and practices from the elder pipe grep to modern stacks are tooled toward answering “what happened” but we really wanted a way to have a conversation about “how it is” @DaveJosephsen #Monitorama

  • massive bounce rates;

  • @acedrew: --- The ever entertaining and informative @DaveJosephsen sharing how he built a logging system to help you understand "How it is" vs "What happened" #monitorama https://t.co/OuXAfN4hIM

  • more bounces? No. More verbose bounce error messages!


  • to book: Tesla story: probe drained battery, and 12V lead acid battery: would no longer start or charge
  • never met a database that I didn't eventually want to set fire to. I'm a Luddite
  • @mattstratton: “I have the word ‘senior’ in my title, so I know I’m supposed to be jaded as hell” - @DaveJosephsen #Monitorama
  • @mipsytipsy: --- it's a terrible fucking experience when your human is copy pasting ids between your three fucking pillars and you're paying 3x for the privilege of being a human unix pipe.

choose arbitrarily wide structured events. choose self respect. choose life. #observability
- @grafana: --- Missing #Monitorama? Check out our preview of the conference, including a Loki talk by @tomwilkie and @geekdave's latest on how to explain monitoring to your kids -- and your colleagues. https://t.co/I9jkvNsRn5
- @acedrew: --- "I'm a complete luddite when it comes to any sort of paxos database chicanery, or, Computer Science, whatever you want to call it" @DaveJosephsen making the case for Schema on Read logging systems at #monitorama https://t.co/UdcfYbyXDG
- @crayzeigh: --- @DaveJosephsen The choice was to deal with schema on read. We store just data, and build the schema based on the query.

Query -> Map -> Schema -> Data

This provides a better query experience for users. @DaveJosephsen #Monitorama
- @lizthegrey: --- [ed: oh no no no :( using distributed regexes over raw only loosely structured logs = :( :( so much performance pain. seriously use @cribl_io instead] #monitorama
- @crayzeigh: --- @DaveJosephsen The logs are dumped into files on S3 and then can be queried a number of ways on the other side. But it’s not just raw logs, Everything expects JSON. Used Fluentd to process logs. @DaveJosephsen #Monitorama
- @petecheslock: --- The only downside of the tools like Athena is lacking the ability to hunt and search across your log files, which of course why Elasticsearch became so popular. That's why @CHAOSSEARCH has the ability to do Schema on Read but still provides hunting and searching.


  • I had the same problem that @DaveJosephsen describes of super expensive queries ($150/query) on Apache Hive. I had same problem using Google BigQuery. Wasn't smart enough or didn't have enough time to figure out how to make queries scan fewer rows.
  • @petecheslock: --- I like where things are going with Presto/Parquet, but ultimately when dealing with sparse datasets can be expensive and difficult to scale.


  • @crayzeigh: --- @DaveJosephsen Columnar stores helps speed of query and reduces the amount of scanning for cost. [this is also how Elastic deals with storage and retrieval but consider it schema on write as we map fields to understand how to query and analyze them most effectively] #Monitorama @DaveJosephsen
  • @markmadsen: --- Enjoyed @DaveJosephsen 's #monitorama talk, particularly the irony of hating databases and having to create the functional equivalent of a database to get the work done, even addressing the problem of "I lost an event"