@ondayd: RT RealGeneKim "#Monitorama sfedov Monitoring networks and user monitoring at Netflix https://t.co/9gZoM2bpsm"
@stephenash: Go to a tech conference and get an art history lesson along the way. Thanks @lukedemi for the great talk.
"..and now I'm going to terminate all the EC2 instances in our Ohio WaveFront AZ. [click click click] After all, if your monitoring system going down, how will you fix what's broken?" (gasps heard from around the auditorium. Nice demo!!)
@crayzeigh: Next up’s a story from @DaveJosephsen talking about Schema on Read for logging systems. #monitorama
@jesusrafael: RT @LibbyMClark: Engineering doesn’t just mean writing software. It’s the art of solving problems. We can’t keep rewarding complexity. Reward people for solving problems. @lizthegrey#Monitoramahttps://t.co/4cqrlo4AyC
@crayzeigh: --- @DaveJosephsen System logs are a running narrative of what happened in a system. But a superior story is understanding “how it is” rather than “what happened’ @DaveJosephsen#monitorama
@lizthegrey: [ed: this is an old military yarn that I'm going to not livetweet and just let folks watch if they're interested in it] #monitorama
@crayzeigh: --- @DaveJosephsen A story is told by @DaveJosephsen about a particular marching cadence that circles through anglican armies and comes from an old english rhyme. Infantry privates want “Beer! Beer! Beer!”… it tells how it is. #monitorama
@PuckPuck: I got to be the @WavefrontHQ chaos monkey today live on stage at #monitorama ... most fun I ever had in a live demo. #BeachOps
@wiredferret: #monitorama@DaveJosephsen: I am HERE for any talk that bothers to define email for The Youth.
@crayzeigh: --- @DaveJosephsen All of the logging and log storage tooling and practices from the elder pipe grep to modern stacks are tooled toward answering “what happened” but we really wanted a way to have a conversation about “how it is” @DaveJosephsen#Monitorama
massive bounce rates;
@acedrew: --- The ever entertaining and informative @DaveJosephsen sharing how he built a logging system to help you understand "How it is" vs "What happened" #monitoramahttps://t.co/OuXAfN4hIM
more bounces? No. More verbose bounce error messages!
to book: Tesla story: probe drained battery, and 12V lead acid battery: would no longer start or charge
never met a database that I didn't eventually want to set fire to. I'm a Luddite
@mattstratton: “I have the word ‘senior’ in my title, so I know I’m supposed to be jaded as hell” - @DaveJosephsen#Monitorama
@mipsytipsy: --- it's a terrible fucking experience when your human is copy pasting ids between your three fucking pillars and you're paying 3x for the privilege of being a human unix pipe.
choose arbitrarily wide structured events. choose self respect. choose life. #observability
- @grafana: --- Missing #Monitorama? Check out our preview of the conference, including a Loki talk by @tomwilkie and @geekdave's latest on how to explain monitoring to your kids -- and your colleagues. https://t.co/I9jkvNsRn5
- @acedrew: --- "I'm a complete luddite when it comes to any sort of paxos database chicanery, or, Computer Science, whatever you want to call it" @DaveJosephsen making the case for Schema on Read logging systems at #monitoramahttps://t.co/UdcfYbyXDG
- @crayzeigh: --- @DaveJosephsen The choice was to deal with schema on read. We store just data, and build the schema based on the query.
Query -> Map -> Schema -> Data
This provides a better query experience for users. @DaveJosephsen#Monitorama
- @lizthegrey: --- [ed: oh no no no :( using distributed regexes over raw only loosely structured logs = :( :( so much performance pain. seriously use @cribl_io instead] #monitorama
- @crayzeigh: --- @DaveJosephsen The logs are dumped into files on S3 and then can be queried a number of ways on the other side. But it’s not just raw logs, Everything expects JSON. Used Fluentd to process logs. @DaveJosephsen#Monitorama
- @petecheslock: --- The only downside of the tools like Athena is lacking the ability to hunt and search across your log files, which of course why Elasticsearch became so popular. That's why @CHAOSSEARCH has the ability to do Schema on Read but still provides hunting and searching.
I had the same problem that @DaveJosephsen describes of super expensive queries ($150/query) on Apache Hive. I had same problem using Google BigQuery. Wasn't smart enough or didn't have enough time to figure out how to make queries scan fewer rows.
@petecheslock: --- I like where things are going with Presto/Parquet, but ultimately when dealing with sparse datasets can be expensive and difficult to scale.
@crayzeigh: --- @DaveJosephsen Columnar stores helps speed of query and reduces the amount of scanning for cost. [this is also how Elastic deals with storage and retrieval but consider it schema on write as we map fields to understand how to query and analyze them most effectively] #Monitorama@DaveJosephsen
@markmadsen: --- Enjoyed @DaveJosephsen 's #monitorama talk, particularly the irony of hating databases and having to create the functional equivalent of a database to get the work done, even addressing the problem of "I lost an event"