by Gene Kim on
elocityconf: FYI - @souders and @allspaw are planning something #awesome to k
ick off #velocityconf keynotes. Catch it
:30a PT; http://t.co/KyZJ0onG.
Nice! RT @skullboxx/@velocityconf #velocityconf is streaming keynotes and selected sessions Tue + Wed - http://t.co/eq50lhoZ
Jay Parikh, Facebook: "Building For A Building Users"
Parikh: "500M ppl visit site/day; in next 30m: 10TB logs -> Hadoop; 105TB data into Hive; 6MM photos; 160MM newsfeeds"
Parikh: "in next 30m: 108B MySQL queries; 2.8T cache ops"
.@jayparikh: "every empl goes thru bootcamp: 1st day: fixed prob in local Facebook instance, depl to 100M users next day"
.@jayparikh: "engrs can pick what team to join after 6 wks of bookcamp; no hiring committees"
@mikeodea: Facebook: 6 billion mobile messages (!!) every 30 minutes
RT @mikeodea: Facebook: 6 billion mobile messages (!!) every 30 minutes
.@jayparikh: "2nd Facebook principle: Move Fast: hundreds of code pushes daily: graph of commits/dayhttps://p.twimg.com/AwU7NjsCAAA3d_V.jpg
.@jayparikh: "Normally, # of code commits goes down over time: ours has gone up
.@jayparikh: "Gatekeeper: like A/B testing, we phase-in features to users: 500M+ gatekeepr chks/sec: fast flow releases"
.@jayparikh: "Lots of
@nphase: FB Perflab: Performance tests every commit against real traffic before exposing to prod
.@jayparikh: "Soln: in 1 hackathon, moved to shared mem, so cache replenish takes days, sometimes hours: sped up flow"
.@jayparikh: "constant focus on automation makes life more fun: repetitive work taken away by tools: equiv to 350 ops engrs
@xthestreams: Scaling to a billion users at facebook is not down to a single person, it's a function of a large number of small tweaks
RT @xthestreams: Scaling to 1B users at facebook is not down to a single person, it's a function of a large # of small tweaks
.@jayparikh: "Facebook 3rd Principle: Be Bold: iterate rapidly, then launch to 1B users overnight
@xthestreams: "People, tools and way way down the list, process." Facebook on automation
Wow. RT @xthestreams: "People, tools and way way down the list, process." Facebook on automation
.@jayparikh: "Forest City, NC data center didn't just copy Prineville, OR; chged everything: network, hw, sw" (Wow.)
.@jayparikh: "chgs included moving HD from back to front; 2x CPU; 40% more throughput"
@mark_barger: RT @nphase: FB Operates 15,000 servers per datacenter technician.
John Rauser, Amazon
Mike Brittain, Etsy
Bryan McQuade, Google
@chrismunns: Sad to have missed #velocityconf this week, but YOU NEED to see Paul Hammond's slides on startup infrastructure http://t.co/rltrFkwS
.@souders: "use of Flash down: 50%->44%; size of av]
Dr Richard Cook: Royal Institute at Sweden
James Turnbull: Myths of Rollback
BTW, I saw 4chan founder talk once. Awesome. Funniest slide was 4chan CAPTCHA spoof of line noise. Wish I could find it
.@mtnygard "has a checkered past: I've been an app dev, architect, web ops; I've seen how code survives over years: not well"
.@mtnygard: "We run tightly coupled systems w/non-linear feedback loops; not good"
@jbarciauskas: economic boundary: making money, safety boundary: not harming, production boundary: producing at maximum capacity
RT @jbarciauskas: economic boundary: making money, safety boundary: not harming, production boundary: producing at maximum capacity
.@mtnygard: "Antipattern #1: integrations are #1 risk to stability: every process call can/will kill you; even db calls"
.@mtnygard: "Ex: random 5am db hangs: because firewall expiring LIFO db connection" (ok, this story defies 140c limit)
@jschauma: "Firewalls exist to break TCP/IP in every way possible." @mtnygard
RT @jschauma: "Firewalls exist to break TCP/IP in every way possible." @mtnygard
.@mtnygard: "Patterns: Circuit Breaker, Use Timeouts, Use Decoupling Middleware, Handshaking, Test Harness"
.@mtnygard: "Failure Pattern: Chain reaction: where all front end servers drop in seq; common in app servers, srch"
.@mtnygard: "Failure Pattern: Cascading failure: failure moves vertically across tiers: SOA: common in enterprises, SOA"
.@mtnygard: "Failure Pattern: Blocked Threads: all req threads blocked; most comm pattern: oft due to homegrown thread libs"
@iandelahorne: .@mtnygard "any websphere users out there? Yeah, you have hung threads right now"
RT @iandelahorne: .@mtnygard "any websphere users out there? Yeah, you have hung threads right now"
Thank you! RT @xthestreams/@mrembetsy: @RealGeneKim rocking the twitters live from #velocityconf. It's good.
Self inflicted Denial of Service: Best Buy XBox 360 promotion: "went down so fat, we thought it was power failure"
PS: Someone asked how I take notes & tweet at same time: @_flynn wrote http://tweetscriber.com for iPad. It's free!
.@mtnygard: "@xthestreams: Scaling ratios between tiers should be tested asymmetrically. Overload the back end with front end requests. #velocityconf @mtnygard
RT @xthestreams: Scaling ratios betw tiers should be tested asymmetrically. Overload back end w/front end requests. #velocityconf @mtnygard
.@mtnygard: "Stable Patt #1: Circuit Breaker: most faults req hours to correct (cable knocked loose, app server down)
.@mtnygard: "Circuit Breaker goal: cut off features that aren't working correctly; "cool off" demand"
.@mtnygard: "Citing Netflix Nov blog entry: showing state of all circuit breakers -- is valuable dashboard"
@thewebvy: Failures often happen on a human time scale, your retry timeouts should reflect that. Use seconds, not milli or nano.
RT @thewebvy: Failures often happen on a human time scale, Retry timeouts should reflect that. Use seconds, not ms or ns.
.@mtnygard: "Stable Patt #2: Bulkheads: save part of the ship, if another part floods: ex: dedicated backends for diff svcs"
@xthestreams: Queuing theory comes up again and again at
Fascinating. RT @xthestreams: Queuing theory comes up again and again at
.@mtnygard wrote "Release It!" -- a great book: http://www.amazon.com/Release-It-Production-Ready-Pragmatic-Programmers/dp/0978739213
@jschauma: When I'm all grown up, I want to give talks like @adamhjk and look sharp in pictures like @@markburgess_osl.
@secolive: RT @xthestreams: I'm starting to think that technical debt is code people are frightened to touch
@DuncNisbet: RT @jrauser: Video of my #VelocityConf keynote http://t.co/e8IloGTB. Thanks folks for indulging my passion for the history of science.
@teresadg: RT @laraswanson: "I love outages in a way that's probably not quite right" - @teresadg during her #velocityconf presentation
@xthestreams: @realgenekim the photos from your #velocityconf party are on Flickr http://t.co/viHpgQUm also emailed Han