2016/11/05: DevOps Enterprise Day 1b

by Gene Kim on


Up: Susannah Brown, Benjamin Chan: American Airlines

  • Brown: "to fully integrate American Airlines and US Airways, 1400 apps, will likely take 5-7 years
  • Brown: "Mergers are almost designed to foment distrust" Haha.
  • Brown: "Needed to get approval from FAA to fly as a single carrier; High stakes releases"
  • Brown: "Passenger reservation system integration; one of the most visible and complex customer-facing systems in airline"
  • Brown: "Single Flight Operating System; largest integration project; 85K empl affected, 500+ apps;
  • Brown: This is hilarious. US Airways didn't complete their passenger reservation systems; like merging >>>3<<< airlines!!!!
  • Brown: "more on scale of Single Flight Operating System (FOS): 15K hours of training; 24 depts affected"
  • Brown: "
  • Maya Liebman, CIO: We cant keep throwing bodies after every problem #does16 Brown @AmericanAir #DevOps
  • Brown: "Our goal: anyone can add value to any team from anywhere"
  • Chan showing CPU load for flight operations systems
  • Brown: "The need shared experiences; learning Bollywood moves
  • Brown: "2014: was connecting; 2015: collaboration and automation; created 2 toolchains for different dev stacks
  • Chan: "How many times do we have to spend hours to figure out environment differences that's breaking app functionality
  • Can: "OpenSSH issue: was able to patch syswtems in 5m"
  • Brown: "over 5K technology professionals across American Airlines" (did I get that number right?)
  • Brown: "Yeah, we're really the group that makes the airline fly. But the other tech groups are important, too". (haha)
  • Chan: "embarking on many programs to pay down technical debt, especially with partners"
  • Brown: "The Achilles Heel: automated testing: we're creating lighthouse apps to showcase success: flight attendant cust experience: 60 -> 86% test coverage
  • Brown: "building out Dev/Test envs: used to take 2 months: now self-service. You'd think devs would run to use it, right? Ah, culture chg."
  • Brown: "Making internal employees happy: flight attendent scheduling/position swap via mobile app; Training Mgmt replaced 5 old systems
    • RT @SeanDMackNYC: "Mergers are the perfect vehicle if you want to cement distrust" Susanna Brown and Ben Chan discuss DevOps in the m… https://t.co/fiPzTDSUmB
  • PS: My Phoenix Project co-author @kevinbehr is here, huzzah!

  • RT @cyetain: RT @semanticwill: “100% loaded organization w. no slack has no ability to learn, stagnates, dies, and becomes fertilizer.” #DOES16
    p - @BarbaraFrontera: RT @semanticwill: “100% loaded organization w. no slack has no ability to learn, stagnates, dies, and becomes fertilizer.” #DOES16

RT @DOESsummit: Ben Chan and Susanna Brown speaking on @AmericanAir #devops journey now at #does16! https://t.co/dtT4qTseUy

- - @scottprugh: "when chgs take more than 24 hours, outcomes are 600x worse!" WOW!

  • - @ericamorrison: "NLB proj was like living in Phoenix Project, like never before" I LOVED the suggested titles of CSG talk from NLB/Ops team!
  • @ericamorrison: "Manual work! Checking swap memory sizes by hand. To stop this, we brought in architects, devs, ops to work together to fix
  • @ericamorrison: "Everything lives in JIRA in 1 kanban board, spanning dev/test/ops and projects" (1 prioritzed work queue)
  • @ericamorrison: "Devs peer review, responsible for implemeting chgs
  • @ericamorrison: "My learnings: Ops is hard, chg process/volumes is daunting; chgs can be scarzy; arch is needed;
  • @ericamorrison: "My learnings: Ops forced to tolerate lots of pain" (HAHAHA!!!)
    RT @electriccloud: Ops from a Dev perspective - "Ops is hard" per @ericarmorrison @csg_i @DOESsummit #does16 #DevOps https://t.co/ZZzc8Ic7le

    • @scottprugh: "It takes so much work to get anything done; too many meetings, lack of esprit de corp; tried lots of experiments
  • @scottprugh: "In 2016, was asked to take over product operations; how do we get rid of dedicated Ops, empower teams to build/run together?

  • @scottprugh: "by combining teams across dev/ops, no more crazy RACI diagrams; real repsonsibility/accountability & motive for improve

  • @ericamorrison: "After reorg, I got a whole new view of DevOps; very educational, given my dev bacground"

  • @ericamorrison: "In Ops, so much more unplanned work; here's experiences with network load balancing

  • @ericamorrison: "Suggested titles:

- @DOESsummit: LOL - some "advice" given to @ericarmorrison at @csg_i #does16 #devops https://t.co/4aV9lJirbG

  • - @scottprugh: "largest print/mail factory in the US"; yes, that's me taking plant tour of it in Kansas! During huge blizzard
  • @scottprugh: "15 people from CSG here" An amazing team doing amazing things. Got to follow Scott around for 3 days
  • @scottprugh: Showing @testobsessed slide from #DOES15: "Amazing courage to shut down QA, b/c it created target rich env for probs
  • @scottprugh: "Functional orgs not only create long queue times & high effort to get anything; but worse, creates I-shaped, ovely specialized ppl
  • @scottprugh: "10x quality improvement; doubled releases to 4x yearly; daily deployments; 98% reduced MTTR
  • @scottprugh: "Current state: thep problem: 98% of incidents not release-related; 92% fixed by Ops; how do we fix?"
  • @scottprugh: "The prob: devs still don't see how their code runs/operates in Prod; RT @spinnakerio: RT @mdcarlson: Our original platform could only handle 2-3 deploys per day. Now using @SpinnakerIO and doing 90 deploys/day. @hmickman #d
  • RT @sureshgp: RT @ericpassmore: Cross-function teams: small batches, better work visibility , and engs see the whole playing field @ScottPrugh @ericarmo