2013/03/07 Chicago ALM Day 2

by Gene Kim on

#almchicago

Reimagining The Application Lifecycle
Sam Guckenheimer, @samguckenheimer
Group Product Planner for Visual Studio at Microsoft

  • Sam Guckenheimer is Group Product Planner for Visual Studio at Microsoft. Up now on "Reimagining The Application Lifecycle"
  • Sam G: "The Agile consensus: 3 things: trustworthy transparency, reduction of waste, flow of value" (and focus on flow)
  • Sam G: While Agile was going, Stanford University professor Steven Blank wrote "Four Steps To The Epiphany" (2005)
  • Sam G: Blank thesis: "before developing products, first develop customers" (don't waste time on products no one wants)
  • Sam G: One of Blank's students was Eric Ries, wrote "Lean Startup" (2011); Ash Maurya wrote "Running Lean" (2012)
  • (In 24 hours, Sam Guckenheimer cited more authoritative books in 24 hrs than anyone I've met. Remarkable.)
  • Sam G: Two backlogs: one for Dev, and one for Ops; Dev backlog are made up of customer hypotheses & testable experiment
  • Sam G: Minimum viable product: 1) small batches, 2) small enough to validate cust need; get most info at least cost quickly
  • Sam G: most information, least cost, adequate sample size
  • Sam G: 2) define experiment as MVP
  • Sam G: 3) during construction, gather measurements from the MVP: deploy it,
  • Sam G: 4) measure the results (eg, "will customers pay money?": click on ad, buy product, or route to monetization)
  • Sam G: 5) gain validated learnings
  • Sam G: 6) preserve or pivot
  • Sam G: Because each cycle creates learning, you want to iterate as quickly/frequently as possible; need culture of learning
  • Sam G: MSN hypothesis: could recent search history increase user engagement (and then revenue per user visit)?
  • Sam G: MSN is an ad driven;
  • Sam G: hypothesis testing enabled by "exposure control:" could turn on functionality to targeted segments/populations
  • Sam G: Result #1: no difference in revenue; in fact, decline in page views; persevere or pivot?
  • Sam G: (in fact, whenever you change anything, you typically get decline in page views, b/c people hate change)
  • Sam G: Result #1b: unexpected discovery: increase in repeat visits; 12% click thru increase through 57 experiments over 1 yr
  • Sam G: Average experiment failure rate: 87% (!!); most experiments didn't work, but constant experimentation enables improve
  • WOW! Sam G: "Which Test Won?" site shows results of A/B testing. Wow. http://whichtestwon.com/ @hertling
  • PS: here's an article about how we did A/B testing for title/subtitle testing for "Phoenix Project" book: http://itrevolution.com/phoenix-project-google-adwords-title-subtitle-testing/
  • Sam G: "If you're doing startup, don't get Series A funding; go to your customer; Series A funding going down"
  • Sam G: "By 2017, the CMO will spend more money on IT than CIO" (high touch marketing requires high tech)
  • Sam G: "Let's look at 2012 US Presidential election: On Elec Day: CNN said, "too close to call; 146 undecided electoral votes"
  • Sam G: "Election day: 11/6/2012: Candidate Romney even posted invitation to the victory/inauguration;"
  • Sam G: "Team Romney had built a 'killer app' for managing volunteers: Project Orca; went live on Elec Day; disastrous failure"
  • Sam G: "Team Romney: app provided millions of users w/65 page PDF, instructions; Comcast shut down network access"
  • Sam G: "Project Orca had no beta test; failed spectacularly on Election Day, wen Team Romney needed the app most"
  • Sam G: "OTOH, other narrative Nate Silver: predicted 50 of 50 state elections; started w/baseball talent minor leagues
  • Sam G: "Silver published election predictions every two wks; spread always diverged (not close race); placed 90% odds on Obama
  • Sam G: "Team Obama: on avg, every fundraising email tested 18 times on smaller groups; worst letters 15% of the best letters"
  • Sam G: "Team Obama: most effective subject line for email campaigns: 'Hey'" (Wow.)
  • Sam G: OMG. "The 'hey' subject line was not even meant to be valid -- was meant to be a test/control subject line" (Haha)
  • Sam G: Harper Reed, Team Obama CTO: ran 6000 Monte Carlo simulations; had engrs from Facebook, Twitter, Google, MSFT
  • Sam G: "Team Obama: killer app: Quick Donate: enabled one click to donate for mobile browsers; multi-channel pmts: SMS
  • Sam G: "Quick Donate: 4x more freq donations, 3x higher donations"
  • Sam G: "Ars Technica: Team Romney outspent Team Obama by 7x" (!!!) "They outsourced it, defined it in contracts, wrote reqs"
  • Sam G: Sorry, to clarify: "Team Romney wrote specs, outsourced dev, had contracts; but obviously lost in the battlefield"
  • Sam G: "Team Obama had Game Days: they rehearsed worst scenarios repeatedly; like Netflix Chaos Monkey; created antifragilty
  • Sam G: NYT Magazine article 2 weeks ago: "Can The Republicans Be Saved From Obsolesence"; conseq are huge
  • Sam G: Look at Groupon:
  • @theWPConsortium: This idea of knowing your disaster recovery bcause you reheaese it is known as antifragility @samguckenheimer
  • PS: Sam G is @samguckenheimer
  • .@samguckenheimer: "Ousted Groupon CEO: 'my one regret that I let the lack of data override my intuition'" (IT is important)
  • .@samguckenheimer: "Key lessons: 1) address technical debt (unclog the drain); 2) amplify flow of value; 3) reduce cycle time
  • .@samguckenheimer: "2 metrics that matter in ALM flows: cycle time (backlog to production) and MTTR (when you break it)"
  • .@samguckenheimer: "Other key measure: how long to get learnings back into the backlog"

  • Q: "how do you bring these ideas to enterprise w/18 mo SAP waterfall?" A: "Contrast Project Phoenix vs. Project Unicorn (Forrester would call a 'system of engagement' that takes little thing that can be loosely coupled to the enterprise app, and show quick results, and work in parallel and work as a sort of lighthouse; through a lot of these Unicorns,

    • those 18 month cycles will/must of change: systems of engagement vs. record
    • The business pattern will start with the edges, and then work towards the core: that's how the business results will be generated
  • Q: "please comment on Team Obama who did 40 deploys on Election Day"; A: "By rehearsing, you have no fear to deploy. Two things: 1) you deploy to a smaller cohort, observe that cohort w/immediate monitoring; 2) that monitoring allows you to test production"

    • .@samguckenheimer: "In reality, whenever you have any site with persistent data, no one gets to rollback." cc @kartar
    • .@samguckenheimer: "Obvious lesson: you have to constantly rehearse and practice deploying and fixing" cc @kartar
  • Wow. @bmc_devops: u'r right: Tale of Team Obama vs. Romney is epic tale of IT, as evidenced by amazing @samguckenheimer talk

  • Notes from .@samguckenheimer amazing PPT here