Yahoo outage caused by Level3 BGP issue

By Steve Ragan Jul 9, 2007, 15:08 GMT

Over the weekend, there were scattered reports about an outage that started Friday and lasted into parts of Saturday by the internet portal giant Yahoo. On the grand scale of things, the outage was minor, but the downtime was felt by anyone who attempted to access Yahoo Mail, Flickr, Yahoo News, and the IM network. The outage was reported by almost everyone; oddly, Yahoo said nothing until prompted by CNet news.

In a statement to CNet, Yahoo said that for a brief period, “some Yahoo users may have experienced delays in accessing Yahoo services in the U.S. due to network carrier-related issues. The problem has since been identified and corrected by our network vendor. We know that this may have caused some inconvenience, and we apologize to our users who might have been affected.” Comments on the CNet website were numerous when the outage was first reported, but when the reasons were given for the downtime one comment asked a question that makes some Yahoo users wonder.

“There's a corporate culture of denial at Yahoo. They never inform their customers of service interruptions/outages. Customers are left to fend for themselves, reduced to posting on Yahoo Answers and Craig's List. Tell me Yahoo, what is so hard about giving customers access to a network status page? You seem to have no problem giving us a home page with the latest news about Paris Hilton or Britney Spears,” asked a user posting as “Stating” on the CNet comments. A question many might have been wondering, and an idea that was greatly supported. Yahoo has no such function on their website.

The Internet Storm Center (ISC) reported the outage almost the second it started, and on Sunday gave a better explanation as to what caused it. “On Friday we reported that there were connectivity issues with Yahoo.  Initially, we thought that it was a problem either at Yahoo or perhaps inside Verizon's networks based on emails we received.  Later we determined that it was not Verizon or Yahoo, but more likely an issue at Level3,” the ISC reports.

After a series of tests, and posts on several mailing lists, the trouble was narrowed down to an BGP (Border Gateway Protocol) issue on Level3’s backbone. “It was a BGP routing issue that affected reachability of many sites that had routes advertised through Level3.  Unfortunately this is one of the Internet's "dirty little secrets" - BGP updates are the lifeblood of the Internet but yet there are many ways these route advertisements can fail.”

Faults like the BGP issue have been addressed as there have been several suggestions to improve routing. Examples can be found in the soBGP and S-BGP projects. Yahoo is currently online with no routing issues reported. More information is online.

ISC: http://isc.sans.org/diary.html?storyid=3112
soBGP: http://tinyurl.com/yyskom
S-BGP: http://www.ir.bbn.com/sbgp/