Monthly Archives: October 2010

NANOG: load-balancing facebook and interfacing IPV6 using LISP

Donn Lee talked about LISP Deployment at Facebook. No, not that LISP. This one:
In the current Internet routing and addressing architecture, the IP address is used as a single namespace that simultaneously expresses two functions about a device: its identity and how it is attached to the network. One very visible and detrimental result of this single namespace is manifested in the rapid growth of the Internet’s DFZ (default-free zone) as a consequence of multi-homing, traffic engineering (TE), non-aggregatable address allocations, and business events such as mergers and acquisitions.

LISP changes this by separating IP addresses into two new namespaces: Endpoint Idenfitiers (EIDs), which are assigned to end-hosts, and Routing Locators (RLOCs), which are assigned to devices (primarily routers) that make up the global routing system.

So Lee used that to load-balance facebook, which you can try out here:

http://www.lisp4.facebook.com/.

If I understood him, he said his group of network engineers did all this without needing to involve software development, because facebook is still “a small, scrappy company” that permits and encourages such things.

-jsq

NANOG: The Impacts of Adding Undersea Capacity to East Africa

Keven Chege of KENET at NANOG 50 talked about rapid deployment of cable for Internet use throughout east Africa, despite vandalism including copper theft and sabotage by competing ISPs. Many national research and eduction networks (NRENs) at least planned in the area. KENET in Kenya has “Made the big leap from VSAT to fiber” and is helping coordinate the region; slides include proposed regional mesh map. Also talking to google and Akamai.

Akamai guy stood up immediately afterwards and said he hear KENET was talking to google and asked that they should talk to Akamai as well.

-jsq

NANOG: Submarine adopts 40G and 100G

Per Hansen of Ciena at NANOG 50 talked about growing capacity not by adding more data cables under the sea, rather by increasing spectral density. Eventually new cables will be needed, but meanwhile he thinks we can get up from about 2 bits to to 5 or 6 bits per Hertz. It does require more power: same energy per bit, but more bits.

Plus mesh networks for rerouting, even if it means rerouting backwards around the world, he notes. We’ve observed that sort of emergency backwards routing as long ago as January 2008, in the U.A.E. Cable Cut.

-jsq

NANOG: Coping with Relentless Demand Growth

David G. Ross ofThe David Ross Group Inc. at NANOG 50 talked about data cables under the sea, in which he revealed that Internet growth has not only not paused during the recession, it has increased, and it continues to increase in every region in which his company operates, including Asia, Middle East, and Africa. North Atlantic hasn’t had any new submarine capacity in years, in “the most competitive capacity market on Earth”. It will probably run out in a few years, so now there is demand to build new cables there. Each cable costs about $200 million to install.

Slight downside: early remark that he was sure things were the same as they were when he worked for a telephone company.

-jsq

NANOG: Botnets, DDoS and Ground-Truth

Here at NANOG 50 Craig Labovitz just gave an interesting talk about botnet data derived from Arbor Network customers enabling anonymous data (37 ISPs over last 12 months), of 5,000 events classified by operators.

60% of DDoS attacks are by flooding. Yet most attacks involve few IP addresses; indicates address spoofing.

Slight problem: only 1/4 of customers have enabled anonymous data. “Real goal of this talk is to encourage participation.”

Well-received talk.

-jsq

Quis custodiet ipsos medici?

Internet security is in a position similar to that of safety in the medical industry. Many doctors have an opinion like this one, quoted by Kent Bottles:
“Only 33% of my patients with diabetes have glycated hemoglobin levels that are at goal. Only 44% have cholesterol levels at goal. A measly 26% have blood pressure at goal. All my grades are well below my institution’s targets.” And she says, “I don’t even bother checking the results anymore. I just quietly push the reports under my pile of unread journals, phone messages, insurance forms, and prior authorizations.”

Meanwhile, according to the CDC, 99,000 people die in the U.S. per year because of health-care associated infections. That is equivalent of an airliner crash every day. It’s three times the rate of deaths by automobile accidents.

The basic medical error problems observed by Dennis Quaid when his twin babies almost died due to repeated massive medically-administered overdoses and due to software problems such as ably analysed by Nancy Leveson for the infamous 1980s Therac-25 cancer-radiation device are not in any way unique to computing in medicine. The solutions to those problems are analogous to some of the solutions IT security needs: measurements plus six or seven layers of aggregation, analysis, and distribution.

As Gardiner Harris reported in the New York Times, August 20, 2010, another problem is that intravenous and feeding tubes are not distinguished by shape or color: Continue reading →

What we can learn from the Therac-25

What does Nancy Leveson’s classic analysis of the Therac-25 recommend? (“An Investigation of the Therac-25 Accidents,” by Nancy Leveson, University of Washington and Clark S. Turner, University of California, Irvine, IEEE Computer, Vol. 26, No. 7, July 1993, pp. 18-41.)
“Inadequate Investigation or Followup on Accident Reports. Every company building safety-critical systems should have audit trails and analysis procedures that are applied whenever any hint of a problem is found that might lead to an accident.” p. 47

“Government Oversight and Standards. Once the FDA got involved in the Therac-25, their response was impressive, especially considering how little experience they had with similar problems in computer-controlled medical devices. Since the Therac-25 events, the FDA has moved to improve the reporting system and to augment their procedures and guidelines to include software. The input and pressure from the user group was also important in getting the machine fixed and provides an important lesson to users in other industries.” pp. 48-49

The lesson being that you have to have built-in audit, reporting, transparency, and user visibility for reputation.

Which is exactly what Dennis Quaid is asking for.

Remember, most of those 99,000 deaths a year from medical errors aren’t due to control of complicated therapy equipment: Continue reading →

What about the Therac-25?

Someone suggested that Dennis Quaid should be reminded of the Therac-25 “if he thinks computers will reduce risk without a huge investment in quality, quality assurance and operational analysis.” For readers who may not be familiar with it, the Therac-25 was a Canadian radiation-therapy device of the 1980s that was intended to treat cancer. It had at least six major accidents and caused three fatalities, because of poor software design and development.

Why should anyone assume Dennis Quaid doesn’t know that quality assurance and operational analysis are needed for anything designed or controled by software? The man is a jet pilot, and thus must be aware of such efforts by aircraft manufacturers, airlines, and the FAA. As Quaid points out, we don’t have a major airline crash every day, and we do have the equivalent in deaths from medical errors. Many of which could be fixed by Computerized Physician Order Entry (CPOE).

Or ask the Mayo Clinic: Continue reading →

Trust the Doctor, or Trust the Doctor’s Report Card?

What can be done about the huge medical error fatality problem Dennis Quaid identified when his baby twins were almost killed? Electronic medical records (EMR) are a start. Then as Kent Bottles suggests, let’s use those records to improve physician care:
“Dr. Kim A. Adcock, the radiology chief at Kaiser Permanente Colorado, created a system that misses one-third fewer cancers on mammograms and “has achieved what experts say is nearly as high a level of accuracy as mammography can offer.” At the heart of the program was his willingness to keep score and confront his doctors with their results. He had to fire three radiologists who missed too many cancers, and he had to reassign 8 doctors who were not reading enough films to stay sharp.”
We could use more report cards for physicians, including firing ones with failing grades, and maybe even paying the really good ones more, or at least getting them to teach the others.

-jsq