Category Archives: Data Mining

eCrime Summit in Prague 25-27 April 2012

These ecrime meetings are always interesting and useful. -jsq

Press release of 29 March:

Containing the Global Cybercrime Threat is Focus of Counter eCrime Operations Summit (CeCOS VI) in Prague, April 25-27

CeCOS VI, in Prague, Czech Republic, to focus on harmonizing operational issues, cybercrime data exchange, and industrial policies to strengthen and unify the global counter-ecrime effort.

CAMBRIDGE, Mass.—(BUSINESS WIRE)—The 6th annual Counter eCrime Operations Summit (CeCOS VI) will convene in Prague, Czech Republic, April 25-27, 2012, as the APWG gathers global leaders from the financial services, technology, government, law enforcement, communications sectors, and research centers to define common goals and harmonize resources to strengthen the global counter-cybercrime effort.

CeCOS VI Prague will review the development of response systems and resources available to counter-cybercrime managers and forensic professionals from around the world.

Specific goals of this high-level, multi-national conference are to identify common forensic needs, in terms of the data, tools, and communications protocols required to harmonize cybercrime response across borders and between private sector financial and industrial sector responders and public sector policy professionals and law enforcement.

Key presentations will include:

Continue reading

Did the February 2012 spam surge come from one botnet? saw
AS 21788NOC
AS 33055BCC-65-182-96-0-PHX
AS 15149EZZI-101-BGP
AS 13768PEER1
a huge surge in spam from some U.S. ASNs, mostly from ones that hadn’t even been in the top 10 before, with possible correlations in one ASN each from Peru and Canada. Did all this spam come from the same botnet?

Maybe not all, but most. Eight out of the U.S. top 10 for February show very close correlation with one botnet, Ogee. They are listed in the table on the right and shown in the chart below:

Left Axis: ASN volume (spam messages); Right Axis: Botnet volume (dotted curves)

The chart also shows some ASNs reacted quickly and stopped the spamming, while others got worse. It’s a busy chart, so let’s look at simpler charts for one example each of resilient and susceptible ASNs.

AS 21788 NOC was one of the first and worst affected by this spam surge: Continue reading

Coal company reputation

Good news from the SEC for a change! They’re requiring coal plant operators to report health and safety violations, including fatalities, within a few days of occurence.

FuelFix posted from AP on 23 December 2011, SEC requiring coal firms to report safety problems

Earlier this week, the SEC announced new rules that require mining companies to start reporting any fatalities and all major health and safety violations, mine by mine, in their quarterly and annual financial reports. The filings are mandated in the wide-ranging Dodd-Frank Wall Street Reform and Consumer Protection Act, which Congress passed to try to increase corporate accountability.

The rules take effect 30 days after publication in the Federal Register. They require companies to report within four days any “significant and substantial” violations, citations, flagrant violations and imminent-danger orders issued by the federal Mine Safety and Health Administration.

Coal operators must also include the dollar value of proposed fines, whether the company has been or may be designated a pattern violator by MSHA, and any pending cases with the Federal Mine Safety and Health Review Commission.

What problem does this reporting solve? As the article points out: Continue reading

Daniel Karrenberg and RIPE Atlas

Daniel Karrenberg shows an animation related to RIPE Atlas, RIPE’s new active measurement project using USB-powered dongles scattered around the Internet.

Video by jsq at RIPE 61 in Rome, Italy, 15 Nov 2010. His slides, the RIPE Atlas home page, and the conference will put up video of all the talks within about a day.


PS: My talk is 11AM Rome time tomorrow, Tuesday 16 Nov.

Outbound Spam Ranking Experiments

Should Uganda Telecom be counted as a Belgian ISP for outbound spam rankings?

Which matters most: history, topology, business headquarters location, or some other criterion?

These are some questions that come up in designing experiments in rolling out a reputation system for outbound spam. More on this in the RIPE Labs article (8 Nov 2010), Internet Reputation Experiments for Better Security.

Such experiments can draw on fifty years of social science research and literature, first crystalized as Social Comparison Theory by Leon Festinger in 1954, that indicate that making personal reputation transparent changes personal behavior. More recent research indicates that the same applies to organizations. Using anti-spam blocklist data, it is possible to make E-Mail Service Provider (ESP) behavior (banks, stores, universities, etc., not just ISPs) in preventing or stopping outbound spam transparent, and this paper is about experiments to see how the resulting reputation actually changes ESP behavior.


Organizing the Cloud Against Spam

In RIPE Labs, here’s a paper on Internet Cloud Layers for Economic Incentives for Internet Security by the IIAR Project (I’m the lead author). Anti-spam blocklists and law enforcement are some Internet organizational layers attempting to deal with the plague of spam, so far reaching a standoff where most users don’t see most spam, yet service providers spend large amounts of computing and people resources blocking it.
The root of the ecrime problem is not technology: it is money.
Continue reading

NANOG: Botnets, DDoS and Ground-Truth

Here at NANOG 50 Craig Labovitz just gave an interesting talk about botnet data derived from Arbor Network customers enabling anonymous data (37 ISPs over last 12 months), of 5,000 events classified by operators.

60% of DDoS attacks are by flooding. Yet most attacks involve few IP addresses; indicates address spoofing.

Slight problem: only 1/4 of customers have enabled anonymous data. “Real goal of this talk is to encourage participation.”

Well-received talk.


Medical Metrics Considered Overrated

One of the presenters at Metricon 5.0 was comparing IT security to other fields in various aspects of metrics and monitoring. I mentioned I thought she was giving far too much green for good to the field of medicine. This provoked repeated back and forth later.

My point was that 150 years after the invention of epidemiology and 100 years after the discovery of bacterial transmission of disease, in medicine application of known preventive measures is so low that Atul Gawande of Harvard has gotten large (on the order of 30%) reductions in deaths from complications of surgery in many hospitals simply by getting them to use checklists for things like washing hands before surgery.

I have an elderly relative in a nursing home who can’t take pills whole due to some damage to nerves in her neck. Again and again visitors sent by the family discover nursing home staff trying to give her pills whole without grinding them up. Why? They don’t read instructions about her, and previous shifts don’t remind later shifts. This kind of communication problem is epidemic not only in nursing homes but in hospitals. I found my father in a diabetic coma because nurses hadn’t paid any attention to him being a diabetic and needing to eat frequently. Fortunately, a bit of honey brought him out of it. Even nurses readily acknowledge this problem, but it persists. I can rattle off many other examples.

To which someone responded, yes, but medicine has epidemiology, and Edward Tufte demonstrated in one of his books that that goes well beyond checklists in to actual analysis, as in a physician’s discovery of a well in London being he source of cholera. I responded, yes, John Snow, in 1854: that was the first thing I said when I stood up to address this. But who now applies what he learned? One-shot longitudinal studies are not the same as ongoing monitoring with comparable metrics to show how well one group is doing compared to both the known science and to other groups.

Many people still didn’t get it, and kept referring to checklists as rudimentary.

So I tried again. If John Snow were alive today, he wouldn’t be prescribing statins for life to people with high blood pressure. He would be compiling data on who has high blood pressure and what they have been doing and eating before they got it. He would follow this evidence back to discover that one of the main contributors to high blood pressure, heart disease, and diabetes in the U.S. is high fructose corn syrup (HFCS). Then he would mount a political campaign to ban high fructose corn syrup, which would be the modern equivalent of his removal of the handle from the pump of the well that stopped the cholera.

To which someone replied, but there are political forces who would oppose that. And I said, yes, of course. Permit me to elaborate.

There were political forces in John Snow’s time, too, and he dealt with them:

Dr Snow took a sample of water from the pump, and, on examining it under a microscope, found that it contained “white, flocculent particles.” By 7 September, he was convinced that these were the source of infection, and he took his findings to the Board of Guardians of St James’s Parish, in whose parish the pump fell.

Though they were reluctant to believe him, they agreed to remove the pump handle as an experiment. When they did so, the spread of cholera dramatically stopped. [actually the outbreak had already lessened for several days]

Snow also investigated several outliers, all of which turned out to involve people actually travelling to the Soho well to get water.
Still no one believed Snow. A report by the Board of Health a few months later dismissed his “suggestions” that “the real cause of whatever was peculiar in the case lay in the general use of one particular well, situate [sic] at Broad Street in the middle of the district, and having (it was imagined) its waters contaminated by the rice-water evacuations of cholera patients. After careful inquiry,” the report concluded, “we see no reason to adopt this belief.”

So what had caused the cholera outbreak? The Reverend Henry Whitehead, vicar of St Luke’s church, Berwick Street, believed that it had been caused by divine intervention, and he undertook his own report on the epidemic in order to prove his point. However, his findings merely confirmed what Snow had claimed, a fact that he was honest enough to own up to. Furthermore, Whitehead helped Snow to isolate a single probable cause of the whole infection: just before the Soho epidemic had occurred, a child living at number 40 Broad Street had been taken ill with cholera symptoms, and its nappies had been steeped in water which was subsequently tipped into a leaking cesspool situated only three feet from the Broad Street well.

Whitehead’s findings were published in The Builder a year later, along with a report on living conditions in Soho, undertaken by the magazine itself. They found that no improvements at all had been made during the intervening year. “Even in Broad-street it would appear that little has since been done… In St Anne’s-Place, and St Anne’s-Court, the open cesspools are still to be seen; in the court, so far as we could learn, no change has been made; so that here, in spite of the late numerous deaths, we have all the materials for a fresh epidemic… In some [houses] the water-butts were in deep cellars, close to the undrained cesspool… The overcrowding appears to increase…” The Builder went on to recommend “the immediate abandonment and clearing away of all cesspools — not the disguise of them, but their complete removal.”

Nothing much was done about it. Soho was to remain a dangerous place for some time to come.

John Snow didn’t shy away from politics. He was successful in getting the local politicians to agree to his first experiment, which was successful in helping end that outbreak of cholera. He even drew his biggest opponent into doing research, which ended up confirming Snow’s epidemiological diagnosis and extending it further to find the original probable source of infection of the well. But even that didn’t suffice for motivating enough political will to fix the problem.

From which I draw two conclusions:

  1. Even John Snow is over-rated. Sure, he found the problem, but he didn’t get it fixed longterm.

  2. Why not? Because that would require ongoing monitoring of likely sources of infection (which sort of happened) compared to actual incidents of disease (which does not appear to have happened), together with eliminating the known likely sources.
Eliminating likely known sources is what Dr. Gawande’s checklist is about, 150 years later, which was my original point. And the ongoing monitoring and comparisons appear not to be happening, even yet.

As someone at Metricon said, who will watch the watchers? I responded, yes, that’s it!

One-shot longitudinal studies can create great information. That’s what John Snow did. That’s what much of scientific experiment is about. But even when you repeat the experiment to confirm it, that’s not the same as ongoing monitoring. And it’s not the same as checklists to ensure application of what was learned in the experiment.

What is really needed is longitudinal experiments combined checklists, plus ongoing monitoring, plus new analysis derived from the monitoring data. That’s at least four levels. All of them are needed. Modern medicine often only manages the first. And in the case of high fructose corn syrup (HFCS), until recently even the first was lacking, and most of the experiments that have happened until very recently have not come from the country with the biggest HFCS health problem, namely the U.S. A third of the entire U.S. population is obese, and another third is overweight, with concomittant epidemics of heart disease, diabetes, and high blood pressure. And the medical profession prescribes statins for life instead of getting to the root of the problem and fixing it.

Yes, I think the field of medicine gets rated too much green for good.

And if IT security wants to improve its own act, it also needs all four levels, not just the first or the second.


FireEye’s Ozdok Botnet Takedown Observed

FireEye coordinated a takedown of botnet Ozdok or MegaD, on 5-6 Nov 2009, with cooperation by many ISPs and DNS registrars.

Good show! What effects did it have on spam? Not just spam from this botnet; spam in general.

Botnets and spam volume

This graph was presented at NANOG 48, Austin, TX, 24 Feb 2010, in FireEye’s Ozdok Botnet Takedown In Spam Blocklists and Volume Observed by IIAR Project, CREC, UT Austin. John S. Quarterman, Quarterman Creations, Prof. Andrew Whinston, PI CREC, UT Austin. That was a snapshot of an ongoing project, Incentives, Insurance and Audited Reputation: An Economic Approach to Controlling Spam (IIAR).

That presentation was enough to demonstrate the main point: takedowns are good, but we need a lot more of them and a lot more coordinated if we are to make a real dent in spam.

The IIAR project will keep drilling down in the data and building up models. One goal is to build a reputation system to show how effective takedowns and other anti-spam measures are, on which ASNs.

Thanks especially to CBL and to Team Cymru for very useful data, and to FireEye for a successful takedown.

We’re all ears for further takedowns to examine.


NSL: Internet Archive Exposes Lack of Security in National Security Letters

Brewster_Kahle_20021120.jpg The Internet Archive has for a decade been a cornerstone of the Internet, and the FBI was foolish to try to break it:
The FBI has withdrawn an illegal National Security Letter seeking information from an online library and has lifted a gag order that until Wednesday prevented any discussion of the information request.

Lawyers from the American Civil Liberties Union and Electronic Frontier Foundation helped the Internet Archive push back against what they say was an overly broad and unlawful request for information on one of its users. The FBI issued its National Security Letter in November, but ACLU, EFF and Archive officials were precluded from discussing it with anyone because of a gag order they say was unconstitutional.

After nearly five months of haggling, the FBI eventually withdrew its NSL, which requested personal information about at least one user of the Internet Archive. Founded in 1996, the archive is recognized as a library by the state of California, and its collections include billions of Web records, documents, music and movies.

Watchdogs prompt FBI to withdraw ‘unconstitutional’ National Security Letter, Nick Juliano, therawstory, Published: Wednesday May 7, 2008

The article goes on to say that the FBI has issued 200,000 National Security Letters, that almost none of those NSL have been challenged, yet every single time someone has challenged an NSL in court, the FBI has withdrawn it.

How do these NSL represent “Security”?

In any case, National Security Letters were authorized by the mis-named Patriot Act. Brewster Kahle has shown us how a real patriot acts: Continue reading