What would be the point of having multiple rankings if they always
showed the same results?
But these are very different results:
none of the CBL top 10 show up in the PSBL top 10!
How can both the PSBL and CBL rankings be correct?
First, “correct” for such rankings does not mean completely accurate
and it does not mean completely precise:
no blocklist will ever detect every spam message emitted by every IP address.
Suppose even mighty NSA (No Such Agency) were to copy every bit that
passed over every major ISP in the U.S.
Even that would miss some bits emitted by for example an ISP in Vietnam
that spammed an ISP in India.
And what heuristics would mighty NSA use to detect all the spam from all
Would those heuristics happen to include the same one CBL is using
to detect the Kelihos rampage?
Would they include some further heuristic of which CBL has not yet thought
that would detect some other rampage?
Quite possibly yes and yes.
Any rankings of anything on the Internet are always approximate
records of hints and whispers of a constantly-shifting reality
that can never be completely pinned down.
Second, correct for rankings
means comparable among the ASNs ranked, so that they can be ranked.
In that sense, yes, both the PSBL and CBL rankings are correct:
they merely show different aspects of the spam symptom of defective
infosec for the ranked ASNs.
Third, any systematically ranked symptom of poor infosec is important.
Does any organization want any of its hosts to be spewing hundreds
of thousands of spam messages a day, as in those ASNs in the CBL top 10?
Does any organization want any of its hosts to be spewing enough
spam in aggregate to turn up in the PSBL top 10?
Besides, actually the CBL data does corroborate the PSBL data,
when viewed in another set of rankings.
Continue reading →
The source of the problem was embarassingly simple and easily fixed:
not enough inodes.
The CBL and PSBL data were affected differently because they arrive
We pick up from CBL daily a text summary table with a line per IP address.
We get from PSBL an NNTP feed of spam messages, each in its own file,
that we boil down to a summary.
So for CBL, we either got the whole file (most days of the month), or we didn’t
store it at all (8 and 11 September).
For PSBL, for each incoming message, we either stored it or we didn’t.
Which is why there are some days with PSBL data between 4 and 15 Sep,
but the volume is lower than usual.
The notice below the chart is dire because we prefer to be conservative
about these things.
In many developing countries, the absence of surface-based air pollution
sensors makes it difficult, and in some cases impossible, to get even a
rough estimate of the abundance of a subcategory of airborne particles
that epidemiologists suspect contributes to millions of premature deaths
each year. The problematic particles, called fine particulate matter
(PM2.5), are 2.5 micrometers or less in diameter, about a tenth the
fraction of human hair. These small particles can get past the body’s
normal defenses and penetrate deep into the lungs.
Even satellite measurements are difficult (clouds, snow, sand, elevation, etc.).
But not impossible:
Two years ago, Chinese officials asked the US Embassy to stop tweeting
about pollution in Beijing on the grounds that the information was
“confusing” and could have “social consequences”,
a confidential US State Department cable made public by WikiLeaks.
Hm, so measurement can affect reputation and have social consequences….
An area where China does not lead the world:
Country rankings by SpamRankings.net.
China is only #13, but Brazil, Russia, and India (the other three BRICs)
are in the top five countries by total spam messages for October 2011.
U.S. is #10.
Vietnam came from behind a few months ago to place second for October.
Brazil had slumped as low as #6 in July, but has pulled back up into the leading pack.
Paul Graham points out that big company checks on purchasing
usually have costs, such as purchasing checks increase the costs of
purchased items because the vendors have to factor in their costs
of passing the checks.
Such things happen constantly to the biggest organizations of all,
governments. But checks instituted by governments can cause much worse
problems than merely overpaying. Checks instituted by governments can
cripple a country’s whole economy. Up till about 1400, China was richer
and more technologically advanced than Europe. One reason Europe pulled
ahead was that the Chinese government restricted long trading voyages. So
it was left to the Europeans to explore and eventually to dominate the
rest of the world, including China.
I would say western governments (especially the U.S.) subsidizing
petroleum production and not renewable energy is one of the biggest
source of current world economic, political, and military problems.
Of course, lack of checks can also have adverse effects as we’ve
just seen with the fancy derivatives the shadow banking system
sold in a pyramid scheme throughout the world.
It’s like there should be a balance on checks.
Which I suppose is Graham’s point: without taking into account
the costs of checks (and I would argue also the risks of not
having checks), how can you strike such a balance?