nilsimsa

kanji le nilsimsa sucta be lo notci gi'e karbi le sucta gi'e tolcri lo smigunma mu'i lonu fespu'i lo se maldustisna

computes nilsimsa codes of messages and compares the codes and finds clusters of similar messages so as to trash spam

What's a nilsimsa code?

A nilsimsa code is something like a hash, but unlike hashes, a small change in the message results in a small change in the nilsimsa code. Such a function is called a locality-sensitive hash. For example, here's a spam I got:

Dear Sir / Madam,
         It is a great honour  to have the chance to introduce our company(Guangdong
Kestar Electronic Co.Ltd.) and  our main products.
     Guangdong Kestar Electronic Co.Ltd. is located in Guangdong province mainland
China.It is a professional manufcturer on  Varistors ,equipped with a complete
set of advanced machinery,possessed high abilities of development & exploitation.We
have 10 years experience in managing large size production.
     Our main seven series varistors are WYG current varistor, MYL lihtningproof
varistor,MYE high load varistor,MYA minitype  varistor,MYP squareness varistor,MYR
highly reliable&varistor,low-pressure power supply lightning arrester..  The
Varistors conform to ISO9002  .The products have passed  several international
authentication.
      if you are interested in our products, please contact us without hesitate.

yours truely  Lin Cang.

Guangdong Kestar Electronic Co.Ltd

Here's the same spam with spacing and spelling cleaned up:

Dear Sir / Madam,
     It is a great honour to have the chance to introduce our company (Guangdong
Kestar Electronic Co. Ltd.) and our main products.
     Guangdong Kestar Electronic Co.Ltd. is located in Guangdong province, mainland
China. It is a professional manufacturer of varistors, equipped with a complete
set of advanced machinery, possessing high abilities of development & exploitation. We
have 10 years experience in managing large size production.
     Our main seven series varistors are WYG current varistor, MYL lightningproof
varistor, MYE high load varistor, MYA minitype varistor, MYP squareness varistor, MYR
highly reliable varistor, low-pressure power supply lightning arrester.  The
Varistors conform to ISO9002. The products have passed  several international
authentications.
      if you are interested in our products, please contact us without hesitation.

yours truly, Lin Cang.

Guangdong Kestar Electronic Co. Ltd

The nilsimsa codes of the two messages are:

773e2df0a02a319ec34a0b71d54029111da90838cbc20ecd3d2d4e18c25a3025 spam1
47182cf0802a11dec24a3b75d5042d310ca90838c9d20ecc3d610e98560a3645 spam2

The nilsimsa of these two codes is 92 on a scale of -128 to +128. That means that 36 bits are different and 220 bits the same. Any nilsimsa over 24 (which is 3 sigma) indicates that the two messages are probably not independently generated.

download source and signature

To try out the unfinished long code option: source and signature

mailing list/mrisi'ugri