Computer Journal

Japanese Spam Analysis (or Artificially Intelligent Teaching by Statistics)

by Javantea
Sept 25, 2008

Japanese AI version 0.1 [sig]

INTRODUCTION

Japanese spam is a good indicator of text in Japanese language. It is also very good tool for understanding common Japanese speech. Most spam is designed to trick the recipient into e-mail or visiting a site. Unlike English spam, most Japanese spam is extremely well-written, targeted at the net savvy and quite well-educated Japanese audience. Also, since spam filters in Japan can pick out words much quicker (since Japan uses Kanji), spammers are using higher quality spam generators.

It is my opinion that Japan is ahead of the curve in spam, so I expect that English spam will be headed in the direction instead of visa- versa. On the other hand in a way Japanese spam lacks variety because the market is small compared to the US (which is small compared to any other advertising medium (websites, snail mail, magazines, tv, newspapers, even billboards)) and so the variety is extremely low. Variety is a trade-off made for quality in spam. The more quality, the less variety, Japanese spam is on the very high end of quality vs variety. As spam grows as a market, it is quite likely that quality and variety will both increase. Quantity, on the other hand is unlikely to grow considering the effectiveness of the medium decreases far faster than quantity grows. As an internet researcher, I am quite interested in the process and the design of different modes of communication especially if they invoke emotions (usually anger and frustration) so readily. Many developers and researchers spend huge amounts of time, energy, and money (including myself) on the topic of spam, and so contributing however small to the common base of knowledge on this subject is an everpresent goal. However today, I will be using spam as a tool to learn about other communciation topics rather than researching it directly. I hope the readers approve even if they may be more interested in something completely different.

I will use histograms of Japanese spam (each unicode character alone) in this essay to teach the reader about the Japanese language. I have a fairly good background in Japanese and can speak basic and almost conversational (at very low speed) Japanese. I have 4 years experience speaking Japanese and I spent one month in Japan. One of my hobbies there was decoding their often cryptic advertisements.

You could almost consider this essay a work of Quantative Linguistics, though I would probably be hard pressed to find a peer to review my paper. If you'd like to peer review this paper or publish it in your Journal (no matter what title it is), feel free to e-mail me or leave a comment.

An excellent paper on decoding real Japanese texts can be found here: http://www.cs.cornell.edu/home/llee/papers/segmentjnle.pdf It was obviously written with English-speaking readers studying Japanese texts in mind. The paper's references should be my bibliography, but it isn't because I have no possibility of reading this wealth of information on Japanese language processing.

Method

The method of creating a database and histogram of spam can be reproduced by anyone who has a directory full of flatfile spam e-mails with some of them being Japanese. If you do not have any Japanese spam, I noticed that creating a website with lots of Japanese characters coincided with Japanese spam increase by a rather large percent. I expect that Japanese spammers are probably using search engines to find websites with Japanese keywords and manually scraping for e-mail addresses. The tools I used to create this database are normal GNU OS utilities as well as custom Python scripts.

# japanese_spam2.sh
# by Javantea
# Oct 26, 2008

# Automatically analyze a directory full of spam

SPAMDIR=~/Mail/.inbox.directory/spam1/
TODAY=$(date +%Y%m%d)

# Put all names of Japanese spam into a text file.
grep -r -i '^ *charset="SHIFT_JIS"' $SPAMDIR > japanese_spam1.txt
grep -r -i '^ *charset="iso-2022-jp"' $SPAMDIR >> japanese_spam1.txt
grep -r -i '^ *charset=utf-8' $SPAMDIR >> japanese_spam1.txt
grep -r -i '^ *charset="utf-8"' $SPAMDIR >> japanese_spam1.txt

# Replace colon (:) with space.
tr ':' ' ' < japanese_spam1.txt > japanese_spam1a.txt

# Create a directory and copy all spams into it.
mkdir spam$TODAY/; awk '{ print $1; }' japanese_spam1a.txt | xargs cp
# Show the user what you've done.
ls spam$TODAY/

# (Optional) Make a histogram of each file seperately for later use.
cd spam$TODAY/
A=$(find . -name '[0-9]*')
cd ..
mkdir spam$TODAY/histogram/
for file in $A; do
	python parse_japanese_email1.py --histogram "spam$TODAY/$file" > "spam$TODAY/histogram/$file"
done

# Make a total histogram of all spam emails.
find spam$TODAY/ -name '[0-9]*' | xargs python parse_japanese_email1.py > spam$TODAY/histogram/tot_hist3.txt

# Turn the histogram into a readable utf8 document.
python parse_japanese_email1.py --pwnhistogram spam$TODAY/histogram/tot_hist3.txt > spam$TODAY/histogram/tot_hist3_utf8.txt

# Sort the histogram by count descending.
sort -k 2 -n -r < spam$TODAY/histogram/tot_hist3_utf8.txt > spam$TODAY/histogram/tot_hist3_utf8_count.txt

Data

Raw Output in HTML format
[uploads/tot_hist3_utf8_count.html]

Raw Output in Text format
[uploads/tot_hist3_utf8_count.txt]

Analysis

Conclusion

I am very pleased with the results of this research. Using a very unuseful set of data, I was able to provide statistical analysis and retrieve interesting and quite useful data from it. The data set is very large and continues to grow daily, so analysis can actually continue as a permanent project.

I am very interested in continuing this research for projects such as my kanji handwriting analysis project and for amateur manga translation projects. Projects such as Kanjilish and the Unicode Fuzzer are also very interesting projects that would definitely benefit from the data in this project.

Bibliography

[5] Japanese For Busy People I, Kana Version, page 66.

If you are interested in analyzing Japanese Spam or other languages using my research methods, feel free to contact me.