[Perl-AI] RE: :Categorize slides from YAPC online

Dear Ken et al.,
I have been saving the spam email sent to me so that one day
I could use a module like this to detect it.
I could make available to anyone a *.csv (comma separated values) file
that contains about 440 spam messages. It has a total of
about 27000 lines and is 1.1 MB (400 KB compressed with zip).

This file would need to be complemented by a file of the same size,
or ideally even larger, than contain non-spam. I could not
distribute a non-spam file as most messages are company confidential.
Anyway each person would want to have their own non-spam file,
as I get messages about databases, XML, etc. and other people
would get other messages on other topics.

I want to pursue how to incorporate your code into an actual
solution. I have some theoretical questions, and some practical
ones.

* If there are only two categories (spam vs. non-spam)
is there some special algorithm that is appropriate?

* Actually a high fraction of the spam messages are in Spanish.
I copuld manually separate these out very quickly.
Would it help improve performance (i.e. better F1 score)
to have these in a separate category?

* An easy way to detect these Spanish messages is to look for
the Perl pattern / esta/i
But I am concerned that this strength of this predictor would
be "diluted" due to the many word forms.

* Probably the best way to detect spam is to look for a number
in the subject line, e.g., FREE Life Insurance Quotes 10077
However I suspect your code would treat all these numbers as
different words and so not notice the pattern. It seems desirable
to first transform the input in certain ways. I might want to
transform strings such as 19\d\d and 20\d\d to e.g. the dummy word
_date and then transform all other numbers with 4 or more digits
to e.g. _number. Then a verey string predictor of spam is
_number in the subject.

* Similarly I want to map punctuation to pseudo-words, e.g. any string
of more than one consecutive ! character would become _bang.

Some additional background: My company uses Microsoft
Outlook, which has a "rules wizard" with some limited ability
to route mail to different folders, based on who it is from, words
in the subject etc. I am currently using Outlook 98, but plan
to go to Outlook 2000 soon.

* I would like to get a list of the words most likely to be associated
with a category. Can I get this from your code? How?
E.g. for the spam category I expect to find Britney, free, etc.
This is very important because Outlook rules can move mail based
on words. I would be willing to move any email containing
"Britney" to a spam_probably folder.

* Does Outlook 2000 add much in the way of what the rules
pay attention to?

* Ideally I could set up my mail system to make a call to
some external program, and it would return a category.
Is this possible to do in the Outlook client,
or in the Exchange server?

* Can Outlook filter messages based on the domain of the sender.
My first filter would accept anything sent from within the company,
or from certain known outside addresses. Then I would
categorize all the remaining messages into spam vs non-spam.

Hopefully helpfully yours,
Steve