Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

Mark B wrote:
> Does anyone have a method to determine the language (e.g. en-US, fr-FR,
> zh-TW etc) of a body of text?
>
> (In my case the body of text is an email received).
>
> Naturally the text, if not English, may have a little bit of English.
> I'm just trying to get the main language used.

I thought of following: take dictionary of the common words of the
language you interested in. Then for each word of the text, calculate
times the word occures. But, this needs several versions of the word;
for example "word" and "words". On some languages this is not possible,
since there can be so many variations of a single word.

But, check article "Language Trees and Zipping" by Dario Benedetto,
Emanuele Caglioti and Vittorio Loreto, downloadable from
http://xxx.uni-augsburg.de/format/cond-mat/0108530 . It seems there is
also perl implementation of the algorithm :
code.activestate.com/recipes/355807 . If I understood it right, zip
archiver is based on the idea that it tries to learn the sequence and
the more it learns (i.e. the bigger the text), the better it compresses.
When you teach zip with English text and then give it two texts A and B;
if A is english it is compressed better than B which is italian.

--
Arto Viitanen

http://code.google.com/apis/ajaxlang...tation/#Detect

"Mark B" <(E-Mail Removed)> wrote in message
news:u$(E-Mail Removed)...
> Does anyone have a method to determine the language (e.g. en-US, fr-FR,
> zh-TW etc) of a body of text?
>
> (In my case the body of text is an email received).
>
> Naturally the text, if not English, may have a little bit of English. I'm
> just trying to get the main language used.

There's some code here that writes the internet headers of an email to
a text file, which you could then parse for the language. For example,
some email headers include a "Content-Type" line which indicates the
character set used.

i.e.: Content-Type: text/plain; charset=US-ASCII

http://blogs.technet.com/kclemson/ar...n-outlook.aspx

HTH,
JP

On Oct 23, 3:59*am, "Mark B" <none...@none.com> wrote:
> Does anyone have a method to determine the language (e.g. en-US, fr-FR,
> zh-TW etc) of a body of text?
>
> (In my case the body of text is an email received).
>
> Naturally the text, if not English, may have a little bit of English. I'm
> just trying to get the main language used.

Look at the MailItem.InternetCodepage property (corresponds to the
PR_INTERNET_CPID MAPI propety).

--
Dmitry Streblechenko (MVP)
http://www.dimastr.com/
OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <(E-Mail Removed)> wrote in message
news:u$(E-Mail Removed)...
> Does anyone have a method to determine the language (e.g. en-US, fr-FR,
> zh-TW etc) of a body of text?
>
> (In my case the body of text is an email received).
>
> Naturally the text, if not English, may have a little bit of English. I'm
> just trying to get the main language used.

Do all incoming emails have this? Even if they are not originating from an
Outlook client?

(It's for an Outlook 2007 Add-in C# BTW).


"Dmitry Streblechenko" <(E-Mail Removed)> wrote in message
news:%23$(E-Mail Removed)...
> Look at the MailItem.InternetCodepage property (corresponds to the
> PR_INTERNET_CPID MAPI propety).
>
> --
> Dmitry Streblechenko (MVP)
> http://www.dimastr.com/
> OutlookSpy - Outlook, CDO
> and MAPI Developer Tool
> -
> "Mark B" <(E-Mail Removed)> wrote in message
> news:u$(E-Mail Removed)...
>> Does anyone have a method to determine the language (e.g. en-US, fr-FR,
>> zh-TW etc) of a body of text?
>>
>> (In my case the body of text is an email received).
>>
>> Naturally the text, if not English, may have a little bit of English. I'm
>> just trying to get the main language used.
>
>

Most of them. But that really tells you more about the defaut code page of
the sender.

--
Dmitry Streblechenko (MVP)
http://www.dimastr.com/
OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <(E-Mail Removed)> wrote in message
news:%(E-Mail Removed)...
> Do all incoming emails have this? Even if they are not originating from an
> Outlook client?
>
> (It's for an Outlook 2007 Add-in C# BTW).
>
>
> "Dmitry Streblechenko" <(E-Mail Removed)> wrote in message
> news:%23$(E-Mail Removed)...
>> Look at the MailItem.InternetCodepage property (corresponds to the
>> PR_INTERNET_CPID MAPI propety).
>>
>> --
>> Dmitry Streblechenko (MVP)
>> http://www.dimastr.com/
>> OutlookSpy - Outlook, CDO
>> and MAPI Developer Tool
>> -
>> "Mark B" <(E-Mail Removed)> wrote in message
>> news:u$(E-Mail Removed)...
>>> Does anyone have a method to determine the language (e.g. en-US, fr-FR,
>>> zh-TW etc) of a body of text?
>>>
>>> (In my case the body of text is an email received).
>>>
>>> Naturally the text, if not English, may have a little bit of English.
>>> I'm just trying to get the main language used.
>>
>>
>

Mark B wrote:
> Does anyone have a method to determine the language (e.g. en-US, fr-FR,
> zh-TW etc) of a body of text?
>
> (In my case the body of text is an email received).
>
> Naturally the text, if not English, may have a little bit of English.
> I'm just trying to get the main language used.

If you are willing to write some code, then you can detect
the language (but probably not the regional dialect).

* dictionary with common words
* special letters (forward and backward accents, umlauts etc.)
* distribution of letters
* distribution of pairs of letters

Arne