Comments on: Language detection, a usability enhancer?

By: Emil Stenström

Emil Stenström — Thu, 15 Mar 2007 22:14:12 +0000

@Peter Vigren: I’ll see what I can do, I’m moving atm so not too much extra time available.

By: Peter Vigren

Peter Vigren — Sun, 11 Mar 2007 16:33:12 +0000

I am also interested in example code for PHP. I find this quite intriguing, maybe because I find programming AND languages so fascinating. :-) But one thing, if you remove or translate characters like é etc, won’t that make the result a bit weird? After all, if a language uses é a lot and others don’t, shouldn’t that be significant? In any case, I really enjoyed this article, first time I ever had heard of this so it tickles my brain like crazy! Thank you. :-)

By: Emil Stenström

Emil Stenström — Sun, 27 Aug 2006 09:56:57 +0000

@Dan Pettersson: so your algorithm gives different numbers for the exact same texts? That’s some bug in your code. If you post a link to it (syntax highlight it) and I’ll have a look.

By: Dan Pettersson

Dan Pettersson — Sat, 26 Aug 2006 22:02:59 +0000

Does anyone have an example of code (e.g. in PHP) that would compare two texts and give you a number…

In my own tests squaring and square rooting messes things up. E.g. if I duplicate a text it won’t show the same result.

I’ve solved it by making my own version without the squares, and I think that works fine…

By: Emil Stenström

Emil Stenström — Mon, 19 Jun 2006 10:36:20 +0000

@Adam Zakreski: The link indeed handles only blog content that technorati indexes. Other sources have managed to approximate the number to 68% which means you are right there. Well done :)

Global Reach meassures the number of people online by language and finds 35%, something that tells much about the future of the web.

By: Adam Zakreski

Adam Zakreski — Fri, 16 Jun 2006 21:54:32 +0000

I find your statement, “In fact, most of the content online is not in English.” A little hard to swallow. Even with the citation, the original source seems to only be refering to a certain type of blog posts. I believe saying, “most blog content online is not in English,” would be more accurate (though still a stretch). I do find it a lot easier to believe that the Japanese are more avid bloggers than English speakers, though.

By: Emil Stenström

Emil Stenström — Tue, 16 May 2006 08:46:04 +0000

@Jesse Skinner: I wouldn’t use a dictionary. Since the texts you will be testing are ordinary texts you should base your text data on that kind of texts. Say if English contains a lot of “in”. Then that bigram should have a high percentage, whether the reason is because it’s in many different words or that those words are common.

By: Jesse Skinner

Jesse Skinner — Mon, 15 May 2006 15:25:33 +0000

Thanks for the algorithm.. I would never have known where to begin to implement something like this.

I think, maybe dictionary files would be the perfect place to build up a bigram statistic list. Those are usually (somewhat) easy to find.

By: Steve Tucker

Steve Tucker — Sun, 14 May 2006 23:28:45 +0000

I try to always put some reference to the language in my markup documents - it makes logical sense. At worst cannot do any harm!

By: Sarven Capadisli

Sarven Capadisli — Sun, 14 May 2006 00:23:50 +0000

Concerning:
# Assisting search engines

I am not sure how effective the lang attribute is.

A while back I had a problem with this in fact.
I’ve placed lang=”en” on my pages, however google had a problem choosing the wrong language for one of my articles.

The only reasoning I could come up with was that the article (at the time) was more popular in the German (de) community.

So I had to write this:
Google Deutschland stole my article

Although this is no longer an issue, I am still spectacle when it comes to search engines (in fact anything) if they are really acknowledging the language of a given document.