{"id":519,"date":"2012-02-17T11:04:21","date_gmt":"2012-02-17T09:04:21","guid":{"rendered":"https:\/\/polyetilen.lt\/?p=519"},"modified":"2023-05-02T10:09:51","modified_gmt":"2023-05-02T07:09:51","slug":"text-language-detection-with-php","status":"publish","type":"post","link":"https:\/\/polyetilen.lt\/en\/text-language-detection-with-php","title":{"rendered":"Text language detection with php"},"content":{"rendered":"

Previously was Google Language API for language detection but it is now paid. I found an alternative way to detect the language of text using Text_LanguageDetect pear package<\/a> with 52 supported languages. Here is lithuanian text language detection example with list of supported languages:<\/p>\n

\r\n<?\r\nheader('Content-Type: text\/plain; charset=utf-8');\r\n\r\nrequire_once 'Text\/LanguageDetect.php';\r\n$l = new Text_LanguageDetect();\r\n\r\n\/\/example text for language detection\r\n$text = 'O mergina, vienplauk\u0117, palaidomis kasomis, atlapu kaklu, smulkiu, bet tvirtu \u017eingsniu \u0117jo toliau, artyn prie e\u017eero.';\r\n\r\n\/\/Detects the closeness of a sample of text to the known languages\r\n$result = $l->detect($text, 4);\r\nprint_r($result);\r\n\r\n\/\/Returns the distribution of unicode blocks in a given utf8 string\r\n$blocks = $l->detectUnicodeBlocks($text, true);\r\nprint_r($blocks);\r\n\r\n\/\/language name\r\n$l->setNameMode(0);\r\necho $l->detectSimple($text)."\\n";\r\n\r\n\/\/ISO 639-1 two-letter language code\r\n$l->setNameMode(2);\r\necho $l->detectSimple($text)."\\n";\r\n\r\n\/\/ISO 639-2 three-letter language code\r\n$l->setNameMode(3);\r\necho $l->detectSimple($text)."\\n";\r\n\r\n\/\/Supported languages list\r\n$l->setNameMode(0);\r\necho "Supported languages:\\n";\r\n$langs = $l->getLanguages();\r\nsort($langs);\r\nprint_r($langs);\r\n\r\n\/\/Total amount of supported languages\r\necho count($langs);\r\n<\/pre>\n
Output:<\/p>\n
\r\nArray\r\n(\r\n    [lithuanian] => 0.24584192439863\r\n    [latvian] => 0.19567010309278\r\n    [estonian] => 0.11316151202749\r\n    [dutch] => 0.11240549828179\r\n)\r\nArray\r\n(\r\n    [Basic Latin] => 89\r\n    [Latin Extended-A] => 4\r\n)\r\nlithuanian\r\nlt\r\nlit\r\nSupported languages:\r\nArray\r\n(\r\n    [0] => albanian\r\n    [1] => arabic\r\n    [2] => azeri\r\n    [3] => bengali\r\n    [4] => bulgarian\r\n    [5] => cebuano\r\n    [6] => croatian\r\n    [7] => czech\r\n    [8] => danish\r\n    [9] => dutch\r\n    [10] => english\r\n    [11] => estonian\r\n    [12] => farsi\r\n    [13] => finnish\r\n    [14] => french\r\n    [15] => german\r\n    [16] => hausa\r\n    [17] => hawaiian\r\n    [18] => hindi\r\n    [19] => hungarian\r\n    [20] => icelandic\r\n    [21] => indonesian\r\n    [22] => italian\r\n    [23] => kazakh\r\n    [24] => kyrgyz\r\n    [25] => latin\r\n    [26] => latvian\r\n    [27] => lithuanian\r\n    [28] => macedonian\r\n    [29] => mongolian\r\n    [30] => nepali\r\n    [31] => norwegian\r\n    [32] => pashto\r\n    [33] => pidgin\r\n    [34] => polish\r\n    [35] => portuguese\r\n    [36] => romanian\r\n    [37] => russian\r\n    [38] => serbian\r\n    [39] => slovak\r\n    [40] => slovene\r\n    [41] => somali\r\n    [42] => spanish\r\n    [43] => swahili\r\n    [44] => swedish\r\n    [45] => tagalog\r\n    [46] => turkish\r\n    [47] => ukrainian\r\n    [48] => urdu\r\n    [49] => uzbek\r\n    [50] => vietnamese\r\n    [51] => welsh\r\n)\r\n52\r\n<\/pre>\n
Another example recognizes the page language:<\/p>\n
\r\n<?\r\nheader('Content-Type: text\/plain; charset=utf-8');\r\n\r\nrequire_once 'Text\/LanguageDetect.php';\r\n$l = new Text_LanguageDetect();\r\n\r\nmb_internal_encoding("UTF-8");\r\n\r\n\/\/example content page\r\n$url = "http:\/\/lt.wikipedia.org\/wiki\/Kalba";\r\n$page = file_get_contents($url);\r\n\r\n\/\/parse page charset\r\npreg_match('\/<meta[^>]+charset=[\\'"]*([a-z0-9\\-]+)[\\'"]*\/i', $page, $a);\r\nprint_r($a);\r\n\r\nif(!$a){\r\n\t$charset = "UTF-8";\r\n}else{\r\n\t$charset = strtoupper($a[1]);\r\n}\r\n\r\n\/\/remove whitespace, html tags and javascript from page content\r\n$search = array('#<script[^>]*?>.*?<\/script>#si',\t\/\/ Strip out javascript\r\n\t\t'#<style[^>]*?>.*?<\/style>#siU',\t\t\t\/\/ Strip style tags properly\r\n\t\t'#<[\\\/\\!]*?[^<>]*?>#si',\t\t\t\t\t\/\/ Strip out HTML tags\r\n\t\t'#<![\\s\\S]*?--[ \\t\\n\\r]*>#',\t\t\t\t\/\/ Strip multi-line comments including CDATA\r\n\t\t'#\\s\\s+#'\t\t\t\t\t\t\t\t\t\/\/ Strip whitespace\r\n);\r\n$content = preg_replace($search, '', $page);\r\n\r\n\/\/First 200 simbols of text content should be enough for language detection\r\n$content = mb_substr($content, 0, 200);\r\n\r\n\/\/convert to utf-8 encoding if necessary\r\nif($charset != "UTF-8"){\r\n\t$content = iconv($charset, "UTF-8", $content);\r\n}\r\n\r\n\/\/Output content\r\necho $content."\\n";\r\n\r\n\/\/language name\r\n$l->setNameMode(2);\r\necho $l->detectSimple($content)."\\n";\r\n\r\n\/\/closeness languages\r\n$result = $l->detect($content, 4);\r\nprint_r($result);\r\n\r\n\/\/distribution of unicode blocks\r\n$blocks = $l->detectUnicodeBlocks($content, true);\r\nprint_r($blocks);\r\n<\/pre>\n
Output:<\/p>\n
\r\nArray\r\n(\r\n    [0] => <meta http-equiv="Content-Type" content="text\/html; charset=UTF-8"\r\n    [1] => UTF-8\r\n)\r\nKalba \u2013 VikipedijaKalbaStraipsnis i\u0161 Vikipedijos, laisvosios enciklopedijos.Per\u0161okti \u012f: navigacij\u0105,paie\u0161k\u0105Vikisritis: KalbosKalba \u2013 lingvistini\u0173 \u017eenkl\u0173 sistema.\r\nSvarbiausia kalbos paskirtis \u2013 b\u016bti \u017emo\r\nlt\r\nArray\r\n(\r\n    [lt] => 0.23644295302013\r\n    [lv] => 0.14548098434004\r\n    [et] => 0.14234899328859\r\n    [la] => 0.12302013422819\r\n)\r\nArray\r\n(\r\n    [Basic Latin] => 161\r\n    [General Punctuation] => 3\r\n    [Latin Extended-A] => 11\r\n)\r\n<\/pre>\n
Download text language detection source code<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Previously was Google Language API for language detection but it is now paid. I found an alternative way to detect the language of text using Text_LanguageDetect pear package with 52 supported languages. Here is lithuanian … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_locale":"en_US","_original_post":"http:\/\/polyetilen.lt\/?p=262","footnotes":""},"categories":[8],"tags":[87,34],"class_list":["post-519","post","type-post","status-publish","format-standard","hentry","category-programavimas","tag-kalba","tag-php","en-US"],"_links":{"self":[{"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/posts\/519","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/comments?post=519"}],"version-history":[{"count":2,"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/posts\/519\/revisions"}],"predecessor-version":[{"id":521,"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/posts\/519\/revisions\/521"}],"wp:attachment":[{"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/media?parent=519"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/categories?post=519"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/polyetilen.lt\/wp-json\/wp\/v2\/tags?post=519"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}