July 24, 2013

Hum bolta ko bolta bolta hain…

Posted in Work & Technology tagged , , , , at 6:11 pm by runa

The findings of the recently conducted survey of the languages of India under the aegis of the People’s Linguistic Survey of India have been the talking point since the past few days. The survey results are yet to be published in its entirety, but parts of it has been released through the mainstream media. The numbers about the scheduled and non-scheduled languages, scripts, speakers are fascinating.

Besides the statistics from the census, this independent survey has identified languages which are spoken in remote corners of the country and by as less as 4 people. From some of the reports[1][2] that have been published, what one can gather is that there are ~780 languages and ~66 scripts presently in use in India. Of which the North Eastern states of India have the largest per capita density of languages and contribute with more than 100 (closer to ~200 if one sums things up) of those. It has also been known that in the last 50 years, ~250 languages have been lost, which I am assuming means that no more speakers of these languages remain.

This and some other things have led onto a few conversations around the elements of language diversity that creep into the everyday Indian life. Things that we assume for normal, yet are so diametrically varied from monolingual cultures. To demonstrate, we picked names of acquaintances/friends/co-workers and put 2 or more of them together to find what was a common language for each group. In quite a few cases we had to settle that English was the only language a group of randomly picked people could converse in. Well if one has been born in (mostly) urban India anytime onwards from the 1970s (or maybe even earlier), this wouldn’t be much of a surprise. The bigger cities have various degrees of cosmopolitan pockets. From a young age people are dragged through these either as part of their own social circle (like school) or their parents’. Depending upon the location and social circumstances English is often the first choice.

When at age 10 I had to change schools for the very first time, I came home open-mouthed and narrated to my mother that in the new school the children speak to each other in Bengali! Until that time, Bengali was the exotic language that was only spoken at home and was heard very infrequently on the telly on sunday afternoons. The conservative convent school where I went was a melting pot of cultures with students from local North East Indian tribes, Nepalis (both from India and Nepal), Tibetans, Chinese, Bhutanese and Indians from all possible regions where Government and Armed Forces personnel are recruited from. Even the kid next door who went to the same school spoke in English with me at school and in Bengali at the playground in the evening.

The alternative would be the pidgin that people have to practice out of necessity. Like me and the vegetable vendor in the sunday market. I don’t know her language fluent enough to speak (especially due to the variation in dialect), she probably hasn’t even heard of mine, and we both speak laughable hindi. What we use is part Hindi and part Marathi and a lot of hand movements to transact business. I do not know what I would do if I was living further south were Hindi is spoken much less. But it would be fun to try out how that works.

An insanely popular comic strip has been running since the past year – Guddu ang Gang, by Garbage Bin Studios. The stories are a throwback to our growing up years from the late 80s and 90s and touched so many chords on a personal level. The conversations are in Hindi, but the script they use is English. Like so many other thousands of people I have been following it and even purchased the book that came out. But maybe it wouldn’t have been the same amount of fun if the script was in Devanagari. I don’t read it fast enough. And no, in this case translating the text won’t make any sense. There is Chacha Chaudhary for that. Or even Tintin comics. Thanks to Anandamela, most people my age have grown up reading Tintin and Aranyadeb (The Phantom) comics in Bengali. There also exist juicy versions of Captain Haddock’s abuses.

Last year I gave a talk at Akademy touching on some of these aspects of living in a multi-cultural environment. TL&DR version: the necessities that requires people to embrace so many languages – either for sheer existence or for the fringes, and how we can build optimized software and technical content. For me, its still an area of curiosity and learning. Especially the balance between practical needs and cultural preservation.

** Note about the title: bolta – Hindi:’saying’, Bengali:’wasp’. Go figure!

April 11, 2012

“The Sun Goes Around The Earth”

Posted in planetarium, Work & Technology tagged , , at 8:29 pm by runa

“THE SUN GOES AROUND THE EARTH”

If one grew up in the city of Kolkata in the 1980s and 90s, they would not be unfamiliar with the above graphiti planted on innumerable walls and lamposts. The graphiti and the adamant proponent of this theory is a legend that a generation would remember.

I was reminded of this, by a rather unfortunate turn of events that happened, on a mailing list of much repute. Just this morning, I was speaking with a colleague about how often and unknowingly we are drawn into stressful situations which make us lose focus from the task at hand. After having responded to a mail thread now crossing the 80+ mark, I wanted to step back, summarize and review this entire situation.

It all started when someone, who by his own admission is not a native speaker of Bangla/Bengali language, wanted to transcribe Sanskrit Shlokas (hymns) in the Bangla script into a digital format and requested for modifications in a in-use keymap. To what final end, is however unclear. This is not an unusual practice as there are numerous books and texts of Sanskrit that have been written in the Bengali script and this effort can be assumed as a natural progression to digitizing texts of this nature. What stands out is the unusual demand for the addition of a certain character, which is not part of the Bengali script, into a Bengali keymap (much in use) that this gentleman wanted to use to transcribe them. The situation worsens with more complications because this character is not a random one and belongs to the Assamese script.

The character in question is the Assamese character RA, written as ৰ and has the Unicode point U+09F0. This is part of the Unicode chart for the Bengali script, which is used to write Bengali, Assamese, and Manipuri (although Meitei is now the primary script for Manipuri). Although exclusively used for Assamese, this character does have a historical connection with the Bengali script. ৰ was also used as the Bengali character RA before the modern form র (Unicode point U+09B0) came into practice. At which exact point of time this change happened is somewhat unclear to me, but references to both the forms can be found as early as 1778 when Nathaniel Brassey Halhed published the A Grammar of the Bengali Language. Dr.Fiona Ross‘ extensively researched The Printed Bengali Character: Its Evolution contains excerpts from texts where the ancient form of র i.e. ৰ has been used. However, this is not the main area of concern.

Given its pan-Indian nature, Sanskrit has been written in numerous regional scripts. I remember, while at school Sanskrit was a mandatory third language of study. The prescribed book for the syllabus used the Devanagari script. On the other hand, the Sanskrit books that I saw in my home were in the Bengali script (some of my ancestors, including my maternal Grandfather were Priests and Sanskrit teachers who had their own tol). Anyway, I digress here. The main concern is around the two characters of ‘BA‘ and ‘VA‘ . In Devanagari, ‘BA‘ i.e. and ‘VA‘ i.e. are two very distinct characters with distinct pronunciations. While ‘BA is used for words that need a pronunciation such as बालक (phonetic: baa-lak), ‘VA is used for words such as विद्या (phonetic:weedh-ya). In Bengali, these two variations are respectively known as ‘Borgiyo BA‘ and ‘Antastya BA‘. However, unlike Devanagari they do not have separate characters. So both of them are represented by (U+09AC in the Unicode chart). Earlier they held two different positions in the alphabet chart, but even that has been relinquished. The pronunciation varies as per the word, a practice not dissimilar to the behaviourial aspects of the letters, ‘C‘ and ‘T‘ in English.

This is where it starts getting muddled. The gentleman in question requests for a representation of the Devanagari equivalent of the separation of BA and VA, for Bengali as well. Reason stated was that the appropriate pronunciations of the Sanskrit words were not possible without this distinction. So as a “solution” he suggested the use of the Assamese RA glyph in place of the Borgiyo BA sounds and the Bengali BA to be reserverd exclusively for the lesser used Antastya BA i.e. VA sounds. Depicted below as a diagram for ease of reference.

On the basis of what legacy this link is to be established or how the pronunciation for the two characters have been determined, meets a dead end in the historical references of the Bengali script[1].

To support his claims he also produces a set of documents[1][2] which proudly announces itself as the “New Bengali character set” (নূতন বর্ণপরিচয়/Nutan Barnaparichay) at the top of the pages. The New Bengali character set seems quite clandestine and no record of it is present in the publications from the Paschimbanga Bangla Academy, Bangla Academy Dhaka or any of the other organisations that are considered as significant contributors for the development and regulation of the language. Along with the New character set, there are also scanned images from books where the use of this character variation can be seen. However the antecedents of these books have not been clearly identified. In one of them, the same word (বজ্র) has been spelt differently in two sentences, which imho adds more confusion to the melee.

On my part, I have also collected some excerpts from Sanksrit content written in Bengali, with particular emphasis on the use of ব. Among them is one from the almanacs (ponjika) which are widely popular amongst householders and priests in everyday reference of religious shlokas and hymns.

The character in the eye of the storm i.e. the Assamse RA and its Bengali counterpart are very special characters. These form two different conjuncts each with the ‘YA’ (U+09AF that is shared by both the scripts) without changing the sequence of the characters:

র + য = র্য
র + য = র‍্য (uses ZWJ)

ৰ + য = ৰ্য
ৰ + য = ৰ‍্য (uses ZWJ)

The Bengali character set as we know it today was created by Ishwar Chandra Bidyasagar, in the form of the বর্ণপরিচয়/Barnaparichay written by him. Since much earlier, the script also saw modern advancements mostly to cater to the requirements of the printing industry. His reforms added a finality to this. The বর্ণপরিচয়/Barnaparichay still remains as the first book that Bengali children read while learning the alphabets. This legacy is the bedrock of the printed character and, coupled with grammar rules, defines how Bengali is written and used since the last 160 years. The major reform that happened after his time was the removal of the character ঌ (U+098C) from everyday use. Other than this, the script has remain unchanged. In such a situation, a New Barnaparichay with no antecedents and endorsements from the governing organisations cannot shake the solid foundations of the language. The way the language is practised allows for some amount of liberty mostly in terms of spellings mainly due to the legacy and origins of the words. Some organisations or publication houses prefer to use the conservative spellings while others recommend reforms for ease of use. The inevitable inconsistencies cannot be avoided, but in most cases, the system of use is documented for the reader’s reference. Bengali as a language has seen a turbulent legacy. An entire nation was created from a revolution centered around the language.

During this entire fiasco the inputs from the Bengali speaking crowd (me included) were astutely questioned. Besides the outright violation of the Bengali script, complications arising out of non-standard internationalized implementations which were highlighted, were waived off. What is more disappointing is the way the representatives from IndLinux handled the situation. As one of the pioneering organisations in the field of Indic localization they have guided the rest of the Indic localization groups in later years. With suggestions for implementing the above requests in the Private Use Area of the fonts (which maybe a risky proposition if the final content, font and keymap are widely distributed) and providing customized keymaps they essentially risked undoing critical implementational aspects of the Bengali and Assamese internationalization. Whether or not the claims from the original requestor are validated and sorted, personally I am critically concerned about the advice that was meted out (and may have also been implemented) by refuting the judgement of the Bengali localization teams without adequate vetting.

Note:A similar situation was seen with the Devanagari implementation of Kashmiri. Like the Bengali Unicode chart, the Devanagari chart caters to multiple languages including Hindi, Marathi, Konkani, Maithili, Bodo, Kashmiri and a few others. Not all characters are used for all the languages. While implementing Kashmiri, a few of the essential characters were not present in the Devanagari chart. However, similar looking characters were present in the Gurumukhi chart and were used while writing Kashmiri. This was rectified through discussions with Unicode, and the appropriate code points were alloted in the Devanagari chart for exclusive use in Kashmiri.

February 3, 2012

Indic Typing Booster – Bengali

Posted in planetarium, Work & Technology tagged , , , , , , at 5:30 pm by runa

My colleagues Pravin Satpute and Anish Patil have been working for sometime on a cool tool called the Indic Typing Booster. The premise for this tool is to aid users new to typing in Indian languages. Using a normal US English keyboard (i.e. the widely available generic keyboard around here) users begin typing a word in a keyboard sequence of their choice and after a couple of key presses the typing booster prompts the user with a series of words that match the initially typed in key sequences.

For instance, if the user wanted to type the word ‘कोमल’ (pronounced as: komal) in a phonetic keyboard sequence that maps क to k and ो to o, they could start by pressing ‘k’ and ‘o’ and lo and behold (no not Baba Yaga, but) a drop down menu opens up with possible words starting with ‘ को’ . From this list the user may then choose one to complete the word they had intended to type. List of words from a backend database feeds this list. Each language gets a database of its own, compiled from available text in that language. Users can add new words to the list as well.

The typing booster requires that the IBus Input Method is installed in the system. The other necessary packages to get Indic Typing Booster working are:

  • ibus-indic-table
  • <language-name>-typing-booster-<keymap-name> (i.e. for Bengali Probhat you would be looking for the bengali-typing-booster-probhat package)

If you are using Fedora, then all these packages can be easily installed with yum. If you are not, then the necessary information for download and installation is available at the Project Home page: https://fedorahosted.org/indic-typing-booster

Besides erasing the need for looking for appropriate keys while maneuvering through the inherent complications of Indic text, the typing booster could evolve into the much needed solution for Indic typing on tablets and smartphones.

After Marathi, Gujarati and Hindi, the Indic Typing Booster is now available for Bengali (yay!). The Bengali database is by far the biggest store so far, thanks to the hunspell list that was created through an earlier effort of Ankur. Pravin announces the new release here.

This is what it looks like.

So to write কিংকর্ত্যবিমূঢ়, I could either type r/f/ZbimwX or just press 4 to complete it.

Do please give the Indic Typing Booster a go and if you’d like to contribute then head over to the mailing list – indic-typing-booster-devel AT lists.fedorahosted.org or IRC channel – #typing-booster channel (FreeNode).

March 26, 2009

The Short(cut) story

Posted in Work & Technology tagged , , , at 7:23 pm by runa

The following is a mail written long time ago justifying the usage of the English shortcut keys for localized Bengali applications.


Hi everyone,

Deepayan Sarkar wrote:

I’m slightly confused about this. There are two types of shortcuts,
one in menu items etc (indicated by _ or & in the translated strings),
and one like CTRL-Q to quit an application. Which ones are we talking
about? The first type are activated by pressing ALT. Where does the
CTRL key come in? If we are talking about the second, I didn’t even
know that they could be translated. Can they? How?


First up to clarify matters, this issue concerns the alt+hotkey combinations. the ctrl+key combinations as far as I am aware cannot be translated. atleast I have never come across it ever. Hence for saving a file: alt+f+s is different from ctrl+s.

Currently, I use a system with bengali locale and interface as my primary production system. The input method I use is IIIMF. This is an application specific input method switcher. Same goes for SCIM. i.e. these two do not change the keyboard for the entire system. Earlier I have used setxkbmap which was a system level input method switcher and looks like most people in the thread are familiar with this method.

What I have come across regarding alt+hotkeys during my regular work are as follows [along with details]:

==================
#1. Dysfunctional
==================

I will not call the hotkeys non-functional, but dysfunctional. Reason being:

@ IIIMF and SCIM

-> En shortcuts work as alt+en key even when the active keyboard is a bengali keyboard [ i use probhat] -> bn shortcuts like alt+bn key do not work even when the active keyboard is a bengali keyboard.

@ setxkbmap

-> bn shortcuts work as alt+bn key or alt+shift+bn key [in case of a character like ফ] with an active bn keyboard.

-> en shortcuts work as alt+key with an active en keyboard

additional info: ctrl+s types shortcuts [which cannot be translated] did not function with an active setxkbmap bn keyboard. but functioned with an active bn keyboard on IIIMF and SCIM.

===============
#2. Inconsistent
===============

Duplication of top-level menu hotkeys as well as submenu hot-key under the same top-level menu item. for e.g. in gnome-games mines বৈশিষ্ট্য -> পূর্ণপর্দা and বৈশিষ্ট্য -> পছন্দ both have hotkeys as alt+প.

========================
#3 Partial Implementation
========================

Now there are two offspins from this one.

@ gtk overrides [for gnome]:

this is specifically for the gnome desktop. as golum was kind of confused about it let me explain in detail. currently in gedit.po file Cut (_C) is translated as কাট করুন (_C), whereas in gtk+ file it is translated as কাট(_ট)। But when populating the menu items for gedit, in some cases instead of the translation being used in gedit.po file the one from gtk+.po file is being used. [http://runa.randomink.org/AnkurBangla/gedit1.png]. I use a .mo file compiled from the original .po file and the same thing looks as this image. [http://runa.randomink.org/AnkurBangla/gedit2.png]

@ application related:

this is for applications that do not have a text editor on its primary interface. Again using the example for gnome-games mines [alternatively same game]. this application uses alt+bn key hotkeys.

** IIIMF and SCIM: bn keyboard for this particular application cannot be activated, because there is no text entry box on the primary interface. hence alt+bn hotkeys do not work. alt+en hotkey works. a text entry box appears only when the user is allowed to write in his/her name for the score.

** setxkbmap: alt+bn hotkey combination works on the main interface as setxkbmap sets the system level keyboard to bn.

================================

The reasons for the above mentioned behaviour is unknown to me and I can only comment about them as observations from the perspective of a user. Whether geeky or not, one cannot assume the requirements of a user. sometime back while doing an installation on a test system i had to resort to the hotkeys due to a malfunctioning mouse. at that point of time the shortcuts on anaconda did not function as they were in english and the keyboard used during installation is en. and unlike the gedit solution I mentioned earlier, hacking on installers is not really an available option. This issue has been resolved and I mention this only to highlight the fact that requirements from users can be varied and at times maybe due to unexpected circumstances.

Secondly, the issue regarding consistency between KDE and Gnome. Barring contexts I guess issues for both the desktops ought to be dealt with separately. Yet, known issues in gnome can be used to reference any similar issues arising in KDE and vice versa. I guess kcontrol would be apt example in this case where multiple backend files are being used and consistency is a key element. Similar to the gedit+gtk scenario.

Given that we have come across multiple results, it might be a good idea to go behind the scenes to figure out where exactly things are going wrong. Whether using en shortcuts is a regressive step backward is somewhat fuzzy as of now. Currently, the bn shortcuts are comparitively more dysfunctional and inconsistent. If we need to implement bn hotkeys successfully, first we need to get our homework done and check in the inconsistency factor. Secondly, given the fact that most distros are shipping with IIIMF and SCIM as the default input method framework for localized versions can we afford to promote a bengali desktop that shows stark flaws on the primary desktop interface. [refer #1]. To conclude, imho, it is always better to provide a functional interface that would be open to change and improvement in the future rather than restricting usage in the present.

regards
Runa

January 9, 2008

Glossaries for Bengali India (bn_IN) l10n – Repetitive Data

Posted in Work & Technology tagged , , at 12:11 am by runa

Sometime back I had compiled a few sets of repetitive data like Country Names, Timezones and Language Names that are regularly used across various translation projects. The information is seldom required to be changed. Since these are mostly proper nouns, the important part is correct transliteration according to pronounciation and consistency of spellings. The translation or rather transliteration was done after checking the following:
1. key to pronounciation in the encyclopedia
2. existing Bengali version if any

The result is up for RFC at my document repository, under Bengali Localization Resources.

The PDF and .po versions for Country Names, Language Names and Timezones are available as well.

May 30, 2006

Modified Bengali India (bn_IN) localedef file

Posted in Work & Technology tagged , , at 6:51 pm by runa

The Bengali India localedef file contained some minor error e.g. Tuesday and tue were written as ‘মঙগলবার’ and ‘মঙগল’ instead of ‘মঙ্গলবার’ and ‘মঙ্গল’.
The file has been modified and can be obtained from here. To use this in your system one has to follow the instructions from either here or here.

Also noticed another interesting thing in FC-5 [probably appilcable to earlier versions as well]. The text displayed on the “Previous” and “Forward” button in Firstboot are obtained from the gtk+ file. Hence the discrepancy in keyboard shortcuts and translated text.

March 31, 2006

Posted in Work & Technology tagged , , at 7:14 pm by runa

This is an interesting thing that Bhaskar has pointed out to me today. Aparently, the BBC Bengali Website uses some wierd font+encoding combination in which the Darhi or the period sign is mapped to a currently blank Unicode point – U+09FB.

[click on image for a bigger view]

The boxed representation of the incorrect code point is shown in red in the above picture.

[click on image for a bigger view]

If one looks at the above picture closely the line is red represents an empty slot at U09FB in the current Unicode chart.

In case you are missing the point, fonts that follow the Unicode encoding should be in tune with the Unicode approved codepoints for each separate alphabet in a language. In case fonts are not compliant with the Unicode standards, the output of usage of such fonts will vary in systems that do use Unicode fonts. And when some new alphabet is assigned to the codepoint that is being misused, then the fonts would break as it would have a conflict in actual representation in the font and Unicode assigned point.

Currently, the darhi, double darhi and nukta signs are borrowed from the devnagari charts at U+0964, U+0965 and U+093C in quite a few languages.

Curiously, Ekushey seems to have developed a set of fonts that use this codepoint for the darhi sign.

Follow

Get every new post delivered to your Inbox.