Despite Telangana dialect is frequently spoken in vocal daily interactions. Oficial Telugu is the language used in books, newspapers, academic journals, and other types of literature. By incorporating Telangana slang into the writings, poetry, and dissertations, just few Telangana local authors have worked to preserve the dialect. As a consequence, Telangana only produces a little quantity of literature and written material in documentary series form. Despite numerous attempts, the Telangana language's range is still confined to vocal forms, that constitute the majority, and written forms, which make up the minority. We are attempting to build a dataset of Telangana words, that are obtained from various documents, novels, essays, plays, and everyday interactions of native speakers, in order to mitigate this barrier and enable the electronic profusion of Telangana dialect. The first phase of the work consisted of extracting some research papers relevant to the topic and gaining some more insight into the objective focused. We then moved on to collect words in the Telangana language as a second phase, i.e., making a dataset. Then using other methods such as tokenization we began with the third phase of our project to implement the proposed work where finally conversion of Telangana dialects are translated to English.
Multidialectal Ontology supports the NLP approach[
https://github.com/hashwnath (H. Sutharapu) the government without any kind of barrier and thus, would improve communication with the Mexican Ethnic Groups.
In addition to Mayan in Mexico, there are numerous dialects for various languages across the
globe. Hence, it is planned to increase the scope of the project to various other dialects by not
confining it to just Mexican dialects. As a part of expansion, we decided to move forward with
the Telangana dialect of Telugu language which is predominantly spoken in Southern Indian
states of Andhra Pradesh and Telangana [
We can create an analogous English statement by using Natural Language Processing algorithms on the information we have collected. A proper translation from English to Telangana using conventional translation services is not possible since they can only convert Telugu and is not familiar with the language’s lexicon. The work presented here hence excels in this situation since it serves as a conduit for converting English content to Telangana content. While, the earlier one could be translated into any language using conventional translation services but fails to address dialects.
The flow of the paper has been organized as follows: Section 2 discusses about Languages and Dialects as the backbone of the paper; Section 3 briefs about Literature work related to the state-of-the-art; Section 4 is the description of the work contributed in the paper; Section 5 is show the experimental work done along with the results obtained; and Section 6 concludes the paper.
The largest family of languages that are used mostly in India is the Indo-European group.
About 20 percent of Indians speak Dravidian languages. The rest 80% speak languages from the
Austroasiatic, and some other smaller linguistic groups. In terms of the number of languages
represented, India comes fourth in the hierarchy [
The majority of Telugu speakers are found in territories like Andhra Pradesh and Telangana,
where it is also the predominant language [
Dialect has various features such as vocabulary, grammar, and pronunciation on the basis of which it is distinguished from other regional varieties . A dialect has its existence only till it reaches an elite level. Much use of dialect transforms it into a national language making it act as a national identity.
Considering Telangana Dialect (of India), Telangana vernacular is a patois dialect based on Telugu which is primarily spoken in the Indian province of Telangana. The majority of people use it for chatting. The Telangana dialect’s roots can be discovered in the Sultanate Period, which began in the 13ℎ century. Other Islamic rulers, such as King Maqbul Tilangani and the Shah Imperial, later had an influence on the society of Hyderabad as well as the adjacent territories.
Telangana dialect carries a rich cultural heritage thus it is vital to protect and pass it to the future generations. Moreover, many intellectuals can pursue their formal education by using dialect form instead of the formal Telugu language. This in a way emphasizes the role of dialects role in passing their knowledge and legacy to society.
Kostareva et. al. [
Ibrahim et al. [
Schalley et al. [
Moussallem et al. [
The work presented in this paper is about the usage of NLP for converting English content to Telangana content with dialects. The work is expected as a stepping stone towards taking Indian regional language dialect (i.e., Telugu) to next level by making people aware of dialects and providing aid which enriches the vast cultural heritage.
The overall flow of the work done in this paper is depicted through Figure 2 For any language, we use sentences to communicate. The basic unit of forming a sentence is words and use grammar to group them to form a valid sentence. Hence, our initial focus is to prepare (gather) a collection of popular words used in the Telangana dialect. The sample dataset constructed is shown in Figure 3.
The dataset is constructed using a spreadsheet/Excel sheet with four entries. In the 1 column we placed Telangana word while in the second column we place its corresponding proper Telugu synonym. In the 3 and 4ℎ columns, we place its English and Spanish counterparts based on proper Telugu word using existing translation services such as google translate, etc. This collection of words are used with Natural Language Processing toolkit to map and form meaningful translations.
European Parliament Proceedings Parallel Corpus is similar to targeted work. There are translations of it available in 21 European languages including Romanic such as French, Spanish; Germanic such as English and Swedish; Slavic ehich includes Czech, Finno-Ugric and Baltic comprising Lithuanian in addition to these Greek is also included. It provides the aligned text for statistical machine translation systems. It uses a tool based on Church & Gale Algorithm (’Gale–Church alignment algorithm’)
The most crucial phase of the project is to implement the transaltion using NLP. We made use of Python language, Google Colab Integrated Development Environment(IDE) and Python libraries such as numpy, indic-nlp, pandas, string etc, for this purpose. Initially, we installed and imported the required libraries, then after taking the English text as input we tokenized it and performed text cleaning from which we built a Telangana sentence using NLP techniques.
The following Figures 4, 5, 6, 7, and 8 illustrate the results obtained upon various test sentences and compared the predicted outcome with the original result.
From the above predictions, we can conclude that the obtained translations are accurate enough to match and convey the original meaning of the given input. Table 1 compares uniqueness of the proposed work from the conventional services
In the first instance, we began with literature review of research papers on ontology, Natural Language Processing. Then comes the most challenging point of the project, the exploration of Telangana dialect because it is not as organized as formal Telugu Language and lacks relevant digital data unlike Telugu. We then collected Telangana words from newspapers, manuscripts, textbooks, and other works of native Telangana authors to prepare a dataset. In addition to this, we referred many pre-existing technologies and methodologies used by researchers to translate languages. The methods include NLP, neural machine translation, etc. At the end, the translations obtained were validated and were found accurate enough to match and convey the original meaning of the given input.
• Github Repository of project: https://github.com/hashwnath/Mexin, • Wikipedia: https://en.wikipedia.org/wiki/Telugu_language, • European Parliament Corpus: https://www.statmt.org/europarl, • Ccelms: https://ccelms.ap.gov.in/adminassets/docs/22032021112743-60587f2f8e76b.pdf
import s t r i n g p i p i n s t a l l i n d i c − nlp − l i b r a r y p i p i n s t a l l i n l t k p i p i n s t a l l g o o g l e t r a n s = = 3 . 1 . 0 a B.2. Importing Libraries import s t r i n g from numpy import a r r a y , argmax , random , t a k e import numpy a s np import pandas a s pd from i n d i c n l p . t r a n s l i t e r a t e . u n i c o d e _ t r a n s l i t e r a t e import I t r a n s T r a n s l i t e r a t o r import d i f f l i b import s t r i n g import n l t k n l t k . download ( ’ stopwords ’ ) B.3. Importing Dataset # I m p o r t i n g data −−−−−− d a t a s e t = pd . r e a d _ c s v ( ’ words . c s v ’ ) # making s a p e r a t e d a t a f r a m e s o f i n d e p e n d e n t v a r i a b l e s and d e p e n d e n t v a r i a b l e s # T e l a n g a n a words y = d a t a s e t . i l o c [ : , 0 ] . v a l u e s # t a k e a l l rows o f 1 s t column # E n g l i s h words X = d a t a s e t . i l o c [ : , 2 ] . v a l u e s # t a k e a l l rows o f 3 r d column X=[ x . lower ( ) for x in X]
# C r e a t i o n o f c o r p u s ( d i c t i o n a r y ) w o r d p a i r s = [ [X[ i ] , y [ i ] ] for i in range ( 0 , len ( X ) ) ] w o r d p a i r s = a r r a y ( w o r d p a i r s ) # making w o r d p a i r s i n t o a r r a y form c o r p u s = d i c t ( zip ( X , y ) ) B.5. Text Cleaning def c l e a n T e x t ( t e s t ) : # p u n c t u a t i o n r e m o v a l t e s t = ” ” . j o i n ( [ i for i in t e s t i f i not in s t r i n g . p u n c t u a t i o n ] ) # make t o l o w e r c a s e t e s t = ” ” . j o i n ( [ i . lower ( ) for i in t e s t i f i not in s t r i n g . p u n c t u a t i o n ] ) # remove s t o p words stopwords = n l t k . c o r p u s . stopwords . words ( ’ e n g l i s h ’ ) t e s t = ” ” . j o i n ( [ i for i in t e s t i f i not in s t r i n g . p u n c t u a t i o n ] ) return t e s t
t e s t E n g = t e s t . s p l i t ( ) # t o k e n i z a t i o n − l e x i c a l a n a l y s i s
for x in t e s t E n g : # h a n d l i n g g r a m a t i c a l d i c r e p e n c i e s i f x== ” i s ” or x== ” a r e ” or x== ” was ” or x== ” were ” : continue B.8. Making a Telangana Sentence for x in t e s t E n g : # h a n d l i n g g r a m a t i c a l d i c r e p e n c i e s i f x== ” i s ” or x== ” a r e ” or x== ” was ” or x== ” were ” : continue i f x in c o r p u s :
temp+= c o r p u s [ x ]+ ” ␣ ” B.9. Applying Neural Machine Translation