Linguistic Pre-Processing of Social Media Texts

There are several existing linguistic pre-processing tools such as tokenizers, part-of-speech taggers, parsers, and named entity recognizers, with a focus on their adaptation to social media data. Following steps are carried out during text preprocessing:



  • Tokenization is the process of breaking down a text into words and sentences. We delete the punctuation and lowercase the words. A letter from the end of each word is removed until the stem is reached in the rudimentary procedure known as stemming. There are numerous exceptions in languages like English.

  • Lemmatization looks beyond word reduction and uses a language's whole vocabulary to apply a morphological analysis to words. For instance, third-person nouns are transformed to first-person, while past and future tense verbs are changed to present tense. Stemming is the process of reducing words to their root form.
  • Normalization is a process that converts a list of words to a more uniform sequence. Other operations can interact with the data and won't have to deal with problems that might jeopardize the process by converting the words to a standard format. For instance, making all terms lowercase will make searching easier
  • Word embeddings, also known as vectorization, is a technique used in natural language processing (NLP) to convert words or phrases from a lexicon into a corresponding vector of real numbers, which is then used to discover word predictions, word similarity, and semantics. Vectorization is the process of turning words into numbers.


Text representation models in NLP:



1. Bag of words

The bag-of-words approach has had considerable success in solving issues like language modelling and document classification because it is straightforward to comprehend and put into practise.It is a condensing representation used in information retrieval and natural language processing (IR). This paradigm ignores syntax and even word order in favour of multiplicity, representing a text as the collection (multiset) of its words.The bag-of-words model is frequently employed in document classification techniques in which the (frequency of) occurrence of each word is used as a feature for instructing a classifier.

2.TF-IDF

The term frequency and Inverse document frequency is calculayed for every word in the corpus, and multiplication of TF and IDF gives the document vectors. So, how is TF-IDF calculated:
Term Frequency (TF) — (No. of repeated words in sentence) / (No. of words in sentence)

Inverse Document Frequency (IDF) — log[ (No. of sentences) / (No. of sentences containing word)]

Comments

Post a Comment

Popular posts from this blog

Sentiment Analysis

Image
Sentiment analysis is contextual mining of text which identifies and extracts subjective information in source material, and helping a business to understand the social sentiment of their brand, product or service while monitoring online conversations. Creative use of advanced artificial intelligence techniques can be an effective tool for doing in-depth research. Sentiment Analysis It is the most common text classification tool that analyses an incoming message and tells whether the underlying sentiment is positive, negative our neutral Intent Analysis Intent analysis steps up the game by analyzing the user’s intention behind a message and identifying whether it relates an opinion, news, marketing, complaint, suggestion, appreciation or query An example of Sentiment analysis : Here's a case study on public data of Facebook comments: Breakdown of Sentiment for Categories Noticeably, comments related to all the categories have a negative sentiment majorly, bar one. The number of pos...
Read more

Applications of NLP in Social Media

Image
NLP in social media is particularly useful to  analyze huge amounts of text data  in a fast and cost-effective way – from internal documents, communications with customers, or all over the web.  Imagine you need to examine a sizable collection of evaluations to determine what consumers are saying about your offering.  Topic labelling and sentiment analysis can be used together to identify the most popular features or characteristics of your product as well as to gauge how people feel about them (are their comments favorable, negative, or neutral?). Aspect-based sentiment analysis is the term used to describe this method. In addition to brand monitoring, topic analysis has many other applications in business intelligence, sales and marketing, SEO, product analytics, and knowledge management. Some important use cases are: Trending Topic Detection Information Retrieval and Extraction Information Summarization Sentiment Detection Rumor detection Adult Content Filtering...
Read more

Data Collection

Image
There are various different complementary aspects of social media text analysis. The results of the information analysis could be influenced by the quality of collected input data. In order to use empirical methods of natural language processing or statistical machine learning algorithms, we need to build or acquire data for training or development, and for testing. These data sets need to be annotated. At least the test data needs to be annotated, so that we can evaluate the algorithms. The training data needs to be annotated in case the algorithms are supervised learning algorithms, while unsupervised learning algorithms can use the data as it is, without additional annotations (though they could benefit from a small annotated development data set). And while doing this, spam in the dataset must be avoided.  Social data collection depends on the intended task and application. Textual data from social media can be collected in various forms such as microblog messages, image descri...
Read more