With the rapid development and wide spread use of social networks, there are countless short text messages posted on Twitter every day. They are carriers of a great deal of information covering almost all areas, such as entertainment, education, sports, etc. In order to effectively sort out the content that we are interested in, it is fundamentally significant to organize these texts into different categories. However, the texts on Twitter are always very short. Thus, when dealing with these shorter text messages, many classic classification algorithms fail to achieve high quality classification performanceas they would have achieved on longer texts. This is according toour intuition, since short texts have the following characteristics:

· Short texts lack context. This means that messages often express the feelings of an event at the time.

· Short texts do not always follow the grammar of language. As we often see on Twitter, the text often contains emoticons, like and T_T.

The texts are short in length, generally less than 100 words. That is, these texts may fail to provide enough knowledge about the text itself, which leads to data sparseness problems.

To effectively tackle the challenges induced by such characteristics in short text classification, many well-developed methods have been devised to capture more information from short texts. The most commonly used methods are based on feature extension, which can be divided into two classes, namely based on rules and statistical approaches. The rule-based feature selection methods are proposed, such as 8F and TFIDF [1-2], to pick on the words that are shared by fewer texts. Nevertheless, using these approaches to build the feature space willlead to the sparseness problemas there are very few words shared by the original texts in the case of the short texts. The statistics-based method, from the perspective of machine learning, introduce sexternal resources to expand the short text such as public online knowledge, e.g., WorldNet and HowNet [3-4]. However, feature extension by using searching engines or external resources is computationally expensive. Manual selection of features is a fully experienced work driven by experience, intuition, as well as domain knowledge.

Nowadays, due to the popularity of deep learning, an increasing number of machine leaning communities have begun to apply deep learning models to text classification. Multi-layer neural network has the capacity to combine low-level text information into a more abstract high-level text representation, thereby enhancing the performance of the classifier model. CNN (Convolutional Neural Networks), for example, is good at extracting features andcan greatly reduce the necessity of manual feature extraction in classification tasks. Meanwhile, word embedding technology established on the neural network model is being applied to representthe semantic vector of short texts[5].

In this paper, borrowing the idea of the work of Lynks [6], we combine the text and an associated image that contains complementary information into a single classifier model. Namely, unlike the previous work, we not only analyze the text, but also use the information contained in the image. Recently, an increasing amount of works has exposed rich resources of semantic information conveyed by online user generated content: the texts and the associated images. This shows that people often attach an image when they use Twitter, and in most cases, this image is related to the content of the text.Thus,we propose a classifier model with related information drawn from the associated images that can deal with short and sparse text successfully. To our best knowledge, this is the first work that combines image and text for short text classification.

We generate image description by image caption technology and then combine the generated sentences with texts as the input of neural networks for text classification. Nevertheless, the image probably has nothing to do with the text, which gives rise to result that the information contained in the image is interference information. Therefore, we need to determine whether the information generated from the images is the available information by calculating the relevance of the image description and the text. Ultimately, we adopt a model that is a bit more complex but can yield significantly better performance compared to the most advanced approaches.

Our contributions in this paper are as follows:

· We propose a novel feature extension approach by utilizing the information contained in images for short text classification. We employ the image caption technology to generate the description of the image.

· We also take the correlation between image and text into account by introducing the similarity module.

Our experiments demonstrate that the model we proposed is indeed effective. Furthermore, the classifier we proposed can obtain nearly 10% accuracy improvement.

We organize the rest of paper as follows: firstly, we give a brief overview of works related to image caption generator and short text categorization in Section 2. Then, we describe each part of our model in Section 3. Experiments and results are presented in Section 4. Lastly, we summarize the contributions of this paper.

In the last few years, researchers in related fieldshave introducedsome new approaches to address short text classification problems [7]. A large number of works have managed to reduce the spatial dimension by integrating additional features. The most widely used feature is to represent text as a vector, e.g., Continuous Bag-of-Words(CBOW), continuous skip-gram model, and glove model [8-9]. Latent topic models like LDA (Latent Dirichlet Allocation) are often used for document classification. Blei et al. [10] developed the LDA algorithm to extract topic information from short text, which can be represented as a multinomial distribution of words in the vocabulary. Then, these potential latent topics can generate the document. However, these approaches require sufficient information of word occurrences. Some methods rely on human selected features to address the short text sparse problem. The characteristics of twitter textwere fully exploited by Sriram et al.[1], and they put forward to present the short texts by using a small set of domain-specific features (such as the existence of shortened words and slangs, opinions, reference to another user, and so on). Some studies have also introduced methods of short texts expansion by using searching engines. It is easy to see that querying these data sources online can cause long-term problems, and snap shots using these data sources can lead to outdated information.

Due to the remarkable performance of deep neural networks, many papers have been devoted to experiments on sentence-level classification tasks with the CNN model trained on pre-trained word vectors [11]. Johnson et al. [12]directly used the one-hot vector as input to the CNN to reduce the number of learning parameters of the model. Kim [13]further improved the performance by using multi-channel embedding. Santos et al. [14]combined the word-level and sentence-level features learned from the short text sequence to improve the accuracy of short text classification. These showed that a simple CNN with few hyper-parameters tuning can achieve good performance ina range of tasks. Inspired by these works, in this paper, we settled on a CNN model that is flexible but still easy enough.

As mentioned before, the information contained in the imagecan be used as a feature of the short text classification. However, understanding the content of images is hard, especially when the images are blurred. The goal of the image caption technology is to generate grammatical and consistent sentences of pictures. There are several models for generating image descriptions. Kiros et al. [15] first addressed the caption generation by introducing a neural network structure. Recently, many models based on RNN (Recurrent Neural Networks) have been proposed, which adopt encoder-decoder framework wildly used in machine translation [16]. Using this framework allows us to “translate” an image into a sentence. LSTM (Long Short Term Memory) [17] units are utilized to “decode” the feature vectors “encoded” by CNN. A generation model based on the deep recursive structure was proposed in [18-19]. By combining the deep convolutional network in computer vision with the recursive network, the natural statement describing the image was generated. In our work, we utilize a deep learning model under probabilistic framework to generate descriptions from images.

We can combine the sentences generated by image caption with the original texts for text categorization. However, as mentioned in Section 1,the information contained in the image maybe interference. Thus, we need to evaluate the correlation between the generated sentence and the text by mapping them into a joint feature space. Several complex lexical, syntactic, and semantic features were used to encode the input text pairs [20], and then various similarity measures between the representations could be obtained. Meanwhile, [13, 21] showed that convolutional neural networks can learn the low-dimensional vector from the input sentences and retain important syntactic and semantic aspects of the input sentence, which makes the results of many NLP tasks the most advanced. The distributed sentence model based on CNN (Section 3.1) is the main building block of the model in this paper. There are two potential sentence models working in parallel, projecting the generated sentence and text into vectors, which are then used to obtain the similarity score between them [22].

Our goal is to classify the short texts based on feature extension using information in images. Thus, we decided to combine texts and images into a single machine learning model, since they contain complementary information. In order to combine original texts and image contents together as a new input of CNN text classifiers, they must be projected to the same vector space. The overall framework of the model is shown in Figure 1. Furthermore, the general idea of our model is this: we extract image features through a CNN image embedder, followed by a LSTM model to generate the corresponding sentence. Then, we project the generated sentence and the original text into the intermediate feature representation, represented by the red grids in the picture. Then, the similarity score is calculated between the generated sentence and the original text to determine whether or not the two sentences are combined. Finally, the concatenated text is sorted by a CNN classification network. Below, we will describe each part of the function and module in detail.

Since the distributional sentence model is involved in both the sentences similarity module and the sentences classification module, we first propose a convolutional architecture for sentence modeling. The goal of this sentence model is to convert input sentence into an intermediate feature representation, which is then used for computing the semantic similarity or performing various sentence classification tasks.

As shown in Figure 2, this convolutional sentence model consists of a convolution layer with multiple filter sizes followed by a simple max pooling layer. It takes as input fixed-length embedding vectors (pre-trained word embedding vectors), through convolution layer and max pooling layer, until a fixed length representation is reached on the last level. Like most convolutional neural network models [13,23], we adopt a convolution filter taking on the characteristic of “ shared weights”.

We denote S∈ Rn×kas the input of the network, where n represents the length of the sentence (padded where necessary) and krepresents the dimension of word embedding. The function of convolution layers is to extract different level features from the input matrix.

Given a convolution filter ω∈ Rh×k, a feature Fi is generated from a window of h words [ wi:wi+h-1] by:

Here, b∈ Rn×k represents the bias term. f is the non-linear activation function. In this work, we take ReLU as the activation function of the convolution layers. The filter slides on the sentence S to produce a series of feature map F∈Rn-h+1. This process can be repeated for various filters with different sizes to increase the feature coverage of the model.

The function of the max pooling layer is to further abstract the features generated from convolution layer. The idea is to choose the maximal value F̂=max(F) to capture the most important feature. With the pooling layer, we can induce a fixed-length vector, which can be used as the input of the other modules.

The image caption problem can be defined as a binary (I,S) form, where I is an image andS is a sequence of target words. The overview of the image caption module can be seen inFigure 3.

The module includes two parts: “encoder” and “decoder”. The encoding part makes use of GoogLeNet [24], which “encodes” the given image into a fixed dimensional feature vector. The decoding part adopts the classical LSTM model to “decode” the feature vector to the desired output sentences. The LSTM consists of a series memory cells (see Figure 4). Each cell performs the same taskso that they can share some parameters. The output of a cell depends on the current input xtand “memory” state ht-1. As we can see from the Figure 4, there are three gates in a cell (input gate i, output gate o, and forget gate f). The function of the forget gate is to decide what information we want to discard from the state of the cell. It puts the input xt and the last hidden state ht-1 into a sigmoid layer and decides how to process the previous information of unit ct-1. Input gate also takes as input xt and ht-1, but it has a different function to update the memory cell. This step is to decide what new information we intend to “remember” in the cell state.With the function of these two gates, we can change the state of the memory unit. Finally, the LSTM unit uses an output gate, which takes the same input as the other two gates, and gets the result according to the state of the unit.

Therefore, the LSTM modelcan be trained to predict word st of the description in the case of known image I as well as all predicted words [ s0, s1, ⋯, st-1], which is defined by P( st| I, s0, s1, ⋯, st-1). Our implementation of LSTMs closely follows the one used in [25].

Where the various W* and b* are trained parameters, we are word embeddings, and σ is the logic sigmoid function. We project each word into a one-hot vector si, and the dimensions of these vectors are equal to the vocabulary size. These gates make it possible to properly handle the exploding and vanishing gradients issues that the traditional RNN suffered [26]. ht in Equation (8) is the input of the Softmax. Equation (9) canobtain the probability distribution of all possible sentence words. The image I is put only at the very beginning when t = 0, to notify the LSTM model of the image content.

We recommend maximizing the probability of a correct description of the given image directly by using the following Equation (10). Thus, we can obtain the probability of asentence by multiplying the probability of each word in the image and the probability of words before the current time.

The log likelihood sum of the correct words in a sentence sequence is defined as the loss of our model. We can optimize Equation(11) by adopting the stochastic gradient descent method throughout the training.

For all parameters of LSTM as well as word embeddings we, our goal is to minimize the above loss.

The architecture of sentences similarity module is presented in Figure 5. The inputs of the model are the output of the previous layers, which can be utilized to compute their similarity.We use the texts and images of tweets where the topic is related as positive samples, otherwise as negative samples. Here, we are going todescribe our similarity module in detail:

A distributional representation can be obtained by feeding the generated sentence and the text into our sentence models (Section 3.1), which can be denoted as xc∈Rd and xo∈Rd respectively. The similarity score is then calculated using the similarity matrix M according to Equation (12). The similarity between the xc and xo vectors can be defined as follows[27]:

Where M∈ Rd×d represents the similarity matrix, which is also the parameter of our model and is optimized through the training process (Section 4.2).

The single result xsim is a measure of the similarity (including syntactic and semantic) of two sentences. Therefore, according to Equation (13), we concatenate two intermediate vectors and the similarity index xsim into a single vector xjoin in the joint layer[27]:

This vector is passed to next hidden layer, which is fully connected. This layer can model the interactions among the concatenated representation vector. This following transformation is conducted by the hidden layer:

Where wh is hidden layer’s weight vector and α represents the non-linear function.

The similarity score between the two sentences is approximately equal to the correlation between the text and the image. Therefore, as described earlier, we can determine whether to consider the information contained in the image or not when classifying sentences according to Equation (15).

Where S is the sentence generated by LSTM model and Y is the original text. ⨁ denotes concatenation,and we concatenatethem as P= ( r1, r2,···, rl), where l is the length of the sentences (padded where necessary) and is a fixed value. Namely, if the two sentences are related, then the image caption and the original text concatenate together; otherwise, they are padded with a special character.

The last step of our model is a fully-connected deep learning model to process the text. The last layer of our model is simply the combination of both parts of the model, and it is used to produce the final classification. Readers who are interested in this subject can find more detailed discussions in [12-13,17].

The dataset we perform experiments on is MS COCO(the Microsoft Common Objects in Context), which was carefully created to provide resources for automatic image caption generation tasks[28]. Currently, the MS COCO 2014 dataset contains one million captions and more than 160,000 images. Compared to the existing image caption dataset, for example, flickr8k or flickr30k, the COCO dataset is advantageous because it has more images and annotations for training and testing. Corresponding to the problem that we need to solve, we regard MS COCO as the positive samples and add some texts and images with irrelevant content as the negative samples.

When we train models on the MS COCO dataset, we inevitably encounter over-fitting problems. In fact, supervised machine learning methods needa mass of data fortraining, whilethe dataset we used has less than 100000 images, which is far from enough. Thus, we explore several techniques to handle overfitting. In the image caption generation module, we decide to make we, the word embeddings randomly initialized. Meanwhile, for the Convolution Sentence Model (Sec 3.1), we initialized the embedding vectors with pre-trained word embedding vectors. Due to the complexity of ourmodel, we implemented some model-level techniques for avoiding over-adaptation. We tried the drop and ensemble models and explored the size of the model by weighing the number and depth of hidden units.

The parameters of the SGD (stochastic gradient descent) network are optimized by back propagation algorithm. We randomly initialized all weights except for the weights of the CNN model. We set the dimension size as 512 for the embeddings as well as the size of the LSTM memory.

The first experiment performs in the absence of the similarity module. Theexperiments are carried out on binary datasets.The combined model we proposed is larger and a bit more complex than ones that focus only on text or images. In order to understand and justify the performance gains of the model, it makes sense to look at each component separately and compare them to the final model:

The experimental result is quite dramatic as the performance of only analysing text is slightly better than analysing only images, while the computational cost is much lower. Moreover, it is striking that the images alone can also achieve good classification performance. We can draw a conclusion from Table 1 that the CNN model out performs the other two traditional text classification models (SVM and LDA). It is intuitive that the combination of both the images and the text should lead to the best performance since they presumably pick up on different things. Indeed, when we focus on the performance of the model, we see a significant increase in precision rate. Figure 6 shows the ROC curve (receiver operating characteristic curve). This curve plots two parameters: True Positive Rate and False Positive Rate. The pre-trained GoogLeNet used here is trained on 1000 classes of images in Image-Net.

The second experiment is to introduce the similarity module on the basis of the experiment one. As mentioned in the introduction, in real life, the image we sent sometimes has nothing to do with the text. In order to simulate this phenomenon as much as possible, we add some negative samples (the images and texts are irrelevant) to the COCO dataset. Experiment with our model on this dataset. The experimental results are as follows:

From the experimental results in Table 2, we can reach the conclusion that the accuracy of the classification decreases after adding the irrelevant data. With the introduction of the similarity module, the accuracy of the classification increases by nearly 10%, which also proves that our idea is correct.

This paper introduces a feature extension method that takes advantage of information in images for short text classification. We enrich the features of short texts with the help of image caption technology to capture the complementary information in images. The method we proposed for feature extension does improve short text classification. Meanwhile, we also take the relevance of images into account by adding the similarity module, and the accuracy is further improved. Then, we compare the proposed method with other existing classification models. A series of experimental results show that our proposed method is better in classifying the short text.