An old adage says that a picture is worth a thousand words, owing to how picture can convey a wide variety of emotions and sentiments.
However, can a computer look at a photo and determine what emotion that image represents?
A group of researchers at the University of Rochester recently asked that question and started working on training computers to recognize sentiments portrayed by various images. The idea is that once a computer learns how to correctly do that, an algorithm could scour social media updates and posts and get an idea of what people's emotions are in certain areas, such as politics.
For example, a photo of a smiling politician waving at a crowd could convey positive emotions, such as confidence and friendliness. However, a photo of a frowning politician, looking down and not making eye contact could portray a more negative emotion, such as defeat. Humans have a way of looking at these photos and identifying how they make them feel, but it's much harder for computers.
Now, with social media, people are using more images and videos than text to convey their opinions. This means that computers looking through social media to gauge attitudes on specific things, such as political beliefs, are missing a lot of data. So if researchers task a computer with predicting an election's outcome, it's working with a limited set of data, just text.
To start with, researchers chose photos from Flickr that a separate algorithm had already labeled with specific emotions. However, that algorithm wasn't always right, so researchers got rid of those photos and worked only with those labeled correctly. With their computer starting with correctly identified photos, their system improved.
After which, it was time to work with humans. Using crowdsourcing, they had humans identify emotions in photos posted to Twitter. They then plugged these photos, along with their tags, into the computer, which improved the emotion identification system even more. In fact, the computer did so well after this that it correctly identified emotions better from Twitter images than from Twitter text.
Perhaps combined with facial recognition, this algorithm can now not only identify subjects in photos, but also identify the emotions those photos represent by that person. This could be something we see in the next election, with computers collecting data on sentiments social media users have during the election period, in turn, helping the computers predict the election results.
