Towards Automatic Detection of Misinformation in Social Media
Symeon (Akis) Papadopoulos - @sympap
Information Technologies Institute (ITI) / Centre for Research and Technology Hellas (CERTH)
Workshop on Tools for Video Discovery & Verification in Social Media Dec 14, 2017 @ Thessaloniki, Greece
The Tweet Verification Assistant
A web-based service for marking an input tweet as “real” or “fake”
2012 first ideas and experiments (SocialSensor)
2013-2016 main research, development and validation (REVEAL)
2016-now incremental refinements and testing (InVID)
Tweet Verification Corpus
• 53 events or hoaxes involving false and/or real imagery and videos
• 257 cases of “fake” content, 261 of “real”
• 10,634 tweets sharing “fake” content, 7,223 tweets sharing “real” content
• Examples events: • Hurricane Sandy
• Boston Marathon bombing
• Sochi Olympics
• MA Flight 370
• Nepal Earthquake…
https://github.com/MKLab-ITI/image-verification-corpus
The “Verifying Multimedia Use” Task
•VMU: Organized in 2015 and 2016 as part of the MediaEval benchmarking initiative
•Goal: compare automated approaches for fake tweet detection
•Outcomes: several methods from different research groups across the globe were tested and compared
Experimental validation
92.5% accuracy in identifying misleading posts
88-98% accuracy depending on language
(major languages tested: en, fr, es, nl)
New features, bagging and agreement-based retraining led to significant improvements! One of the top performing methods in the VMU 2015 & 2016 tasks!
Context Analysis and Aggregation
• Available at: http://caa.iti.gr
• YouTube, Facebook and Twitter videos
• metadata from APIs
• mentioned locations
• “verification”-related comments
• thumbnails for near-duplicate search
• weather at time and location of video
• video sharing on Twitter
Tip in comment led to debunking
A comment points to second 23 of the video where suddenly the snake appears out of nowhere
# verification comments too high
1550 verification-related comments out of 4219 total number of comments
Video verification experiments
• 117 fake videos and 110 real videos
• The dataset covers different types of manipulation: • staged videos,
• videos misrepresenting the depicted event,
• videos of past events claimed to be captured now,
• digitally manipulated videos.
• A supervised learning approach using credibility features extracted from video comments and video metadata managed to achieve promising accuracy:
P=72%, R=86%, F=79%
Limitations
• Models are still based on aged training data (could be affected by concept drift…)
• Results not always easy to justify or explain to end users
• A well-informed adversary can easily fool the model by emulating “credible-looking” posts
• Journalists are still expected to make the final decision!
Acknowledgements
• Christina Boididou (Feature extraction, model building, initial REST API development)
• Olga Papadopoulou (Validation, code refactoring, REST API refinement and support)
• Lazaros Apostolidis (UI/UX)
• Markos Zampoglou (Evaluation)
• Yiannis Kompatsiaris (PI)
Thank you!
http://reveal-mklab.iti.gr/reveal/fake/
http://caa.iti.gr
Get in touch!
Akis Papadopoulos [email protected] / @sympap
References
• Boididou, C., Papadopoulos, S., Kompatsiaris, Y., Schifferes, S., & Newman, N. (2014, April). Challenges of computational verification in social multimedia. In Proceedings of the 23rd International Conference on World Wide Web (pp. 743-748). ACM
• Boididou, C., Middleton, S. E., Jin, Z., Papadopoulos, S., Dang-Nguyen, D. T., Boato, G., & Kompatsiaris, Y. (2017). Verifying information with multimedia content on twitter. Multimedia Tools and Applications, 1-27
• Boididou, C., Papadopoulos, S., Apostolidis, L., & Kompatsiaris, Y. (2017, June). Learning to Detect Misleading Content on Twitter. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval (pp. 278-286). ACM
• Castillo, C., Mendoza, M., & Poblete, B. (2011, March). Information credibility on twitter. In Proceedings of the 20th international conference on World Wide Web (pp. 675-684). ACM
• Liu, M. Y., Breuel, T., & Kautz, J. (2017). Unsupervised Image-to-Image Translation Networks. arXiv preprint arXiv:1703.00848
• Papadopoulou, O., Zampoglou, M., Papadopoulos, S., & Kompatsiaris, Y. (2017, June). Web Video Verification using Contextual Cues. In Proceedings of the 2nd International Workshop on Multimedia Forensics and Security (pp. 6-10). ACM
• Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593.