Communications in Information and Systems
Volume 23 (2023)
Special issue dedicated Professor Avner Friedman in celebration of his 90th birthday
Guest Editors: Jong-Shenq Guo, Bei Hu, Robert Jensen, and Stephen S.-T. Yau
Prediction of human looking behavior using interest-based image representations
Pages: 245 – 262
Looking behavior allows human to understand and interact with an enormous amount of information, a capacity challenging to replicate in AI systems. One of the core elements of this work is an effort to predict scan-paths from a combination of image information and past looking behavior. The success of this scan-path predication relies heavily on whether this image information can provide a sufficiently rich representation for prediction. In this paper, we show that changing representations dramatically simplifies and improves predictions of looking behavior. We introduce a representation of looking behavior that centers around interest-regions in images, defined by natural and collective looking behavior. These regions (called interest-based regions) can be used to partition images for semantic labeling and to provide a basis for shared representation across observers. Without any additional label or image information, we achieve highly accurate sequence prediction using this interest-based image representation.
This work was supported in part by the Ministry of Science and Technology of Taiwan under the grants 111-2115-M-032-005 (JSG) and 111-2115-M-126-001 (KG). We would like to thank the anonymous referee for some valuable comments. Also, appreciation for the help from Ching-Yao Chen, Yu-Fang Chou, and Chiao-Ying Cheng from Providence University, who provided help on running and evaluation of MultiMatch on SALICON and MIT1003.
Received 6 December 2022
Published 12 October 2023