Dost Shahi

Computer science for societal challenges and innovation, XXXIII series
Grant sponsor

AnnaPaola Marconi
Gianluca Campana, Alessandro Sperduti, Luciano Serafini


Project description
Semantic Interpretation of Image and Text is the process to recognize the entities and relations between them shown in the image and mentioned in the text and to align them with some ontological resource that contain structured knowledge about these entities and relations. Semantically describing entities and relations shown in an image and described in the text to machine will allow a more accurate retrieval and searching of image processing tasks. In my PhD, we are proposing a framework for Semantic Interpretation of Image and Text, which will utilize both the low level and semantic features of image and text by using background knowledge extracting from online Knowledge base. A solution for Semantic Interpretation of Image and Text requires to address the following challenges. First we have to recognize entities shown in images and describe in text. Second to make links between these entities and third to make links with the entities in Knowledge base. To solve these complex tasks, we will use state of the art methods for image object detection and textual entities recognition. Furthermore for mapping textual, and visual entities with entities in the knowledge base, we will use supervised machine learning techniques that exploits background knowledge. In order to provide a method for training our algorithms, and to evaluate properly the results we develop a dataset, which will consists of images, image captions, bounding box annotations, links between visual and textual entities, linked to Knowledge base and semantic meaning of entities.