Dost Shahi

Computer science for societal challenges and innovation, XXXIII series
Grant sponsor

AnnaPaola Marconi
Gianluca Campana, Alessandro Sperduti, Luciano Serafini

Project description
Semantic Interpretation of Image and Text is the process to recognize the entities and relations between them shown in the image and mentioned in the text and to align them with some ontological resource that contain structured knowledge about these entities and relations. Semantically describing entities and relations shown in an image and described in the text to machine will allow a more accurate retrieval and searching of image processing tasks. In my PhD, we are proposing a framework for Semantic Interpretation of Image and Text, which will utilize both the low level and semantic features of image and text by using background knowledge extracting from online Knowledge base. A solution for Semantic Interpretation of Image and Text requires to address the following challenges. First we have to recognize entities shown in images and describe in text. Second to make links between these entities and third to make links with the entities in Knowledge base. To solve these complex tasks, we will use state of the art methods for image object detection and textual entities recognition. Furthermore for mapping textual, and visual entities with entities in the knowledge base, we will use supervised machine learning techniques that exploits background knowledge. In order to provide a method for training our algorithms, and to evaluate properly the results we develop a dataset, which will consists of images, image captions, bounding box annotations, links between visual and textual entities, linked to Knowledge base and semantic meaning of entities.

Shahi Dost, Semantic Interpretation of Image and Text, Proceedings of the Doctoral Consortium (DC) co-located with the 17th Conference of the Italian Association for Artificial Intelligence (AI*IA 2018), Trento, Italy, November 20-23, 2018.)
Dost, Shahi, Sajid Anwer, Faryal Saud, and Maham Shabbir. "Outliers classification for mining evolutionary community using Support Vector Machine and Logistic Regression on Azure ML." In Communication, Computing and Digital Systems (C-CODE), International Conference on, pp. 216-221. IEEE, 2017