Description

f
Motivation
New sources of information include massive and heterogeneous social media and contain so much affective information that are able to bring new nuances that do not substitute but complement traditional mass media. The challenges and goals of affective computing and multimedia analytics must merge to accomplish the task of providing a comprehensive analysis of each topic of interest. Automatic extraction of the information contained in multimedia data including those pieces of information coming from social media is essential nowadays for multiple purposes. Affective information plays a key role in audiovisual and textual information that must be taken into account in order to assist multimedia computing (processing, indexing, retrieval …). Despite of the progress in multimedia content analysis there is much work to be done in the field to provide effective ways of integrating all possible sources of information in this analysis: emotional cues, figurative language, stance, preference, reputation…
The aim of this project is on the one hand, to extract affective information automatically from multimedia data coming from traditional mass media but also from massive and heterogeneous social media. On the other hand, we are committed to find efficient methods to manage and integrate these new sources of information into multimedia analytics systems and to provide effective ways of including affective aspects into natural and inclusive human machine interaction systems.
Objectives
Which aspects of this project contribute to progress the frontiers of knowledge? How these advances are going to be achieved? and Why these achievements are necessary and how will impact in the society? The answer to the first question constitutes our Strategic Objectives, those areas where the consortium is planning its contribution in progressing the frontiers of knowledge within the area of affective multimedia analytics and natural and inclusive interaction. The second question is related with the Scientific and Technological Objectives, that describe specific techniques and methods which will contribute to fulfill the strategic objectives. Finally, through the answer to the third question, the Impact Objectives, we will outline some of the areas of application, in which the progress made can be used. These aspects are part of the demonstrator in which the achievement of the project and the knowledge transfer to the society will become apparent.

Strategic objectives

With this proposal our goal is to progress a set of diverse technologies and use them to deal with affective analysis on multimedia documents and affect-aware person-computer interactive systems. We are committed to contribute to an improved study on all kind of sources of information including traditional broadcast media and new massive and heterogeneous social media. We aim at proposing novel technological solutions to support a comprehensive information extraction of multimedia sources that includes

Developing audio, image, speech and language technologies devoted to

  1. Multimedia information extraction and processing
  2. Affect aware Multimedia Analytics
  3. Natural, affective and inclusive communication

Transferring the acquired knowledge to the society through dissemination and technology transfer actions

Scientific-Technological objectives

Following the project structure, our scientific-technological goals are:

To develop technologies for audio, video, speech and text processing intended to

  1. Transcribe the speech content of multimedia documents into text
  2. Use Web of Data as a source of knowledge to improve language, understanding, and aspect-based polarity models
  3. Identify the language and the speaker automatically from the audio
  4. Analyze the video of each multimedia document to extract useful information such as scenarios or characters

To develop technologies for affective analysis intended to

  1. Extract emotional cues from video, text and audio documents
  2. Study the impact of multimedia content on users while they are watching or listening to this content.
  3. Process figurative language, detecting and interpreting pun, irony and sarcasm.
  4. Automatically detect the stance of people involved in conversations, identify the reputation of an institution or company and track trends in social media.

To develop technologies for natural, affective and inclusive communication devoted to

  1. Automatically generate reports and summaries out of the information extracted from multimedia documents using simple language
  2. Synthesize speech with affective aspects such as expressivity control through emotion and style transplantation
  3. 3. Develop person-computer interactive systems taking into account emotional and inclusive aspects including alternative and augmentative communication

Transferring knowledge objectives

To deal with the second strategic objective we propose three main objectives related to the knowledge transfer to the society:

To develop and evaluate an application demonstrator

  • Affective Multimedia analysis platform with inclusive and natural interface

To develop multimedia annotated resources and software tools freely available

To train experts in the developed technologies that may be employed by companies interested in our results.