Computación y señales sociales

Bla, bla, bla

Computación y señales sociales

Jordi Vitrià

Universitat de BarcelonaDepartament de Matemàtica Aplicada i Anàlisi

&&Centre de Visió per Computador

Computación y señales socialesPág. 2


Index

A new look to human behavior

A new opportunity for machine perceptionA new opportunity for machine perception

Social Signals ProcessingSocial Signals Processing


A new look to human behavior

Racism and racial segregation.

The thin‐slice theoryy

FROM TWO APPARENTLY NON‐CONNECTEDFROM TWO APPARENTLY NON CONNECTED TOPICS TO A NEW COMPUTING PARADIGM…


Thomas C. Schelling,2005 Economy Nobel Laureate


Nobel Prize winner Thomas C. Schelling in a 1971article applied game theory to explain on aarticle applied game theory to explain, on ahypothetical level, how racially mixed neighborhoodscould quickly become segregated even if the peoplecould quickly become segregated, even if the peopleinvolved are not very segregationist.


THE BIRTH OF SOCIAL PHYSICS/NETWORK SCIENCE

Schelling’s game implies that social outcomes needn’t, l i fl i b i hat least in some cases, reflect in any obvious way the

desires or intentions, habits or attitudes of anyone t llat all.

Hi i li i i i h i l h i hHis implicit point is that social happenings can have simple origins and are subject to laws not unlike those of physics (complexity, network analysis, small world effects, power laws, etc.)


Nalini Ambady & the Thin Slice Theory


THE EXPERIMENT

In a 1993 study published in the Journal of Personality and SocialP h l (V l 64 N 3) A b d d ll id t dPsychology (Vol. 64, No. 3), Ambady and a colleague videotaped13 graduate teaching fellows as they taught their classes.

She then took three random 10‐second clips from each tape,combined them into one 30‐second clip for each teacher andcombined them into one 30‐second clip for each teacher andshowed the silent clips to students who did not know theteachers. Students assigned a rate to each teacher.g

Then, Ambady correlated that rating with the teachers' end‐of‐

THIN SLICE of BEHAVIOR

, y gsemester evaluations from actual students.


THE RESULTTHE RESULT

"We were shocked at how high the correlationgwas," she says. It was 0.76. In social psychologyanything above 0.6 is considered very strong.y g y g

These results have been re‐validated in several experimentsand extended to other tasks and environments.


THIN SLICING and PERCEPTION

What is important is not what we say but h b h d i h i ihow we behave during the interaction.

Voice Face Body

Characteristics

Pitch

Expressions

Frowning

Head Postures

Head NodLoudnessSpeaking rateTurn taking…

gSmilingLip‐PoutTense‐Mouth…

…

Head NodHead ShakeHead Tilt‐back…

Body Gestures

Gaze

Who is she looking at?Gaze dynamics

y

Shoulder shrugForward leanHand behind head…

Conjugate lateral eye movementFast blink…


THIN SLICING THEORY

Thin slices have been shown to have predictivevalidity in a number of different contexts:validity in a number of different contexts:

• teaching (teacher expectancy on studentsteaching (teacher expectancy on students, teacher bias, teacher effectiveness), • learning (student achievement),learning (student achievement), • job performance (telephone operators, sales managers, management consultants),managers, management consultants), • employment interviews, • health care (doctors effectiveness, patienthealth care (doctors effectiveness, patient satisfaction), etc.


THIN SLICING THEORY

One of the most impressive examples of thin slicesof data predicting important, long‐termconsequences is marital research conducted byGottman and his colleagues.

Carrère and Gottman (1999) were able to predictmarital outcomes over a six year period based onhuman social signaling micro‐coding over just thefirst 3 minutes of a marital conflict (i.e., a thin sliceof expressive behavior).


• Importance interest and other social attitudes are reflected in• Importance, interest and other social attitudes are reflected in speaking style and body language.• People automatically perceive this information and can use it p y pto accurately predict behavior (even being unaware of it!).


Are we taking about emotions?Are we taking about emotions?


Paul Ekman

Are we taking about emotions?


P. Ekman is the most well‐known advocate of this approach, which is basedroughly on the theory that people perceive others’ emotions throughstereotyped displays of facial expression, tone of voice, etc.

The simplicity and perceptual grounding of this theory has recently given riseThe simplicity and perceptual grounding of this theory has recently given riseto considerable interest in the computational literature .

Are we taking about emotions?


Conclusions

1. Social psychology, behavioral economics and otheri l i l i h h hexperimental social sciences have shown that some

interesting human behaviors are not directly related toour conscious experience but are driven by ratherour conscious experience, but are driven by rather“simple” unconscious decision mechanisms.

2 Moreover some of these decision mechanisms are based2. Moreover, some of these decision mechanisms are basedon the (unconscious) perception of social signals that are(unconsciously) emitted by all of us when interacting with( y) y gothers.

3 These two approaches defy the classical “cognitive3. These two approaches defy the classical cognitivescience” approach for understanding human behavior.


Conclusions

A fundamental assumption of cognitive science is thatthe individual is the correct unit of analysis forunderstanding human intelligence.

These approaches present evidence that the socialnetworks containing the individuals are an importantadditional unit of analysis, and that this `networkintelligence’ is significantly mediated by non‐linguisticprocesses.


And what about computer perception of the world?

A new opportunity for machineA new opportunity for machine perceptionp p


The Original Sin of Computer Vision

Computer Vision models are based/inspired in

The Original Sin of Computer Vision

Computer Vision models are based/inspired incognitive models from the beginning (Marr & thetheory of reconstruction)theory of reconstruction).

This models are based on the individual and do notThis models are based on the individual, and do notconsider the social world in which we live.

People is considered just an another “object” that canbe identified tracked etcbe identified, tracked, etc.


One legend in AI: In 1967 Marvin Minsky assigned a graduate student (Gerald Sussman) to the problem of computer vision, believing that the problem could be largely solved within one summer.


The summer project was not successful, but its approach to solve the problem has not changed a lot from then….

Definition:

As a scientific discipline, computer vision is concerned with thetheory and technology for building artificial systems that obtaintheory and technology for building artificial systems that obtaininformation from images. The image data can take many forms,such as a video sequence, views from multiple cameras, or multi‐di i l d t f di ldimensional data from a medical scanner.

obtain information from images =

physical word description


From David Marr's book: Vision 1982From David Marr s book: Vision, 1982.


What about understanding people?

THE CANONICAL VIEW

1 Th i t d f t th t1. There is a great need for computer programs that can describe and predict people activities from video,

2. This is difficult to do, because it is hard to identify and2. This is difficult to do, because it is hard to identify and track people in video sequences, because we have no common vocabulary for describing what people are doing, and because the interpretation of what people are doing depends very strongly on context.

That’s true, but this is not the whole truth: there is also a lack of appropriate models for understanding pp p g

people and their social world.


Artificial H man Perception is foc sed onArtificial Human Perception is focused on:

Identity, gender, ethnicity, age, etc.EmotionsActivities

Gestures (for interaction)Gestures (for interaction)

But this model of human behavior is not complete…


From: A.Vinciarelli, M.Pantic, H.Boulard, Social signal processing: Survey of an emerging domain, Image and Vision Computing, Volume 27, Issue 12, November 2009, Pages 1743‐1759


Social Signals

Voice Face Body

Characteristics

PitchLoudnessSpeaking rate

Expressions

FrowningSmilingLip‐PoutTense Mouth

Head Postures

Head NodHead ShakeHead Tilt‐back

… Tense‐Mouth…

Gaze

……

Body Gestures

Shoulder shrugForward lean

Who is she looking at?Gaze dynamicsConjugate lateral eye movementFast blink…

Forward leanHand behind head…


What we know about social signaling?

It’s short term: People need about 30+ seconds of observation to infer information from it.

It’s important: there are some real social contexts where human decisions are mainly (but not exclusively) basedhuman decisions are mainly (but not exclusively) based on the content conveyed by this specific, non consciouscommunication channel between humans.

It’s a collective game: Information is mainly encoded in h d ( d b lthe interaction dynamics (voice and gestures = nonverbal cues).

Turn‐taking, emphasys, Arms, body, face, eyes...


We can built social signal detectors by analyzing speech andbody movementsbody movements.

This signals can be aggregated in communicative patterns(some of them were proposed by Charles Darwin!)(some of them were proposed by Charles Darwin!)

1 Influence patterns (speaking pitch patterns of turn taking1. Influence patterns (speaking pitch, patterns of turn taking, etc.) It is an indicator of dominance.

2. Activity patterns (energy expended in the communication) It . Activity patterns (energy expended in the communication) Itis an indicator of interest and excitement.

3. Mimicry patterns (reflexive copying of smiles, interjections, y p py g jhead nodding) It is an indicator of empathy.

4. Consistency (emphasys and timing). It is an indicator of determination (+) and openess (‐).


F iFunctions

1. Forming impressions.

2. Expressing emotions.2. Expressing emotions.

3. Sending relational messages (power, persuasion,dominance deception )dominance, deception…)

4. Managing interactions.

5. Etc.


Social cues Example social behaviours Tech.

Emotion Personality Status Dominance Persuasion Regulation Rapport Speech analysis Computer vision Biometry

Physical appearance√ √ √ √Height √ √ √ √

Attractiveness √ √ √ √ √ √ √

Body shape √ √ √ √

Gesture and posture

Hand gestures √ √ √ √ √ √ √

Posture √ √ √ √ √ √ √ √ √Walking √ √ √ √ √

Face and eyes behaviourFace and eyes behaviourFacial expressions √ √ √ √ √ √ √ √ √

Gaze behaviour √ √ √ √ √ √ √ √

Focus of attention √ √ √ √ √ √ √ √

Vocal behaviourProsody √ √ √ √ √ √Turn taking √ √ √ √ √ √ √

Vocal outbursts √ √ √ √ √ √ √√ √ √ √ √ √ √

Silence √ √ √ √

Space and environmentDistance √ √ √ √ √ √

SeatingSeatingarrangement √ √ √ √

From: A.Vinciarelli, M.Pantic, H.Boulard, Social signal processing: Survey of an emerging domain, Image and Vision Computing, Volume 27, Issue 12, November 2009, Pages 1743‐1759


Social Signal Processing


H ?How?

1. Processing real‐time video/audio and detecting andanalyzing social signals.

2. Building rich descriptions from these signals (socialmessages).g )

3. Understanding syntax and semantics of socialmessages (social language) by mining examples andmessages (social language) by mining examples andlearning to predict.


Example: Reading FacesExample: Reading Faces

• Demographics: Identity, age, gender…

• Emotional ExpressionsEmotional Expressions

• Lip Reading

• ?


Reading Faces

People automatically evaluate faces on multiple trait dimensions and thesemultiple trait dimensions, and these evaluations predict important social outcomes, ranging from electoral success to sentencing decisionsto sentencing decisions.

Aggressiveness, dominance, confidence, attractiveness, trustworthiness,attractiveness, trustworthiness, competence, etc.


DominanceDominance


CompetenceCompetence


Body Behavior + Non Verbal SpeechBody Behavior + Non Verbal Speech

(MIT)

1. Speed dating prediction.

2 Call center customer2. Call center customer satisfaction.

3 “One minute elevator3. One‐minute elevator pitch” assessment.

4 Depression monitoring4. Depression monitoring.

5. Etc.


In these scenarios, we can get anwe can get an outcome

prediction with a level of accuracylevel of accuracy between 80 and

90%


Organization Dynamics:

Other experiments show that it ispossible to identify:p y

•Connectors and central peoplel kin a social network

•The boss in an organization•The leader of a team•The leader of a team•The outcome of negotiations•The degree of persuasiveness in•The degree of persuasiveness inspeech•Group affiliations

Automatically captured group dynamicsp


Long term behavior analysis


Agitation in ICU


Dominance in computer mediated discussions

Data Blogging heads New gg g

York Times data base (http://video.nytimes.com/)

Experiments Observers inquiry Observers inquiry Manual test Automatic testSi l Signals Head nodding Speaking time (A/V)

S f l Successfulinterruptions

Floor grabbingi l i

52

Gesticulation


Social Signals and Reality ShowsCan we predict who will be fired?Can we predict who will be fired?

:

The Apprentice is a reality television show that originated in the UnitedStates on NBC. Billed as "The Ultimate Job Interview," the show depicted agroup of 15‐18 businessmen and ‐women competing in an elimination‐stylecompetition for a one‐year, $250,000 job of running one of host andexecutive producer Donald Trump's companies.


Conclusions• Social signal processing can contribute to the understanding of the

role of non‐verbal behaviour during social interactions (a new toolrole of non‐verbal behaviour during social interactions (a new toolfor social sciences).

• The range of application areas for socially aware systems toucheson many aspects of computing, and as computing becomes morebi i i ll f i i i h bjubiquitous, practically every aspect of interaction with objects,

and the environment, as well as human‐human interaction willmake use of these techniques (situated communication supportq ( pptools, human‐computer interaction tools, intelligent tutoringsystems, telecommunication facilities, intelligent multimediaser ers health monitorin s stems intelli ent robots )servers, health monitoring systems, intelligent robots )

Bla, bla, bla

Muchas Gracias

Computación y señales sociales

Technology

Transcript of Computación y señales sociales