ATR Presentation
-
Upload
oliver-wang -
Category
Documents
-
view
126 -
download
3
Transcript of ATR Presentation
Pain Avoidance Learning “Model-based” and “Model-free” systems
Oliver WangDepartment of Cognitive Neuroscience
Project Concepts
● Creation of a cognitive map of the space● Learning the values of actions as well as states
● Only learning the values of each action
Model-Based System Model-Free System
Project Basis● Neural Computations Underlying Arbitration between Model-based and Model-free Learning
by Sang Wan Lee, Shinsuke Shimojo, and John P. O’Doherty (2014)
● We hypothesized that a similar arbitration method might exist in aversion learning as it does in reward learning.
LiteratureMinimal research has been done on pain aversion learning
● Hendersen and Graham (Avoidance of Heat by Rats, 1979)● Prevost and O’Doherty (Pavlovian Aversive Learning, 2013)● Gillan and Robbins (Enhanced Avoidance Habits, 2014)
Task Design: Two-layer Markov Decision Task
● Training session followed by 2 sessions, each with 48 blocks, each with on average 5 trials ● Sequential 2-choices (L/R) to final state● Following states are determined by the choice and the probabilities of each branch at that time● 4 block conditions:
o Flexible, high uncertaintyo Flexible, low uncertaintyo Specific, high uncertaintyo Specific, low uncertainty
Block ConditionsFlexible Final state values are set and you receive the number of shocks indicatedEncourages a Model-Free strategy
SpecificBin color must match final state color to receive the number of shocks indicated. Otherwise you receive 4, the maximum number, of shocks.Encourages a Model-Based strategy
Example of 1 trial, flexible condition, high uncertainty
Probability: UncertaintyHigh uncertainty vs. Low uncertainty
1. High uncertainty refers to a (.5,.5) chance between the 2 resulting states.
2. Low uncertainty refers to a (.9,.1) chance, and thus a state (left state in the diagram to the right) is much more highly favorable.
*Uncertainty is maintained throughout each block
Behavioral Results
Participants: 16 subjects
Behavioral Results con’tObservation● Significantly higher proportion
observed in flexible, low uncertainty condition
Conclusion● Some difference in the arbitrator
must exist
Model-free Simulation
16 Simulated Subjects Alpha = .03, Beta = 1
MF Learning is able to replicate only the results of the flexible condition
Subject Choices: MB/MFLeft
● Refers to the flexible condition
● MB system is not adequate
Right● Refers to the specific
condition● MB system is
adequate
Parameters
Observation● Parameter for “learning rate for the estimate of absolute reward prediction error” is much
greater in our pain aversion task
Interpretation● Suggests a more dynamic arbitration system exists
*Pain Aversion Parameters (left bar) and Reward Based Parameters (right bar) (Lee)
Conclusion1. Both Model-free and Model-based systems
exist in aversion learning.
2. Although the arbitration process between the two systems share many similarities to reward-based learning, there exists subtle differences between the two.
Next Steps1. fMRI 2. Modeling
ありがとうございますThank you for listening.