Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author...
Transcript of Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author...
![Page 1: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/1.jpg)
![Page 2: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/2.jpg)
![Page 3: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/3.jpg)
![Page 4: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/4.jpg)
![Page 5: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/5.jpg)
![Page 6: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/6.jpg)
![Page 7: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/7.jpg)
![Page 8: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/8.jpg)
![Page 9: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/9.jpg)
![Page 10: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/10.jpg)
![Page 11: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/11.jpg)
![Page 12: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/12.jpg)
![Page 13: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/13.jpg)
![Page 14: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/14.jpg)
![Page 15: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/15.jpg)
Slide Credits:Agrawal
![Page 16: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/16.jpg)
Slide Credits:Agrawal
![Page 17: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/17.jpg)
Slide Credits:Agrawal
![Page 18: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/18.jpg)
![Page 19: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/19.jpg)
![Page 20: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/20.jpg)
![Page 21: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/21.jpg)
![Page 22: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/22.jpg)
![Page 23: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/23.jpg)
![Page 24: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/24.jpg)
![Page 25: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/25.jpg)
![Page 26: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/26.jpg)
![Page 27: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/27.jpg)
![Page 28: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/28.jpg)
![Page 29: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/29.jpg)
Kolmogorov-Smirnov Test
p(Captions vs (Q+A))<0.001
![Page 30: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/30.jpg)
![Page 31: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/31.jpg)
![Page 32: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/32.jpg)
![Page 33: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/33.jpg)
LSTM : one hidden layer MLP : 2 hidden layer fc network output size 1024 1000 dropout(0.5) units tanheach word size 300 end-to-end learning cross-entropy
Deeper LSTM: two hidden layeroutput :
2048 > fc+tanh >1024
Input Vocabulary : All question words
![Page 34: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/34.jpg)
2-Channel VQA Model
Convolution Layer+ Non-Linearity
Pooling Layer Convolution Layer+ Non-Linearity
Pooling Layer Fully-Connected MLP
4096-dim
Embedding
Embedding
“How many horses are in this image?”
Neural Network Softmax
over top K answers
Image
Question
1024-dim
Slide Credits:Agrawal
![Page 35: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/35.jpg)
Ablation #1: Language-alone
Convolution Layer+ Non-Linearity
Pooling Layer Convolution Layer+ Non-Linearity
Pooling Layer Fully-Connected MLP
1k outputunits
EmbeddingNeural Network
Softmaxover top K answers
Image
“How many horses are in this image?”
Question Embedding
1024-dim
Slide Credits:Agrawal
![Page 36: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/36.jpg)
Ablation #2: Vision-alone
Convolution Layer+ Non-Linearity
Pooling Layer Convolution Layer+ Non-Linearity
Pooling Layer Fully-Connected MLP
4096-dim
EmbeddingNeural Network
Softmaxover top K answers
Image
“How many horses are in this image?”
Question Embedding
Slide Credits:Agrawal
![Page 37: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/37.jpg)
Slide Credits:Agrawal
![Page 38: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/38.jpg)
![Page 39: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/39.jpg)
Slide Credits:Agrawal
![Page 40: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/40.jpg)
Current Leaderboard
![Page 41: Untitled Presentation - Hacettepe Üniversitesiaykut/classes/spring...Untitled Presentation Author Unknown Creator Created Date 4/12/2016 2:11:59 PM ...](https://reader030.fdocuments.es/reader030/viewer/2022040214/5ec3d3649f28f64d86678ace/html5/thumbnails/41.jpg)
Questions&Discussion&Demo