Day 22

Thursday, April 27, 2017

From last time ...

The shifted averages of each character class:

average_09.png average__0.png average__1.png average__2.png average__7.png average__8.png average__9.png average__m.png average__o.png average__r.png

Recognition of the character represented in a given image by forming the "dot products" of that image with each of the above shifted averages and saying the one that gives the largest value is our character prediction worked not too badly.

But can we do better? If we call the shifted average images "weights", can we find modified weights that will give us better accuracy in recognition? How? Perhaps we could define a scalar measure of success and then try to maximize it by following the gradient of that measure with respect to the 10x125x100 = 125000 variables under our control (the pixel values of the set of all the weights images).

A measure of error for our recognition method, and gradient descent

softmax

2017_04_27/20170427_101617.jpg 2017_04_27/20170427_101624.jpg 2017_04_27/20170427_105424.jpg 2017_04_27/20170427_105431.jpg

Exercise: Use a program like Gimp or Inkscape to draw a path of steepest descent (negative gradient of altitude, orthogonal to level sets a.k.a. contours) on the following image starting from the tip of the pointer, in order to "find" a low point in the neighborhood.

steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_adhishch_attempt_2017-04-27-09-56-29_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_aforthma_attempt_2017-04-27-09-51-22_steepest_descent_exercise-aforthma.png students_steepest_descent/thumbs320/thumb_Steepest_descent_aishanib_attempt_2017-04-27-09-52-09_edited_gradient.png students_steepest_descent/thumbs320/thumb_Steepest_descent_annakwie_attempt_2017-04-27-09-59-44_steepest_descent_exercise1.png students_steepest_descent/thumbs320/thumb_Steepest_descent_artaboni_attempt_2017-04-27-09-46-12_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_dwseaber_attempt_2017-04-27-09-56-53_steepest_descent_exercise_line.png students_steepest_descent/thumbs320/thumb_Steepest_descent_eacrow_attempt_2017-04-27-09-39-36_steepest_descent_exercise.png.png students_steepest_descent/thumbs320/thumb_Steepest_descent_hedychen_attempt_2017-04-27-09-55-50_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_huiduan_attempt_2017-04-27-09-55-50_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_jboatrit_attempt_2017-04-27-09-50-29_steepest_descent_exercise_line.png students_steepest_descent/thumbs320/thumb_Steepest_descent_jllottes_attempt_2017-04-27-09-51-53_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_kk242_attempt_2017-04-27-09-53-40_gradient_descent.png students_steepest_descent/thumbs320/thumb_Steepest_descent_knjames_attempt_2017-04-27-09-49-57_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_maggiepe_attempt_2017-04-27-09-48-35_steepest_descent_exercise_maggiepe_bigseteps.png students_steepest_descent/thumbs320/thumb_Steepest_descent_mdrandle_attempt_2017-04-27-09-50-00_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_megantus_attempt_2017-04-27-09-56-11_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_mharnold_attempt_2017-04-27-09-45-38_gradient.png students_steepest_descent/thumbs320/thumb_Steepest_descent_miasette_attempt_2017-04-27-09-56-28_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_msaditya_attempt_2017-04-27-09-54-30_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_njmoscat_attempt_2017-04-27-09-47-43_steepest_descent_exercise_steep.png students_steepest_descent/thumbs320/thumb_Steepest_descent_pshekhar_attempt_2017-04-27-09-53-40_prashant.png students_steepest_descent/thumbs320/thumb_Steepest_descent_sakarsaw_attempt_2017-04-27-09-51-00_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_srsiegar_attempt_2017-04-27-09-47-58_gradient.png students_steepest_descent/thumbs320/thumb_Steepest_descent_wzhu28_attempt_2017-04-27-09-55-01_steepest_descent_exercise.png students_steepest_descent/thumbs320/thumb_Steepest_descent_xulishen_attempt_2017-04-27-09-55-10_xulishen.png students_steepest_descent/thumbs320/thumb_Steepest_descent_yiqingzh_attempt_2017-04-27-10-00-49_steepest_descent_exercise.png

We can implement gradient descent in Tensorflow.

Gradient descent (or steepest descent) in TensorFlow

Download the Jupyter notebook supplied with this excellent video tutorial by Magnus Erik Hvass Pedersen.

Quiz: "hvass01"

As we watch and discuss the video, each student should ask at least one question via the quiz form.

Your questions and my answers on hvass01 are here

For next time:

Recognition by multilayer networks: deep learning I

Excellent explanation by Brandon Rohrer: (the 13 minutes up to discussion of gradient computation that we can skip, then from 21:34 to 23:30).

Quiz: "rohrer1" As we watch and discuss the video, each student should ask at least one question via the quiz form.

Convolutional neural networks

This weekend, please carefully watch this tutorial by Brandon Rohrer, and ask a question to Quiz: "rohrer2".

Download the Jupyter notebook supplied with this excellent video tutorial by Magnus Erik Hvass Pedersen.

Quiz: "hvass01" As we watch and discuss the video, each student should ask at least one question via the quiz form.

TensorFlow II

Another great tutorial by Magnus Erik Hvass Pedersen <https://www.youtube.com/watch?v=HMcx-zY8JSg>`_. Download the Jupyter notebook supplied with it.