MTH 448/563 Data-Oriented Computing

Fall 2019

Day 26 = Day -3

Monday, November 25

Training our network

over-fitting

The loss on our training set can continue to get smaller without actual improvement in performance.

overfitting_f19.png

What's happening and how can we fix it?

regularization

example: L2 penalty on W

drop-out

ignore random selections of nodes during learning

efficiency

stochastic gradient descent

Can be more efficient to use just a subset ("mini-batch") of the inputs (images) for each step.

epoch = one cycle through all the mini-batches

Tensorflow

Simplification of network construction and automatic computation of gradient.

Reproduction of our "from-scratch" network in Tensorflow (2.0)

Convolutional network layers

A gentle introduction by Brandon Rohrer on Youtube

Convolution is a misnomer - actually cross-correlation (though they are simply related)

1D

1D cross-correlation

(g*f)(τ) =  − ∞g(t − τ)f(t)dt

Exercise: on paper.

cross-correlation_exercise.png

2D

natural generalization to functions of 2 variables. Examples: images!

Application: "scantron" code I wrote for grading multiple-choice exams, now being used in MTH 311

scantron_example.png

scantron_convolution_0.png scantron_convolution_1.png

2D example. Copy and paste this: 2d_cross-correlation.py, and here is the image maple_leaf_small.jpg (though you can use any grayscale image you like).

Regression problem

SUBJECT OF REPORT 6

Prediction of continuum-valued variables rather than categorical, for example, the positions of the pupils in images like this:

two_eyes_inkscape.svg.png

Code to draw a collection of examples like the one below: draw_one_eye.py

0012_one_eye_x-27.356_y-10.098.png

Acknowledgments: student Edwin Medrano, Adrian Rosebrock.

A lot of the needed code: one_eye_starter.py (copy and paste to notebook).

How to assess accuracy

Classification problem: confusion matrix gives more detail than overall accuracy

Regression problem: is there an analog of the confusion matrix?