stevenmiller888.github.io - Mind: How to Build a Neural Network (Part Two)

Mind: How to Build a Neural Network (Part Two)

Stevenmiller888.github.io Spined HTML

Mind: How to Build a Neural Network (Part Two) Steven Miller Engineering Manager at Segment Twitter GitHub RSS Follow @stevenmiller888 Home Mind: How to Build a Neural Network (Part Two) Thursday, 13 August 2015 In this second part on learning how to build a neural network, we will swoop into the implementation of a flexible library in JavaScript. In specimen you missed it, here is Part One, which goes over what neural networks are and how they operate. Building the Mind Building a well-constructed neural network library requires increasingly than just understanding forward and when propagation. We moreover need to think well-nigh how a user of the network will want to configure it (e.g. set total number of learning iterations) and other API-level diamond considerations. To simplify our subtitle of neural networks via code, the lawmaking snippets unelevated build a neural network, Mind, with a single subconscious layer. The very Mind library, however, provides the flexibility to build a network with multiple subconscious layers. Initialization First, we need to set up our constructor function. Let’s requite the option to use the sigmoid vivification or the hyperbolic tangent vivification function. Additionally, we’ll indulge our users to set the learning rate, number of iterations, and number of units in the subconscious layer, while providing sane defaults for each. Here’s our constructor: function Mind(opts) { if (!(this instanceof Mind)) return new Mind(opts); opts = opts || {}; opts.activator === 'sigmoid' ? (this.activate = sigmoid, this.activatePrime = sigmoidPrime) : (this.activate = htan, this.activatePrime = htanPrime); // hyperparameters this.learningRate = opts.learningRate || 0.7; this.iterations = opts.iterations || 10000; this.hiddenUnits = opts.hiddenUnits || 3; } Note that here we use the sigmoid, sigmoid-prime, htan, and htan-prime npm modules. Forward Propagation The forward propagation process is a series of sum products and transformations. Let’s summate the first subconscious sum with all four input data: This can be represented as such: To get the result from the sum, we wield the vivification function, sigmoid, to each element: Then, we do this then with the subconscious result as the new input to get to the final output result. The unshortened forward propagation lawmaking looks like: Mind.prototype.forward = function(examples) { var vivify = this.activate; var weights = this.weights; var ret = {}; ret.hiddenSum = multiply(weights.inputHidden, examples.input); ret.hiddenResult = ret.hiddenSum.transform(activate); ret.outputSum = multiply(weights.hiddenOutput, ret.hiddenResult); ret.outputResult = ret.outputSum.transform(activate); return ret; }; Note that this.activate and this.weights are set at the initialization of a new Mind via passing an opts object. multiply and transform come from an npm module for performing vital matrix operations.WhenPropagationWhenpropagation is a bit increasingly complicated. Let’s squint at the last layer first. We summate the output error (same equation as before): And the equivalent in code: var errorOutputLayer = subtract(examples.output, results.outputResult); Then, we determine the transpiration in the output layer sum, or delta output sum: And the code: var deltaOutputLayer = dot(results.outputSum.transform(activatePrime), errorOutputLayer); Then, we icon out the subconscious output changes. We use this formula: Here is the code: var hiddenOutputChanges = scalar(multiply(deltaOutputLayer, results.hiddenResult.transpose()), learningRate); Note that we scale the transpiration by a magnitude, learningRate, which is from 0 to 1. The learning rate applies a greater or lesser portion of the respective welding to the old weight. If there is a large variability in the input (there is little relationship among the training data) and the rate was set high, then the network may not learn well or at all. Setting the rate too upper moreover introduces the risk of ‘overfitting’, or training the network to generate a relationship from noise instead of the very underlying function. Since we’re dealing with matrices, we handle the semester by multiplying the delta output sum with the subconscious results matrices’ transpose. Then, we do this process then for the input to subconscious layer. The lawmaking for the when propagation function is below. Note that we’re passing what is returned by the forward function as the second argument: Mind.prototype.back = function(examples, results) { var activatePrime = this.activatePrime; var learningRate = this.learningRate; var weights = this.weights; // compute weight adjustments var errorOutputLayer = subtract(examples.output, results.outputResult); var deltaOutputLayer = dot(results.outputSum.transform(activatePrime), errorOutputLayer); var hiddenOutputChanges = scalar(multiply(deltaOutputLayer, results.hiddenResult.transpose()), learningRate); var deltaHiddenLayer = dot(multiply(weights.hiddenOutput.transpose(), deltaOutputLayer), results.hiddenSum.transform(activatePrime)); var inputHiddenChanges = scalar(multiply(deltaHiddenLayer, examples.input.transpose()), learningRate); // retread weights weights.inputHidden = add(weights.inputHidden, inputHiddenChanges); weights.hiddenOutput = add(weights.hiddenOutput, hiddenOutputChanges); return errorOutputLayer; }; Note that subtract, dot , scalar, multiply, and add come from the same npm module we used surpassing for performing matrix operations. Putting both together Now that we have both the forward and when propagation, we can pinpoint the function learn that will put them together. The learn function will winnow training data (examples) as an variety of matrices. Then, we assign random samples to the initial weights (via sample). Lastly, we use a for loop and repeat this.iterations to do both forward and wrong-side-up propagation. Mind.prototype.learn = function(examples) { examples = normalize(examples); this.weights = { inputHidden: Matrix({ columns: this.hiddenUnits, rows: examples.input[0].length, values: sample }), hiddenOutput: Matrix({ columns: examples.output[0].length, rows: this.hiddenUnits, values: sample }) }; for (var i = 0; i < this.iterations; i++) { var results = this.forward(examples); var errors = this.back(examples, results); } return this; };Increasinglyinformation well-nigh the Mind API here. Now you have a vital understanding of how neural networks operate, how to train them, and moreover how to build your own! If you have any questions or comments, don’t hesitate to find me on twitter. Shout out to Andy for his help on reviewing this. Additional Resources Neural Networks Demystified, by Stephen Welch Neural Networks and Deep Learning, by Michael Nielsen The Nature of Code, Neural Networks, by Daniel Shiffman Artificial Neural Networks, WikipediaVitalConcepts for Neural Networks, by Ross Berteig Artificial Neural Networks, by Saed Sayad How to Decide the Number ofSubconsciousLayers and Nodes in aSubconsciousLayer How to Decide size of Neural Network like number of neurons in a subconscious layer & Number of subconscious layers?