NNF - Neural Net Framework

NNF – Neural Net Framework

An high speed implementation of feed-forward neural nets

Introduction to neural networks

There are many books that explain the theory of neural networks, and also very good articles all over the net.

This small introduction might be useful if you're new to the argument.

The biological neuron

Neurologists' studies are proving that the brain is the organ of intelligence, and the fundamental unit of that organ is the neuron.

There are about 10 billion neurons in the human brain.

A neuron is a cell interconnected to other neurons with dendrites, synapses and an axon, and uses biochemical reactions to send and receive information.

The body of the neuron is called soma.

A neuron can be connected to a number of other neurons that goes from 12 to 100000:.

When the signal received by the neuron is higher than a certain threshold, an electric pulse is sent through the axon.

Then other neurons receive the signal from the synapses, that can be excitatory if they increase the signal, inhibitory if they decrease it.

The interaction of neurons is the biological basis of the human thought.

Artificial neural networks

If we hypothetically try to reproduce the brain with an artificial neural network, we will have to realize a recurrent neural network, that means a very complex network that can be represented by a common graph, each node being a single neuron.

But NNF implements a particular kind of neural net, a feed-forward neural network, which is easier to train and can be represented by a directed acyclic graph.

As showed in the image above, a multilayer feed-forward network is made of a number of layers, each of them containing a variable number of neurons. Every neuron is connected to the neurons of the adjacent layers. In the simplified model, each connection is called a synapse.

The first layer is the input layer, the last is the output layer.

Each synapse is associated to a weight; each neuron has its proper bias weight.

When a signal arrives to a neuron, the neuron processes it and redirects it to the following neurons.

The activation value of a neuron is the weighted sum of all the input signals:

A = input 1 * weight 1 + input 2 * weight 2 ... + bias

The output of the neuron can be calculated with different mathematic functions; each of them has its advantages and disadvantages.

Sigmoidal logistic function: f(x) = 1 / (1 + e^-x)

Sigmoidal hyperbolic tangent function: f(x) = tanh(x) = (e^x- e^-x) / (e^x+ e^-x)

Sign function: f(x) = { 1 if x >= 0 ; -1 if x < 0 }

Step function: f(x) = { 1 if x >= step ; 0 if x < step }

The sigmoidal functions can assume continuous values, but the other side of the coin is that they don't reach the limits.

The sign and step functions can assume only two values.

The learning process is led by the delta error back-propagation algorithm: each neuron (from the output layer to the first layer) calculates its error and its delta value. Then the weights are updated. Epsilon is the learning constant, a floating point positive value (0 < epsilon <= 1).

error = desired output – effective output

delta = error * derivative of the activation function

weight 1 = weight 1 + epsilon * delta * input

bias = bias + epsilon * delta

If we use the logistic function, the derivate will be

f '(x) = e^-x/ ( e^-2x+ 2e^-x+ 1)

that can be optimized if we store the last computed value in a variable

f ' (x) = output * (1 – output)

So the delta value is

delta = error * output * (1 – output)

For the hyperbolic tangent:

f '(x) = 4 / (e^x+ e^-x)²

that corresponds to

f '(x) = 1 – output²

and delta becomes

delta = error * (1 – output²)

With the step and sign functions, the derivative is replaced by the constant value 1.

delta = error

This works quite well.

After calculating the error, the neuron back-propagates it to the neurons of the layer before, and this is done until the first layer is reached.

The potentiality of artificial neural networks

The peculiarity of neural networks against other strategies is that neural networks work at a sub-symbolic level, while traditional artificial intelligence software works at a symbolic level: concepts, ideas, functions are only words, numbers, “symbols”. In a neural network, they are represented by the synaptic weights of a group of interconnected, sub-symbolic units.

While a single neuron doesn't mean anything particular, a group of neurons can acquire a semantic value.

This is what is called abstraction: a neural network learns from examples, and tries to abstract the rule that associates the input to the output.

In this way, a neural network can learn to understand the human voice and handwritten digits, to find relations and functions in physic phenomena, to drive a car... and so on.

Certainly, there are also many contexts in which a traditional approach is preferred instead of a neural approach.

But it isn't too strange to imagine the human brain as a big, recurrent neural network.

“Brain causes mind” (Searle, 1992)