NNF – Neural Net Framework

An high speed implementation of feed-forward neural nets

Tutorial

Setting up the library

First download the NNF package (.zip or .tar.gz format) and extract it.

Now you'll have four directories: doc, img, lib, src.

doc contains the entire documentation; img contains some images; lib contains a precompiled Linux version of the library (“libnnf.a”); src contains the source and a Unix makefile.

If you use a Unix or Linux system you can build the static library by typing in a shell:

make

This will recompile the library and replace the library archive in the lib directory.

If you use a different OS (such as Windows) you can compile it with your own compiler (remember that MingW32 can use Unix makefiles), or simply add the source to your project.

Compiling your project with the library

If you use the library archive, you have to set up the linker giving the library as a parameter.

Let's assume you have put the library in the same directory of your source code.

On GCC you type:

gcc -o MyApp MyApp.c -L. -lnnf -lm

Note that -lm is required on Linux to link to the math library.

If you use a development environment, most of them let you set up the linker visually.

Building our first neural net application

Now that we know how to set up NNF, we can write our first simple application: a program that teaches a neural network the mathematic function “sum”.

Two numbers (0 < number < 50) will be given as input, and we expect their sum as the output.

First, we include the library:

#include "nnf.h"

Then we declare the main() function, and we set up three variables: c is the iterator for the training epochs; i and k are the iterators for the numbers that we will give as input to the neural net

int main()

{

int c,i,k;

And here is our neural net:

NNF_NeuralNet net;

Then we store an array (in[]) that will contain the input that we will give to the net, a pointer that will receive the net's output (*out), and an array of one member which will contain the desired output (des[]). neurs[] is the array that indicates the number of neurons for each layer: we have chosen to build a little net with 3 layers, with respectively 2 neurons for the input layer, 4 neurons for the only hidden layer, and of course 1 neuron which will give us the output

float in[2],*out,des[1];
int neurs[3] = { 2,4,1 };

Now we initialize the net, calling NNF_Init() and passing as arguments a pointer to the net, the number of layers, the array of the neurons, and the range of randomization of the synaptic weights

Then we set the “epsilon” value (0 < epsilon <= 1), a floating point number that indicates the learning constant: the highest we set it, the faster our net changes its weights; but if it's too high, the learning process could be wrong.

We also choose the activation function, which is the sigmoidal logistic function: 1 / (1 + e^-X).

This functions returns a value between 0 and 1.

If we had chosen the step function, we would have to define the step value.

But now we didn't, so we simply ignore it.

NNF_Init(&net,3,neurs,5);

net.epsilon = 0.6;
net.activationFunction = NNF_SIGMA_LOGISTIC;

Now we are ready to teach our neural net the function “sum”: we will give the net two values as input, but we have to make them lesser than 1.

So if we want to sum 5 and 7, we will give 5 / 100 and 7 / 100 (remember we have chosen a range of 50, so the maximum sum is 0,5 + 0,5 = 1,0).

The first loop represents the learning epochs; the other two loops represent the two values we will give as input. Then we will call the function NNF_Execute(), which accepts as parameters a pointer to the net and the input array, and returns the output computed by the net; we store it in the variable “out”.

NNF_BackPropagate() is used to indicate the net what would be the correct output, so that it'll try to recalibrate its weights.

for (c=0; c<100; c++)
for (i=0; i<50; i++)

for (k=0; k<50; k++) {
in[0] = (float) i / 100;
in[1] = (float) k / 100;
des[0] = (float) (in[0] + in[1]);
out = NNF_Execute(&net,in);
NNF_BackPropagate(&net,des);
}

We have now finished to use the net, so we free the memory and terminate the application:

NNF_Delete(&net);

return 0;

}

Ok, that's all. It was very simple, wasn't it?

Monitoring the neural network's learning progress

But now how can we know if our net has effectively learned how to sum two numbers?

If we make our application show some of the input, output and desired values, we will see how the net learns.

For example, I made this program write to screen the values of a sum in the first learning epoch, and then after 100 epochs:

learning epoch 0: 10 + 7 = 0.068219 DESIRED: 17

learning epoch 99: 10 + 7 = 17.361418 DESIRED: 17

This is representative of how a neural net can abstract a function: we only showed some examples of sums, and now our net has learned how to sum two numbers quite decently.

This time we have showed all the possible operations in a loop, but the result would be quite the same if we only showed some of them.

Another great way to monitor the network's learning path is making a graph of the medium error during the epochs: we can sum the difference between the total output and the total desired output of an epoch, then make an average of the epoch, and store it on a file.

Then we can use Gnuplot or similar programs to draw the graph.

Here is a graph that represents 100 learning epochs of the “sum” neural net:

If we also make a graph with all the particular errors of one learning epoch, we will see that the error is higher near the “limit values”: this is because the sigmoidal function, which has an asymptotic behavior, can't easily reach the limits.

Saving a neural net

If we have a complex neural network with a very large number of layers and neurons, we'll probably need to save its progress in order not to waste a lot of time every time we run the neural net.

This can be easily done with the NNF_Save() and NNF_Load() functions.

NNF_Save() and NNF_Load() accept as arguments a pointer to a neural net and a pointer to a plain text file opened with fopen():

#include <stdio.h>
...
NNF_NeuralNet net;
FILE *load, *save
load = fopen(“mynet.nnf”,”r”);
NNF_Load(&net,load);
...
save = fopen(“mynet.nnf”,”w”);
NNF_Save(&net,save);
...

The standard extension for neural nets is “.nnf”, but you can change it if you want.

If you need to work also with negative numbers, you can use the hyperbolic tangent function:

#include "nnf.h"

int main()
{
int i,k,c;
NNF_NeuralNet net;
float *out;
float in[2],des[1];
int n[4]= {2,3,3,1};
NNF_Init(&net,4,n,2);
net.activationFunction = NNF_SIGMA_HYPERBOLIC;
net.epsilon = 0.8;
for (c=0;c<500;c++)
for (i=0;i<20;i++)
for (k=0;k<20;k++) {
in[0] = (float) (i-10)/20;
in[1] = (float) (k-10)/20;
des[0] = in[0] + in[1];
out = NNF_Execute(&net,in);
NNF_BackPropagate(&net,des);
}

NNF_Delete(&net);
return 0;
}

Here are two outputs I made it write to screen:

learning epoch 0: -5 + 2 = -1.600248 DESIRED: -3

learning epoch 499: -5 + 2 = -3.071974 DESIRED: -3

Another example

Let's imagine we are studying a physic phenomena which has four variables: s, t, u, v.

We don't know the relation among that variables, but s, t and u depend from the material we choose, and v can be measured with a particular instrument.

So we'll repeat the experiment with different materials, and every time we will measure the value v.

If we give the neural net the variables s, t, u as the input and v as the desired output, we'll expect the net to abstract the relationship among the four variables, so that in the future we won't need to repeat the experiment every time we want to know the value v.

In this example, I have chosen the following relation:

v = u * s / t

For simplicity, the program that teaches the network will know the relation, but in reality the physician should store the experimental data in a file and give it to the neural net.

Here is the code, not very different from the previous:

#include "nnf.h"
int main()
{
int c,s,t,u;
NNF_NeuralNet net;
float in[3],*out,des[1];
int neurs[5] = { 3,4,5,3,1 };
NNF_Init(&net,5,neurs,5);
net.epsilon = 0.7;
net.activationFunction = NNF_SIGMA_LOGISTIC;
for (c=0;c<5000;++c)
for (s=1;s<10;s++)
for (t=1;t<10;t++)
for (u=1;u<10;u++) {
in[0] = (float) s / 10;
in[1] = (float) t / 10;
in[2] = (float) u / 10;
out = NNF_Execute(&net,in);
des[0] = (float) (in[2] * in[0] / in[1])/10; // v
NNF_BackPropagate(&net,des);
if (c == 4999 && s == 7 && t == 5 && u == 9)
printf("v: %f desired: %f",out[0]*100,des[0]*100);
}
NNF_Delete(&net);
return 0;
}

Note that we chose the number of layers and neurons casually, so it can be optimized.

The program showed an output of the last learning epoch:

v: 12.393750 desired: 12.599999

The result is quite good, and this example shows that a neural net can abstract equations that relate empirical data.

Using step and sign functions

In some cases it could be useful to use the “step” or the “sign” functions, which respectively work with { 0,1 } and { -1,1 }.

In the following example we will use a neural network in this way: we will give the net three input values; the first can be 0 or 1, the second and third are constant (1 and 0) and they really don't matter: we add them only to “confuse” the network (don't try this at home guys!).

We expect (1;0) as output if the first input value is 1, otherwise (0;1).

#include "nnf.h"
#include "math.h"
int main()
{
int i;
NNF_NeuralNet net;
int neurs[3] = {3,5,2};
float in[3],des[3];
float *out;
NNF_Init(&net,3,neurs,10);
net.activationFunction = NNF_STEP;
net.epsilon = 0.7;
net.step = 4;
for (i=0;i<50;i++) {
if (i%2)
in[0] = 1;

else
in[0] = 0;
in[1] = 1;
in[2] = 0;
des[0] = in[0];
des[1] = fabs(des[0]-1);
out = NNF_Execute(&net,in);
NNF_BackPropagate(&net,des),
}
NNF_Delete(&net);
return 0;
}

Here are the first attempts:

INPUT: 0 1 0 OUTPUT: 0 0 DESIRED: 0 1

INPUT: 1 1 0 OUTPUT: 0 0 DESIRED: 1 0

INPUT: 0 1 0 OUTPUT: 1 1 DESIRED: 0 1

INPUT: 1 1 0 OUTPUT: 1 1 DESIRED: 1 0

Not very good... and here are the last ones:

INPUT: 0 1 0 OUTPUT: 0 1 DESIRED: 0 1

INPUT: 1 1 0 OUTPUT: 1 0 DESIRED: 1 0

INPUT: 0 1 0 OUTPUT: 0 1 DESIRED: 0 1

INPUT: 1 1 0 OUTPUT: 1 0 DESIRED: 1 0

They are all exact.

Now let's try the sign function: we are giving the network two numbers from { -1,1 } and it will return 1 if they are the same, -1 otherwise.

#include "nnf.h"
int main()
{
int i;
NNF_NeuralNet net;
int neurs[4] = {2,4,5,1};
float in[2],des[1];
float *out;
NNF_Init(&net,4,neurs,2);
net.activationFunction = NNF_SIGN;
net.epsilon = 0.8;
for (i=0;i<1000;i++) {
if (i%2)
in[0] = 1;
else
in[0] = -1;
if (i%3)
in[1] = -in[0];
else
in[1] = in[0];
if (in[0] == in[1])
des[0] = 1;
else
des[0] = -1;

out = NNF_Execute(&net,in);
NNF_BackPropagate(&net,des),
}
NNF_Delete(&net);
return 0;
}

First output:

INPUT: 1 -1 OUTPUT: 1 DESIRED: -1

INPUT: -1 -1 OUTPUT: -1 DESIRED: 1

INPUT: 1 -1 OUTPUT: 1 DESIRED: -1

INPUT: -1 1 OUTPUT: -1 DESIRED: -1

...totally casual...

After all the learning epochs:

INPUT: -1 -1 OUTPUT: 1 DESIRED: 1

INPUT: 1 -1 OUTPUT: -1 DESIRED: -1

INPUT: -1 1 OUTPUT: -1 DESIRED: -1

INPUT: 1 1 OUTPUT: 1 DESIRED: 1

Totally exact.

That's all, have fun!

Copyright (C) 2005 by Alessandro Presta