dcsimg

Deep Neural Networks and .NET

WEBINAR:
On-Demand

Building the Right Environment to Support AI, Machine Learning and Deep Learning


Artificial Intelligence is quickly becoming part of everyday life. Whether you are using your PC or your mobile phone, you are making use of some variation of AI. The Web has given Artificial Intelligence wings and it just keeps on improving.

I have written about AI and .NET before. To understand the underlying concepts of Deep Neural Networks, I recommend that you read the following two articles first. These articles explain how a neural network is put together, as well as setting up a genetic algorithm.

Deep Learning

Deep Learning (or Hierarchical Learning or Deep Structured Learning) is a type of machine learning method that is based on learning data representations instead of task-specific algorithms. Deep Learning can be unsupervised, semi-supervised, or supervised.

Some Deep Learning architectures, such as deep neural networks, deep belief networks, and recurrent neural networks have been applied to the following fields:

  • Computer Vision
  • Speech Recognition
  • Natural Language Processing
  • Audio Recognition
  • Social Network Filtering
  • Machine Translation
  • Bioinformatics

Deep Neural Networks

A Deep Neural Network (DNN) is an artificial neural network that has multiple hidden layers between the input and output layers. Deep Neural Networks models complex non-linear relationships. Deep Neural Networks are usually feedforward networks in which data flows from the input layer to the output layer without looping back.

Our Project

The small program that you will build today is a small example of how to implement Deep Learning logic into your application. You can use either C# or VB.NET. Let's jump right in, shall we?

Create a new Console App and add a class to hold all the Deep Learning Logic. Name this class DeepNeuralNetwork or DNN.

Add the following fields into the class you have just created.

C#

      private int intInput;
      private int intHiddenA;
      private int intHiddenB;
      private int intOutput;

      private double[] dblInputs;

      private double[][] dbliaWeights;
      private double[][] dblabWeights;
      private double[][] dblboWeights;

      private double[] dblaBiases;
      private double[] dblbBiases;
      private double[] dbloBiases;

      private double[] dblaOutputs;
      private double[] dblbOutputs;
      private double[] dbloutputs;
      private static Random rndRand;

VB.NET

   Private intInput As Integer
   Private intHiddenA As Integer
   Private intHiddenB As Integer
   Private intOutput As Integer

   Private dblInputs As Double()

   Private dbliaWeights As Double()()
   Private dblabWeights As Double()()
   Private dblboWeights As Double()()

   Private dblaBiases As Double()
   Private dblbBiases As Double()
   Private dbloBiases As Double()
   Private dblaOutputs As Double()
   Private dblbOutputs As Double()
   Private dbloutputs As Double()

   Private Shared rndRand As Random

These fields will hold the input Weights and their corresponding biases. Add the Constructor:

C#

      public DNN(int iInput, int iHiddenA, int iHiddenB,
         int iOutput)
      {

         intInput = iInput;
         intHiddenA = iHiddenA;
         intHiddenB = iHiddenB;
         intOutput = iOutput;

         dblInputs = new double[iInput];

         dbliaWeights = CreateMatrix(iInput, iHiddenA);
         dblabWeights = CreateMatrix(iHiddenA, iHiddenB);
         dblboWeights = CreateMatrix(iHiddenB, iOutput);

         dblaBiases = new double[iHiddenA];
         dblbBiases = new double[iHiddenB];
         dbloBiases = new double[iOutput];

         dblaOutputs = new double[iHiddenA];
         dblbOutputs = new double[iHiddenB];
         dbloutputs = new double[iOutput];

         rndRand = new Random(0);

         InitWeight();

      }

VB.NET

   Public Sub New(ByVal iInput As Integer, ByVal iHiddenA _
         As Integer, ByVal iHiddenB As Integer, _
         ByVal iOutput As Integer)

      intInput = iInput
      intHiddenA = iHiddenA
      intHiddenB = iHiddenB
      intOutput = iOutput

      dblInputs = New Double(iInput - 1) {}

      dbliaWeights = CreateMatrix(iInput, iHiddenA)
      dblabWeights = CreateMatrix(iHiddenA, iHiddenB)
      dblboWeights = CreateMatrix(iHiddenB, iOutput)

      dblaBiases = New Double(iHiddenA - 1) {}
      dblbBiases = New Double(iHiddenB - 1) {}
      dbloBiases = New Double(iOutput - 1) {}
      dblaOutputs = New Double(iHiddenA - 1) {}
      dblbOutputs = New Double(iHiddenB - 1) {}
      dbloutputs = New Double(iOutput - 1) {}

      rndRand = New Random(0)

      InitWeight()

   End Sub

Add the CreateMatrix function:

C#

      private static double[][] CreateMatrix(int iRows, int iCols)
      {

         double[][] dblRes = new double[iRows][];

         for (int r = 0; r < dblRes.Length; ++r)
         {

            dblRes[r] = new double[iCols];

         }
         return dblRes;

      }

VB.NET

   Private Shared Function CreateMatrix(ByVal iRows As Integer, _
         ByVal iCols As Integer) As Double()()

      Dim dblRes As Double()() = New Double(iRows - 1)() {}
      Dim r As Integer

      While r < dblRes.Length

         dblRes(r) = New Double(iCols - 1) {}

         Threading.Interlocked.Increment(r)

      End While

      Return dblRes

   End Function

This create a matrix for each of our weights. Add the Weight functions to set random values for each weight.

C#

      private void InitWeight()
      {

         int intWeights = (intInput * intHiddenA) + intHiddenA +
            (intHiddenA * intHiddenB) + intHiddenB + (intHiddenB *
            intOutput) + intOutput;

         double[] dblWeights = new double[intWeights];

         double dblLO = -0.01;
         double dblHI = 0.01;

         for (int i = 0; i < dblWeights.Length; ++i)
         {

            dblWeights[i] = (dblHI - dblLO) * rndRand.NextDouble()
               + dblLO;

         }

         SetWeight(dblWeights);

      }

      public void SetWeight(double[] dblWeights)
      {
         int numWeights = (intInput * intHiddenA) + intHiddenA +
            (intHiddenA * intHiddenB) + intHiddenB + (intHiddenB *
            intOutput) + intOutput;

         if (dblWeights.Length != numWeights)
         {

            throw new Exception("Weight Length Error");

         }

         int j = 0;

         for (int i = 0; i < intInput; ++i)
         {

            for (int j = 0; j < intHiddenA; ++j)
            {

               dbliaWeights[i][j] = dblWeights[j++];

            }

         }

         for (int i = 0; i < intHiddenA; ++i)
         {

            dblaBiases[i] = dblWeights[j++];

         }

         for (int i = 0; i < intHiddenA; ++i)
         {

            for (int j = 0; j < intHiddenB; ++j)
            {

               dblabWeights[i][j] = dblWeights[j++];

            }

         }


         for (int i = 0; i < intHiddenB; ++i)
         {

            dblbBiases[i] = dblWeights[j++];

         }

         for (int i = 0; i < intHiddenB; ++i)
         {

            for (int j = 0; j < intOutput; ++j)
            {

               dblboWeights[i][j] = dblWeights[j++];

            }

         }

         for (int i = 0; i < intOutput; ++i)
         {

            dbloBiases[i] = dblWeights[j++];

         }

      }

VB.NET

   Private Sub InitWeight()

      Dim intWeights As Integer = (intInput * intHiddenA) + _
         intHiddenA + (intHiddenA * intHiddenB) + intHiddenB + _
         (intHiddenB * intOutput) + intOutput

      Dim dblWeights As Double() = New Double(intWeights - 1) {}

      Dim dblLO As Double = -0.01
      Dim dblHI As Double = 0.01

      Dim i As Integer

      While i < dblWeights.Length

         dblWeights(i) = (dblHI - dblLO) * rndRand.NextDouble() + _
            dblLO
         Threading.Interlocked.Increment(i)

      End While

      SetWeight(dblWeights)

   End Sub

   Public Sub SetWeight(ByVal dblWeights As Double())

      Dim numWeights As Integer = (intInput * intHiddenA) + _
         intHiddenA + (intHiddenA * intHiddenB) + intHiddenB + _
         (intHiddenB * intOutput) + intOutput

      If dblWeights.Length <> numWeights 

         Throw New Exception("Weight Length Error")

      End If

      Dim j As Integer = 0
      Dim i As Integer = 0

      While i < intInput

         While j < intHiddenA

            dbliaWeights(i)(j) = dblWeights(Math.Min(Threading. _
               Interlocked.Increment(j), j - 1))
            Threading.Interlocked.Increment(j)

         End While

         Threading.Interlocked.Increment(i)

      End While

      While i < intHiddenA

         dblaBiases(i) = dblWeights(Math.Min(Threading. _
            Interlocked.Increment(j), j - 1))
         Threading.Interlocked.Increment(i)

      End While

      While i < intHiddenA

         While j < intHiddenB

            dblabWeights(i)(j) = dblWeights(Math.Min(Threading. _
               Interlocked.Increment(j), j - 1))
            Threading.Interlocked.Increment(j)

         End While

         Threading.Interlocked.Increment(i)

      End While

      While i < intHiddenB

         dblbBiases(i) = dblWeights(Math.Min(Threading. _
            Interlocked.Increment(j), j - 1))

         Threading.Interlocked.Increment(i)

      End While

      While i < intHiddenB

         While j < intOutput

            dblboWeights(i)(j) = dblWeights(Math.Min(Threading. _
               Interlocked.Increment(j), j - 1))

            Threading.Interlocked.Increment(j)

         End While

         Threading.Interlocked.Increment(i)

      End While

      While i < intOutput

         dbloBiases(i) = dblWeights(Math.Min(Threading. _
            Interlocked.Increment(j), j - 1))

         Threading.Interlocked.Increment(i)

      End While

   End Sub

Complete the class.

C#

      public double[] Compute(double[] dblValues)
      {

         double[] dblaSums = new double[intHiddenA];
         double[] dblbSums = new double[intHiddenB];
         double[] dbloSums = new double[intOutput];

         for (int i = 0; i < dblValues.Length; ++i)
         {

            dblInputs[i] = dblValues[i];

         }

         for (int j = 0; j < intHiddenA; ++j) {

            for (int i = 0; i < intInput; ++i)
            {

               dblaSums[j] += dblInputs[i] * dbliaWeights[i][j];

            }

         }

         for (int i = 0; i < intHiddenA; ++i)
         {

            dblaSums[i] += dblaBiases[i];

         }

         Console.WriteLine("\nInternal aSums:");

         Program.ShowGrid(dblaSums, dblaSums.Length, 4, true);

         for (int i = 0; i < intHiddenA; ++i)
         {

            dblaOutputs[i] = HyperbolicTangent(dblaSums[i]);

         }

         Console.WriteLine("\nInternal aOutputs:");

         Program.ShowGrid(dblaOutputs, dblaOutputs.Length, 4,
            true);

         for (int j = 0; j < intHiddenB; ++j)
         {

            for (int i = 0; i < intHiddenA; ++i)
            {

               dblbSums[j] += dblaOutputs[i] * .dblabWeights[i][j];

            }

         }

         for (int i = 0; i < intHiddenB; ++i)
         {

            dblbSums[i] += dblbBiases[i];

         }

         Console.WriteLine("\nInternal bSums:");

         Program.ShowGrid(dblbSums, dblbSums.Length, 4, true);

         for (int i = 0; i < intHiddenB; ++i)
         {

            dblbOutputs[i] = HyperbolicTangent(dblbSums[i]);
         }

         Console.WriteLine("\nInternal bOutputs:");

         Program.ShowGrid(dblbOutputs, dblbOutputs.Length, 4,
            true);

         for (int j = 0; j < intOutput; ++j)
         {
            for (int i = 0; i < intHiddenB; ++i)
            {

               dbloSums[j] += dblbOutputs[i] * dblboWeights[i][j];

            }

         }

         for (int i = 0; i < intOutput; ++i)
         {

            dbloSums[i] += dbloBiases[i];

         }

         Console.WriteLine("\nInternal oSums:");

         Program.ShowGrid(dbloSums, dbloSums.Length, 4, true);

         double[] dblAllOut = MaxOutput(dbloSums);
         Array.Copy(dblAllOut, dbloutputs, dblAllOut.Length);

         double[] dblResult = new double[intOutput];
         Array.Copy(this.dbloutputs, dblResult, dblResult.Length);

         return dblResult;

      }

      private static double HyperbolicTangent(double val)
      {

         if (val < -20.0)
         {

            return -1.0;

         }

         else if (val > 20.0)
         {

            return 1.0;

         }

         else
         {

            return Math.Tanh(val);

         }

      }

      private static double[] MaxOutput(double[] oSums)
      {
         double dblMax = oSums[0];

         for (int i = 0; i < oSums.Length; ++i)
         {

            if (oSums[i] > dblMax)
            {

               dblMax = oSums[i];

            }

         }

         double dblScale = 0.0;

         for (int i = 0; i < oSums.Length; ++i)
         {

            dblScale += Math.Exp(oSums[i] - dblMax);

         }

         double[] dblRes = new double[oSums.Length];

         for (int i = 0; i < oSums.Length; ++i)
         {

            dblRes[i] = Math.Exp(oSums[i] - dblMax) / dblScale;

         }

         return dblRes;
      }

VB.NET

   Public Function Compute(ByVal dblValues As Double()) As Double()

      Dim dblaSums As Double() = New Double(intHiddenA - 1) {}
      Dim dblbSums As Double() = New Double(intHiddenB - 1) {}
      Dim dbloSums As Double() = New Double(intOutput - 1) {}

      Dim i As Integer
      Dim j As Integer

      While i < dblValues.Length

         dblInputs(i) = dblValues(i)
         Threading.Interlocked.Increment(i)

      End While

      While j < intHiddenA

         While i < intInput

            dblaSums(j) += dblInputs(i) * dbliaWeights(i)(j)
            Threading.Interlocked.Increment(i)

         End While

         Threading.Interlocked.Increment(j)

      End While

      While i < intHiddenA

         dblaSums(i) += dblaBiases(i)
         Threading.Interlocked.Increment(i)

      End While

      Console.WriteLine(vbLf & "Internal aSums:")
      ShowGrid(dblaSums, dblaSums.Length, 4, True)

      While i < intHiddenA

         dblaOutputs(i) = HyperbolicTangent(dblaSums(i))
         Threading.Interlocked.Increment(i)

      End While

      Console.WriteLine(vbLf & "Internal aOutputs:")

      ShowGrid(dblaOutputs, dblaOutputs.Length, 4, True)

      While j < intHiddenB

         While i < intHiddenA

            dblbSums(j) += dblaOutputs(i) * dblabWeights(i)(j)

            Threading.Interlocked.Increment(i)

         End While

         Threading.Interlocked.Increment(j)

      End While

      While i < intHiddenB

         dblbSums(i) += dblbBiases(i)

         Threading.Interlocked.Increment(i)

      End While

      Console.WriteLine(vbLf & "Internal bSums:")

      ShowGrid(dblbSums, dblbSums.Length, 4, True)

      While i < intHiddenB

         dblbOutputs(i) = HyperbolicTangent(dblbSums(i))

         Threading.Interlocked.Increment(i)

      End While

      Console.WriteLine(vbLf & "Internal bOutputs:")

      ShowGrid(dblbOutputs, dblbOutputs.Length, 4, True)

      While j < intOutput

         While i < intHiddenB

            dbloSums(j) += dblbOutputs(i) * dblboWeights(i)(j)
            Threading.Interlocked.Increment(i)

         End While

         Threading.Interlocked.Increment(j)

      End While

      While i < intOutput

         dbloSums(i) += dbloBiases(i)
         Threading.Interlocked.Increment(i)

      End While

      Console.WriteLine(vbLf & "Internal oSums:")

      ShowGrid(dbloSums, dbloSums.Length, 4, True)

      Dim dblAllOut As Double() = MaxOutput(dbloSums

      Array.Copy(dblAllOut, dbloutputs, dblAllOut.Length)

      Dim dblResult As Double() = New Double(intOutput - 1) {}
      Array.Copy(Me.dbloutputs, dblResult, dblResult.Length)

      Return dblResult

   End Function

   Private Shared Function HyperbolicTangent(ByVal val As Double) _
         As Double

      If val < -20 Then

         Return -1

      ElseIf val > 20 Then

         Return 1

      Else

         Return Math.Tanh(val)

      End If

   End Function

   Private Shared Function MaxOutput(ByVal oSums As Double()) _
         As Double()

      Dim dblMax As Double = oSums(0)
      Dim i As Integer

      While i < oSums.Length

         If oSums(i) > dblMax Then

            dblMax = oSums(i)

         End If

         Threading.Interlocked.Increment(i)

      End While

      Dim dblScale As Double = 0

      While i < oSums.Length

         dblScale += Math.Exp(oSums(i) - dblMax)

         Threading.Interlocked.Increment(i)

      End While

      Dim dblRes As Double() = New Double(oSums.Length - 1) {}

      While i < oSums.Length

         dblRes(i) = Math.Exp(oSums(i) - dblMax) / dblScale

         Threading.Interlocked.Increment(i)

      End While

      Return dblRes

   End Function

The calculated inputs, along with the biases, get computed and outputted. Add the code for the Program class or the Module to make use of the DeepNeuralNetwork Class.

C#

      static void Main(string[] args)
      {
         Console.WriteLine("\nStart Creating a 3-4-5-2 neural
            network");

         int intInput = 3;
         int intHiddenA = 4;
         int intHiddenB = 5;
         int intOutput = 2;

         DNN dnn = new DNN(intInput, intHiddenA, intHiddenB,
            intOutput);

         double[] dblWeights = new double[] {

            0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09,
               0.10,
            0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19,
               0.20,
            0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29,
               0.30,
            0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39,
               0.40,
            0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49,
              0.50,
         };

         dnn.SetWeight(dblWeights);

         double[] dblX = new double[] { 1.0, 2.0, 3.0 };

         Console.WriteLine("\nDummy Weights + Bias Values");

         ShowGrid(dblWeights, 10, 2, true);

         Console.WriteLine("\nDummy Inputs");

         ShowGrid(dblX, 3, 1, true);

         double[] dblY = dnn.Compute(dblX);

         Console.WriteLine("\nComputed Outputs");

         ShowGrid(dblY, 2, 4, true);


         Console.WriteLine("\nEnd");

         Console.ReadLine();
      }

      static public void ShowGrid(double[] dblVector, int intCols,
         int intDec, bool blnNewLine)
      {
         for (int i = 0; i < dblVector.Length; ++i)
         {

            if (i % intCols == 0)
            {

               Console.WriteLine("");

            }

            Console.Write(dblVector[i].ToString("F" +
               intDec).PadLeft(intDec + 4) + " ");
         }

         if (blnNewLine == true)
         {
            Console.WriteLine("");

         }

      }

VB.NET

   Sub Main(ByVal args As String())

      Console.WriteLine(vbLf & "Start Creating a 3-4-5-2 _
         neural network")

      Dim intInput As Integer = 3
      Dim intHiddenA As Integer = 4
      Dim intHiddenB As Integer = 5
      Dim intOutput As Integer = 2

      Dim dnn As DNN = New DNN(intInput, intHiddenA, intHiddenB, _
         intOutput)

      Dim dblWeights As Double() = New Double() {

         0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1,
         0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2,
         0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3,
         0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4,
         0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5
      }
      dnn.SetWeight(dblWeights)

      Dim dblX As Double() = New Double() {1, 2, 3}

      Console.WriteLine(vbLf & "Dummy Weights + Bias Values")

      ShowGrid(dblWeights, 10, 2, True)

      Console.WriteLine(vbLf & "Dummy Inputs")

      ShowGrid(dblX, 3, 1, True)

      Dim dblY As Double() = dnn.Compute(dblX)

      Console.WriteLine(vbLf & "Computed Outputs")

      ShowGrid(dblY, 2, 4, True)

      Console.WriteLine(vbLf & "End")

      Console.ReadLine()

   End Sub

   Public Sub ShowGrid(ByVal dblVector As Double(), _
         ByVal intCols As Integer, ByVal intDec As Integer, _
         ByVal blnNewLine As Boolean)

      Dim i As Integer

      While i < dblVector.Length

         If i Mod intCols = 0 Then

            Console.WriteLine("")

         End If

         Console.Write(dblVector(i).ToString("F" & intDec) _
            .PadLeft(intDec + 4) & " ")
         Threading.Interlocked.Increment(i)

      End While

      If blnNewLine = True Then

         Console.WriteLine("")

      End If

   End Sub

Here you supply values for the weights. The ShowGrid procedure displays the results.

Conclusion

Deep Neural Networks are not too complicated. They are just the beginning. Hopefully, in a future article, I can delve a bit deeper. Until then, good bye.



About the Author

Hannes DuPreez

Hannes du Preez is an ex MVP for Visual Basic from 2008 to 2017. He loves technology and loves Visual Basic and C#. He loves writing articles and proving that Visual Basic is more powerful than what most believe. You are most welcome to reach him at: ojdupreez1978[at]gmail[dot]com

Related Articles

Comments

  • compile error - Wekkel

    Posted by Hannes on 04/10/2018 12:27am

    Hi Wekkel. Thanks for your interest in my article. The code works. The IF statement checks to see if dblWeights.Length is not equal to numWeights then it should produce an error. Make sure all your Weight values are correct

    Reply
  • compile error

    Posted by Wekkel on 04/03/2018 08:24am

    If I run this in Visual Basic, I get an error: "Throw New Exception("Weight Length Error")" at the following lines If dblWeights.Length numWeights Then Throw New Exception("Weight Length Error") End If Debugging shows that in the given code, dblWeights.Length is 50 while numWeights is 53. I am not sure how I should fix this.

    Reply
  • You must have javascript enabled in order to post comments.

Leave a Comment
  • Your email address will not be published. All fields are required.

Most Popular Programming Stories

More for Developers

RSS Feeds

Thanks for your registration, follow us on our social networks to keep up-to-date