{"id":20406,"date":"2024-10-21T20:32:00","date_gmt":"2024-10-21T20:32:00","guid":{"rendered":"https:\/\/liquidinstruments.com\/?p=20406"},"modified":"2025-12-03T22:12:50","modified_gmt":"2025-12-03T22:12:50","slug":"what-is-a-neural-network","status":"publish","type":"post","link":"https:\/\/liquidinstruments.com\/blog\/what-is-a-neural-network\/","title":{"rendered":"What is a neural network?","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Moku Version 3.3<\/a> brings a new Neural Network<\/a> instrument to <\/span>Moku:Pro<\/span><\/a> that enables users to implement artificial neural networks for fast, flexible signal analysis, denoising, sensor conditioning, closed-loop feedback, and more. If you are unfamiliar with the basics of a neural network or want to know how one can benefit your research and development goals, read on to explore real-life applications and tutorials.<\/span><\/p>\n

First, it\u2019s important to note that \u201cartificial neural network\u201d (ANN) is the more accurate term for these systems, since they are software-based, rather than biological neurons. With this clarification in mind, we will use the term \u201cneural network\u201d to refer to an ANN. For this introduction, we\u2019ll consider only fully connected neural networks, rather than complicated setups such as convolutional, recurrent, and transformer architectures.<\/span><\/p>\n

A neural network is a system of interconnected nodes, or neurons, arranged in layers. The first layer takes external data as its input. Each subsequent neuron computes a weighted sum of its inputs from the previous layer, and then adding a value called a bias. This value is then passed through an activation function, which introduces non-linearity, enabling the network to learn complex patterns. The final layer produces the network\u2019s output, while all layers in between are known as hidden layers. Through training, the network adjusts its weights and biases to improve its accuracy over time.<\/span><\/p>\n

In mathematical terms, imagine the input layer as an <\/span>N \u2715 1 <\/span><\/i>matrix, where <\/span>N<\/span><\/i> is the number of nodes in the input layer and each element in the matrix corresponds to the activation value, as shown in Figure 1.<\/span><\/p>\n

\"\"<\/p>\n

Figure 1: Neural network architecture, showing input, hidden, and output layers.<\/span><\/p>\n

Next are the hidden layers. The number of hidden layers and nodes within them depends on the complexity of the model and available computational power. The activations in the hidden layers are calculated using a combination of the activations from the previous layer, with each node applying different weights and biases to the input layer values. This is shown in linear algebra terms in Figure 2. <\/span><\/p>\n

\"\"<\/a><\/p>\n

Figure 2: The activations in the hidden layers are calculated using a combination of the activations from the previous layer.<\/span><\/p>\n

If the input layer is an <\/span>N \u2715 1<\/span><\/i> matrix (\\(n_1\\)<\/span>,<\/span><\/i> \\(n_2\\)<\/span>),  the next activations are obtained by multiplying this with an <\/span>M \u2715 N<\/span><\/i> matrix, where <\/span>M <\/span><\/i>is the number of nodes in the hidden layer. Each element in the matrix is a weight represented by \\(w_{mn}\\)<\/span>, meaning that <\/span>MN<\/span><\/i> parameters are needed for each layer. The result is an <\/span>M \u2715 1 <\/span><\/i>matrix, which is then offset by a bias value (\\(b_1\\),<\/i> \\(b_2\\)<\/span>\u2026). After calculating the new activations, they are passed through an \u201cactivation function\u201d. The activation function can provide nonlinear behavior such as clipping and normalizing, making the network much more powerful than it would be if it were simply a series of matrix multiplications.<\/span><\/p>\n

Each activation function has distinct properties, and the “correct” choice depends heavily on the application. Common options include ReLU (Rectified Linear Unit), which is computationally efficient but can struggle with certain training scenarios, as it truncates negative values. Others, like Tanh and Sigmoid functions, produce smooth, bounded outputs useful for classification but can weaken learning in deep networks, with the curve flattening out at large input values. Lastly, Linear functions work well for regression tasks but limit the network’s ability to model complex, non-linear patterns. Different layers can use different activation functions, and the choice depends on the specific application requirements and network architecture.<\/p>\n

After passing through several hidden layers, the data finally arrives at the output layer. In the output nodes, the value of the activation corresponds to some parameter of interest. As an example, suppose that time series data collected from an oscilloscope is fed into the input, and the goal of the network is to classify the signal as a sine wave, square wave, sawtooth wave, or DC signal. In the output layer, each node would correspond to one of these options, with the highest value activation representing the network\u2019s best guess as to the form of the signal. If one activation is close to 1 and the others are close to 0, the confidence of the network\u2019s guess is high. If the activations are similarly valued, this indicates low confidence in the answer.<\/span><\/p>\n

How do neural networks work?<\/span><\/h2>\n

Without adjusting the weights and biases of the hidden layers, a neural network ends up being nothing more than a complex random number generator. To improve the accuracy of the model, the user must provide training data, where both the input dataset and expected answer are known. The model can then calculate its own answer to the training set, which can be compared to the true value. The calculated difference, known as the cost function, gives a quantitative assessment of the model\u2019s performance.<\/span><\/p>\n

The model aims to reduce this cost function through training. For instance, Mean Squared Error (MSE) finds the average of the squared differences between the predicted and real values. An optimizer is the method used to efficiently minimize this cost function. Machine learning packages for Python, such as Keras, help guide the choice of the right cost function and optimizer.<\/p>\n

After calculating the cost function for a given dataset, the weights and biases of the hidden layers can then be adjusted through various calculus operations, with the goal of minimizing the cost function. This is similar to the concept of gradient descent in vector calculus and can be explored further in literature [1]. This process, called backpropagation<\/span>,<\/span><\/i> allows information obtained via the cost function to work backward through the layers, resulting in the model learning, or adjusting itself, without human input.<\/span><\/p>\n

Training data is often run through the neural network multiple times. Each instance of this data being provided to the model is known as an epoch. Typically, some training data will be reserved for validation. In validation, a trained network is used to infer outputs from this reserved data set and its predictions compared to the known correct output. This gives a more accurate picture of the model\u2019s performance than the cost function value alone, as it indicates how well the model can generalize results to new and novel inputs.<\/span><\/p>\n

What are the different types of neural networks?<\/span><\/h2>\n

Neural networks, while operating on similar principles, can take various forms depending on the application. A few common neural network examples include:<\/span><\/p>\n