Are Neural Networks complex "Linear Functions" with many parameters?

Question

5. Neural Networks are complex ______________ with many parameters.
   a) Linear Functions;
   b) Nonlinear Functions;
   c) Discrete Functions;
   d) Exponential Functions

Answer: a. Explanation: Neural networks are complex linear functions with many parameters.

From: https://www.sanfoundry.com/ai-multiple-choice-questions-answers/

I am not sure NNs are just "linear" functions? We were asked to answer this question in a quiz at my university. However, I think NNs with non-linear activation functions are not linear.

Solution 1

The nice thing about linear functions is that you can combine them ahead of time, because combining linear functions gets you another linear function (matrix multiplication and all that).

So if it was just a linear function, you'd essentially have just a one-layer network. It's the non-linear transformations in between the layers that cause the layers to actually be useful, and so I'd say that these are non-linear functions.

Solution 2

The question is slightly incorrect in its portrayal. While you could say the core of Neural Networks are linear equations on matrix scale (or n-dimensional scale), it is only the non-linear activations that make it learn something relevant.

Although, you could have your activation function be activation(x) = k * x + c in which case the answer could be complex linear functions. However, a linear composition of a linear function would always yield a linear function. But, neural networks clearly learn non-linear functions to solve machine learning problems.

So, we can not really put it by saying they're linear functions.

Solution 3

Question is little ambiguous. If we use a linear activation function(Identity activation function) or alternatively, if we don't have an activation function, then no matter how many layers our neural network has, all it does is just computes a linear activation function. So we might as well have no hidden layers.

Only after adopting a non-linear activation function, neural networks learn relevant information. This is the reason why activation functions are non-linear.

So according to me, it would be apt to consider them as complex non-linear functions with many parameters.

Comments

[dedObed]

Complex does not really go well with linear.