Shannon's Entropy calculation

Question

I have a probability distribution that defines the probability of occurrence of n possible states.

I would like to calculate the value of Shannon's entropy, in bits, of the given probability distribution.

Can I use wentropy(x,'shannon') to get the value and if so where can I define the number of possible states a system has?

Solution

Since you already have the probability distribution, call it p, you can do the following formula for Shannon Entropy instead of using wentropy:

H = sum(-(p(p>0).*(log2(p(p>0)))));

This gives the entropy H in bits.

p must sum to 1.

Comments

[Composer]

Thank you. One more question, how to define the probability distribution in matlab? As a vector? [0,0.5,0.8...]

[chappjc]

Yes, values <1, summing to 1. You seem to be showing the start of a cdf, not a pdf.

[Composer]

The main concern that I have is the approach. I have a vector of probabilities of each state of the system. For example if a state has a probabilistic value of 1 it's likely to be the current state, and if it's value is 0 it's an impossible state of the system. Thank you again.

[chappjc]

Yes, that's the idea of a probability distribution, but remember that the probability of all states must sum to one.

[Composer]

You are right, but I'm hesitant to call it a pdf because the probabilities act like a constraints on the possible states of a system therefore decreasing its entropy level. For example if I have three of the states that are equally likely and one that's impossible I would write it down as [1,1,1,0]. I hope that my approach isn't too unorthodox.

[chappjc]

I follow, but you must divide by the sum to get p. The nonzero states in [1 1 1 0]/3 are all equally likely.

[Composer]

I could write [0.33,0.33,0.33,0], but i have to be able to use additive constraints, for example i would like to impose another constraint to further limit the possibilities.

[chappjc]

Then normalize p each time you introduce a constraint.

[Composer]

Lets say I have constraints imposed by one probability distribution [0.33,0.33,0.33,0], lets call this a constraint1. But when i have another constraint that I would like to impose to further limit the states(change their probability distributions), for example [0,0.5,0.5,0], how should I consolidate those two to get a final probability distribution of the system.

[Composer]

I really appreciate your help.

[chappjc]

If I understand correctly: Say you have p=[0.33 0.33 0.33 0] and c=[0 0.5 0.5 0];, then element-wise multiply and normalize: p_c = p.*c; p_c = p_c/sum(p_c);. Not the best example since the result is p_c = [0 0.5 0.5 0]; but I guess that is what you want.

[A. Donda]

@user2703038, what kinds of constraints do you need?

[Composer]

@chappjc Thank you, that makes perfect sense. And the results are logical.