Probabilidades, entropías y SCT - UdL...

ProbabilidadesEntropías

Shannon: SCT

Probabilidades, entropías y SCTCTI: Lección 1, Primer teorema de Shannon (SCT)

Ramiro Moreno Chiral

Dpt. Matemàtica (UdL)

6 de febrero 2010

Ramiro Moreno (Matemàtica, UdL) Probabilidades, entropías y SCT 6 de febrero 2010 1 / 31


Shannon: SCT

Índice

1 Teoría de la Probabilidad

2 Información y entropía asociadas a variables aleatorias

3 Primer teorema de Shannon: SCT



Shannon: SCT

NotacionesDefinicionesTeoremas

Índice

1 Teoría de la ProbabilidadNotacionesDefinicionesTeoremas





Shannon: SCT


Probabilidad y variables aleatorias

Variable aleatoria (v.a.) X , sobre X = {x1, x2, . . . , xr}.Suceso: X = xi , se realiza el experimento asociado a lav.a. X y el resultado es xi .Probabilidades asociadas a X .

PX (X = xi) = pi = p(x),

tales que

r∑

i=1pi =

∑x∈X

p(x) = 1, con

0 ≤ pi ,p(x) ≤ 1, xi , x ∈ X .



Shannon: SCT



Variable aleatoria (v.a.) X , sobre X = {x1, x2, . . . , xr}.

Suceso: X = xi , se realiza el experimento asociado a lav.a. X y el resultado es xi .Probabilidades asociadas a X .

PX (X = xi) = pi = p(x),

tales que

r∑

i=1pi =

∑x∈X

p(x) = 1, con

0 ≤ pi ,p(x) ≤ 1, xi , x ∈ X .



Shannon: SCT



Variable aleatoria (v.a.) X , sobre X = {x1, x2, . . . , xr}.Suceso: X = xi , se realiza el experimento asociado a lav.a. X y el resultado es xi .

Probabilidades asociadas a X .

PX (X = xi) = pi = p(x),

tales que

r∑

i=1pi =

∑x∈X

p(x) = 1, con

0 ≤ pi ,p(x) ≤ 1, xi , x ∈ X .



Shannon: SCT



Variable aleatoria (v.a.) X , sobre X = {x1, x2, . . . , xr}.Suceso: X = xi , se realiza el experimento asociado a lav.a. X y el resultado es xi .Probabilidades asociadas a X .

PX (X = xi) = pi = p(x),

tales que

r∑

i=1pi =

∑x∈X

p(x) = 1, con

0 ≤ pi ,p(x) ≤ 1, xi , x ∈ X .



Shannon: SCT


Media. Distribuciones conjuntas y marginales

Esperanza matemática o media,

EX X =r∑

i=1xiPX (X = xi) =

r∑i=1

xipi =∑

x∈Xxp(x);

EX f (X ) =∑

x∈Xf (x)p(x).

Distribuciones conjuntas y marginales,

Conjunta: PXY (X = xi ,Y = yj) = pij = p(x , y),

Marginales:

PX (X = x) =

∑y∈Y

p(x , y) = p(x),

PX (Y = y) =∑

x∈Xp(x , y) = p(y).



Shannon: SCT


Del producto y de la suma

Teorema (Regla del producto)

p(x , y) = p(y |x)p(x) = p(x |y)p(y).

Si X e Y son independientes es p(x , y) = p(x)p(y).

Teorema (Regla de la suma)

p(x) =∑

y∈Yp(x , y) =

∑y∈Y

p(x |y)p(y),

p(y) =∑

x∈Xp(x , y) =

∑x∈X

p(y |x)p(x).



Shannon: SCT

DefinicionesResultadosInformación mutua

Índice


2 Información y entropía asociadas a variables aleatoriasDefinicionesResultadosEntropía relativa e Información mutua




Shannon: SCT


Información

Información asociada a una v.a. X :

I(X = x) = log1

p(x)= − log p(x).

A priori: incertidumbre, desconocimiento.A posteriori: información, conocimiento.

Propiedades:1 Decreciente con la probabilidad.2 Aditiva: I(X = x ,Y = y) = log 1

p(x)p(y) =

− log p(x)− log p(y) = I(X = x) + I(Y = y).3 No–negativa: I(X = x) = − log p(x) ≥ 0, ya que

0 ≤ p(x) ≤ 1.



Shannon: SCT


Información


I(X = x) = log1

p(x)= − log p(x).


Propiedades:

1 Decreciente con la probabilidad.2 Aditiva: I(X = x ,Y = y) = log 1

p(x)p(y) =


0 ≤ p(x) ≤ 1.



Shannon: SCT


Información


I(X = x) = log1

p(x)= − log p(x).


Propiedades:1 Decreciente con la probabilidad.

2 Aditiva: I(X = x ,Y = y) = log 1p(x)p(y) =


0 ≤ p(x) ≤ 1.



Shannon: SCT


Información


I(X = x) = log1

p(x)= − log p(x).



p(x)p(y) =

− log p(x)− log p(y) = I(X = x) + I(Y = y).

3 No–negativa: I(X = x) = − log p(x) ≥ 0, ya que0 ≤ p(x) ≤ 1.



Shannon: SCT


Información


I(X = x) = log1

p(x)= − log p(x).



p(x)p(y) =


0 ≤ p(x) ≤ 1.



Shannon: SCT


Entropía de una v.a.

Entropía asociada a una v.a. X : Es el promedio de lainformación asociada a X .

H(X ) = EX I(X = x) = −EX log p(x) = −∑x∈X

p(x) log p(x),

donde si p(x) = 0 se define p(x) log p(x) = 0. Con otranotación,

H(X ) = H(p1, . . . ,pn).



Shannon: SCT



Entropía asociada a una v.a. X :

Es el promedio de lainformación asociada a X .


p(x) log p(x),


H(X ) = H(p1, . . . ,pn).



Shannon: SCT





p(x) log p(x),


H(X ) = H(p1, . . . ,pn).



Shannon: SCT





p(x) log p(x),

donde si p(x) = 0 se define p(x) log p(x) = 0

. Con otranotación,

H(X ) = H(p1, . . . ,pn).



Shannon: SCT





p(x) log p(x),


H(X ) = H(p1, . . . ,pn).



Shannon: SCT


Entropías conjunta y condicionadas

Entropía conjunta de dos v.a.’s X e Y ,

H(X ,Y ) = EXY log1

p(x , y)= −

∑x∈X

∑y∈Y

p(x , y) log p(x , y).

Entropía condicionada de la v.a X cuando Y = y ,

H(X |Y = y) = EX |Y=y log 1P(X |Y=y)

= −∑

x∈Xp(x |y) log p(x |y).



Shannon: SCT


Propiedades de la entropía

No–negatividad: H(X ) ≥ 0, siendo H(X ) = 0 sii p(x) = 1,para algún x ∈ X .Cambio de base del logaritmo: Hb(X ) = logb a Ha(X ).El condicionamiento reduce la entropía: H(X |Y ) ≤ H(X ),siendo H(X |Y ) = H(X ) sii X e Y son independientes.Cota superior: H(X ) ≤ log |X |, llegando a la igualdad si Xes una v.a. uniforme.Regla de la cadena:H(X ,Y ) = H(X ) + H(Y |X ) = H(Y ) + H(X |Y ).



Shannon: SCT



No–negatividad: H(X ) ≥ 0, siendo H(X ) = 0 sii p(x) = 1,para algún x ∈ X .

Cambio de base del logaritmo: Hb(X ) = logb a Ha(X ).El condicionamiento reduce la entropía: H(X |Y ) ≤ H(X ),siendo H(X |Y ) = H(X ) sii X e Y son independientes.Cota superior: H(X ) ≤ log |X |, llegando a la igualdad si Xes una v.a. uniforme.Regla de la cadena:H(X ,Y ) = H(X ) + H(Y |X ) = H(Y ) + H(X |Y ).



Shannon: SCT



No–negatividad: H(X ) ≥ 0, siendo H(X ) = 0 sii p(x) = 1,para algún x ∈ X .Cambio de base del logaritmo: Hb(X ) = logb a Ha(X ).

El condicionamiento reduce la entropía: H(X |Y ) ≤ H(X ),siendo H(X |Y ) = H(X ) sii X e Y son independientes.Cota superior: H(X ) ≤ log |X |, llegando a la igualdad si Xes una v.a. uniforme.Regla de la cadena:H(X ,Y ) = H(X ) + H(Y |X ) = H(Y ) + H(X |Y ).



Shannon: SCT



No–negatividad: H(X ) ≥ 0, siendo H(X ) = 0 sii p(x) = 1,para algún x ∈ X .Cambio de base del logaritmo: Hb(X ) = logb a Ha(X ).El condicionamiento reduce la entropía: H(X |Y ) ≤ H(X ),siendo H(X |Y ) = H(X ) sii X e Y son independientes.

Cota superior: H(X ) ≤ log |X |, llegando a la igualdad si Xes una v.a. uniforme.Regla de la cadena:H(X ,Y ) = H(X ) + H(Y |X ) = H(Y ) + H(X |Y ).



Shannon: SCT



No–negatividad: H(X ) ≥ 0, siendo H(X ) = 0 sii p(x) = 1,para algún x ∈ X .Cambio de base del logaritmo: Hb(X ) = logb a Ha(X ).El condicionamiento reduce la entropía: H(X |Y ) ≤ H(X ),siendo H(X |Y ) = H(X ) sii X e Y son independientes.Cota superior: H(X ) ≤ log |X |, llegando a la igualdad si Xes una v.a. uniforme.

Regla de la cadena:H(X ,Y ) = H(X ) + H(Y |X ) = H(Y ) + H(X |Y ).



Shannon: SCT



No–negatividad: H(X ) ≥ 0, siendo H(X ) = 0 sii p(x) = 1,para algún x ∈ X .Cambio de base del logaritmo: Hb(X ) = logb a Ha(X ).El condicionamiento reduce la entropía: H(X |Y ) ≤ H(X ),siendo H(X |Y ) = H(X ) sii X e Y son independientes.Cota superior: H(X ) ≤ log |X |, llegando a la igualdad si Xes una v.a. uniforme.Regla de la cadena:H(X ,Y ) = H(X ) + H(Y |X ) = H(Y ) + H(X |Y ).



Shannon: SCT


Definiciones

Entropía relativa o distancia de Kullback–Leibler,

D(p‖q) = Ep logp(X )

q(X )=∑x∈X

p(x) logp(x)

q(x).

Información mutua,

I(X ; Y ) = D (p(x , y)‖p(x)p(y))

= EXY logp(x , y)

p(x)p(y)

=∑x∈X

∑y∈Y

p(x , y) logp(x , y)

p(x)p(y)



Shannon: SCT


Propiedades

Simetría:I(X ; Y ) = H(X )− H(X |Y ) = H(Y )− H(Y |X ) = I(Y ; X ).Relación con las entropía conjunta:I(X ; Y ) = H(X ) + H(Y )− H(X ,Y ).No–negatividad o Lema de Gibbs: D(p‖q) ≥ 0, llegando ala igualdad sii p(x) = q(x), ∀x .No–negatividad de la información mutua:I(X ; Y ) = D(p(x , y)‖p(x)p(y)) ≥ 0, siendo nula siip(x , y) = p(x)p(y), i.e., cuando X e Y son independientes.



Shannon: SCT


Propiedades

Simetría:I(X ; Y ) = H(X )− H(X |Y ) = H(Y )− H(Y |X ) = I(Y ; X ).

Relación con las entropía conjunta:I(X ; Y ) = H(X ) + H(Y )− H(X ,Y ).No–negatividad o Lema de Gibbs: D(p‖q) ≥ 0, llegando ala igualdad sii p(x) = q(x), ∀x .No–negatividad de la información mutua:I(X ; Y ) = D(p(x , y)‖p(x)p(y)) ≥ 0, siendo nula siip(x , y) = p(x)p(y), i.e., cuando X e Y son independientes.



Shannon: SCT


Propiedades

Simetría:I(X ; Y ) = H(X )− H(X |Y ) = H(Y )− H(Y |X ) = I(Y ; X ).Relación con las entropía conjunta:I(X ; Y ) = H(X ) + H(Y )− H(X ,Y ).

No–negatividad o Lema de Gibbs: D(p‖q) ≥ 0, llegando ala igualdad sii p(x) = q(x), ∀x .No–negatividad de la información mutua:I(X ; Y ) = D(p(x , y)‖p(x)p(y)) ≥ 0, siendo nula siip(x , y) = p(x)p(y), i.e., cuando X e Y son independientes.



Shannon: SCT


Propiedades

Simetría:I(X ; Y ) = H(X )− H(X |Y ) = H(Y )− H(Y |X ) = I(Y ; X ).Relación con las entropía conjunta:I(X ; Y ) = H(X ) + H(Y )− H(X ,Y ).No–negatividad o Lema de Gibbs: D(p‖q) ≥ 0, llegando ala igualdad sii p(x) = q(x), ∀x .

No–negatividad de la información mutua:I(X ; Y ) = D(p(x , y)‖p(x)p(y)) ≥ 0, siendo nula siip(x , y) = p(x)p(y), i.e., cuando X e Y son independientes.



Shannon: SCT


Propiedades

Simetría:I(X ; Y ) = H(X )− H(X |Y ) = H(Y )− H(Y |X ) = I(Y ; X ).Relación con las entropía conjunta:I(X ; Y ) = H(X ) + H(Y )− H(X ,Y ).No–negatividad o Lema de Gibbs: D(p‖q) ≥ 0, llegando ala igualdad sii p(x) = q(x), ∀x .No–negatividad de la información mutua:I(X ; Y ) = D(p(x , y)‖p(x)p(y)) ≥ 0, siendo nula siip(x , y) = p(x)p(y), i.e., cuando X e Y son independientes.



Shannon: SCT


Propiedades: visualización

La figura permite recordar las relaciones aditivas más usadas:

H(X,Y)

H(X|Y) H(Y|X)

H(X) H(Y)

I(X;Y)

Figura: Entropías e información mutua



Shannon: SCT

AEPCSTSCT

Índice



3 Primer teorema de Shannon: SCTPrincipio de Equipartición Asintótica, AEPConjunto de secuencias típicas, CSTPrimer teorema de Shannon, SCT



Shannon: SCT

AEPCSTSCT

V.a.’s independientes e idénticamente distribuidas, iid

Convergencia en probabilidad. Sea una sucesión de v.a.’s(Xn)n∈N, escribiremos Xn

P−→ X si ∀ε > 0

limn↑∞

P(|Xn − X | > ε) = 0.

V.a.’s independientes e idénticamente distribuidas, iid.(Xn)n∈N son v.a.’s iid ∼ X cuando

P(Xn = x) = P(X = x),∀n y ∀x ,

es decir, todas las Xn siguen el “modelo” X .



Shannon: SCT

AEPCSTSCT

V.a.’s independientes e idénticamente distribuidas, iid

Convergencia en probabilidad. Sea una sucesión de v.a.’s(Xn)n∈N, escribiremos Xn

P−→ X si ∀ε > 0

limn↑∞

P(|Xn − X | > ε) = 0.

V.a.’s independientes e idénticamente distribuidas, iid.(Xn)n∈N son v.a.’s iid ∼ X cuando

P(Xn = x) = P(X = x), ∀n y ∀x ,

es decir, todas las Xn siguen el “modelo” X .



Shannon: SCT

AEPCSTSCT

Ley Débil de los Grandes Números, LDGN

Ley Débil de los Grandes Números, LDGN. SeanXn iid ∼ X , se cumple

1n

n∑i=1

xiP−→ EX X

La LDGN también se puede escribir,

∀ε, δ > 0,∃ n(ε, δ) ∈ N, tal que si n > n(ε, δ) es

P(∣∣∣∣1n n∑

i=1xi − EX X

∣∣∣∣ > δ

)< ε,

P(∣∣∣∣1n n∑

i=1xi − EX X

∣∣∣∣ ≤ δ) > 1− ε



Shannon: SCT

AEPCSTSCT

Enunciado del AEP

Teorema (AEP)

Si (Xn)n∈N es una sucesión de v.a.’s iid ∼ X, se cumple

−1n

log PX n(X1, . . . ,Xn)P−→ H(X ).

O también con otra notación,

∀ε > 0, ∃ n(ε) ∈ N, tal que si n > n(ε) esP(∣∣−1

n log PX n(X1, . . . ,Xn)− H(X )∣∣ ≤ ε) > 1− ε.



Shannon: SCT

AEPCSTSCT

Conjunto de secuencias típicas, CST

Definición (CST)

Consideramos (Xn)n∈N una sucesión de v.a.’s iid ∼ X , definidasen X , y un ε > 0, se define el CST, A(n)

ε , como el conjunto desecuencias x (n) = (x1, x2, . . . , xn) ∈ X n que verifican el AEPpara esos ε y n.

A(n)ε =

{x (n) :

∣∣− 1n log p(x1, . . . , xn)− H(X )

∣∣ ≤ ε}={

x (n) : H(X )− ε ≤ − 1n log p(x1, . . . , xn) ≤ H(X ) + ε

}={

x (n) : 2−n(H(X)+ε) ≤ p(x1, . . . , xn) ≤ 2−n(H(X)−ε)}



Shannon: SCT

AEPCSTSCT


Definición (CST)



A(n)ε

={

x (n) :∣∣− 1

n log p(x1, . . . , xn)− H(X )∣∣ ≤ ε}

={


}={

x (n) : 2−n(H(X)+ε) ≤ p(x1, . . . , xn) ≤ 2−n(H(X)−ε)}



Shannon: SCT

AEPCSTSCT


Definición (CST)



A(n)ε =

{x (n) :

∣∣− 1n log p(x1, . . . , xn)− H(X )

∣∣ ≤ ε}

={


}={

x (n) : 2−n(H(X)+ε) ≤ p(x1, . . . , xn) ≤ 2−n(H(X)−ε)}



Shannon: SCT

AEPCSTSCT


Definición (CST)



A(n)ε =

{x (n) :

∣∣− 1n log p(x1, . . . , xn)− H(X )

∣∣ ≤ ε}={


}

={

x (n) : 2−n(H(X)+ε) ≤ p(x1, . . . , xn) ≤ 2−n(H(X)−ε)}



Shannon: SCT

AEPCSTSCT


Definición (CST)



A(n)ε =

{x (n) :

∣∣− 1n log p(x1, . . . , xn)− H(X )

∣∣ ≤ ε}={


}={

x (n) : 2−n(H(X)+ε) ≤ p(x1, . . . , xn) ≤ 2−n(H(X)−ε)}



Shannon: SCT

AEPCSTSCT

Propiedades del CST

La propia definición: si x(n) ∈ A(n)ε , entonces

2−n(H(X)+ε) ≤ p(x(n)) ≤ 2−n(H(X)−ε).

P(A(n)ε ) > 1− ε.

(1− ε)2n(H(X)−ε) ≤˛̨̨A(n)

ε

˛̨̨≤ 2n(H(X)+ε).

��

��

��

� ��

��

��

� ��

Figura: Conjunto de secuencias típicas y AEP



Shannon: SCT

AEPCSTSCT

CST: visualizaciónSecuencias muy probables, típicas y poco probables

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.3369; 0.3456; 0.3175. Cards.: 6; 10; 16.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.2333; 0.5875; 0.1792. Cards.: 7; 35; 22.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.1586; 0.5516; 0.2898. Cards.: 8; 56; 64.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.1064; 0.7200; 0.1736. Cards.: 9; 154; 93.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.2318; 0.6689; 0.0993. Cards.: 46; 336; 130.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.1673; 0.6665; 0.1662. Cards.: 56; 582; 386.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.1189; 0.7817; 0.0994. Cards.: 67; 1419; 562.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0834; 0.7583; 0.1582. Cards.: 79; 2431; 1586.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0579; 0.8444; 0.0977. Cards.: 92; 5720; 2380.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.1243; 0.7255; 0.1501. Cards.: 470; 9438; 6476.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0905; 0.8145; 0.0950. Cards.: 576; 22243; 9949.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0651; 0.8765; 0.0583. Cards.: 697; 49946; 14893.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0464; 0.8617; 0.0919. Cards.: 834; 89012; 41226.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0942; 0.8482; 0.0576. Cards.: 4048; 195092; 63004.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0696; 0.8419; 0.0885. Cards.: 5036; 349486; 169766.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0509; 0.8925; 0.0565. Cards.: 6196; 778430; 263950.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0369; 0.8781; 0.0849. Cards.: 7547; 1393745; 695860.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0722; 0.8727; 0.0551. Cards.: 35443; 3061071; 1097790.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0540; 0.9111; 0.0349. Cards.: 44552; 6645896; 1698160.



Shannon: SCT

AEPCSTSCT


20 40 60 80 100Xn

0.01

0.02

0.03

0.04

0.05

0.06

0.07

pHx1 , ..., xn L 5= n

20 40 60 80 100Xn

0.01

0.02

0.03

0.04

pHx1 , ..., xn L 6= n

20 40 60 80 100Xn

0.005

0.01

0.015

0.02

0.025

pHx1 , ..., xn L 7= n

20 40 60 80 100Xn

0.0025

0.005

0.0075

0.01

0.0125

0.015

pHx1 , ..., xn L 8= n

20 40 60 80 100Xn

0.002

0.004

0.006

0.008

0.01

pHx1 , ..., xn L 9= n

20 40 60 80 100Xn

0.001

0.002

0.003

0.004

0.005

0.006

pHx1 , ..., xn L 10= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

0.0025

0.003

0.0035

pHx1 , ..., xn L 11= n

20 40 60 80 100Xn

0.0005

0.001

0.0015

0.002

pHx1 , ..., xn L 12= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

pHx1 , ..., xn L 13= n

20 40 60 80 100Xn

0.0002

0.0004

0.0006

0.0008

pHx1 , ..., xn L 14= n

20 40 60 80 100Xn

0.0001

0.0002

0.0003

0.0004

pHx1 , ..., xn L 15= n

20 40 60 80 100Xn

0.00005

0.0001

0.00015

0.0002

0.00025

pHx1 , ..., xn L 16= n

20 40 60 80 100Xn

0.000025

0.00005

0.000075

0.0001

0.000125

0.00015

pHx1 , ..., xn L 17= n

20 40 60 80 100Xn

0.00001

0.00002

0.00003

0.00004

0.00005

pHx1 , ..., xn L 18= n

20 40 60 80 100Xn

5´10-6

0.00001

0.000015

0.00002

0.000025

0.00003

pHx1 , ..., xn L 19= n

20 40 60 80 100Xn

2.5´10-6

5´10-6

7.5´10-6

0.00001

0.0000125

0.000015

0.0000175

pHx1 , ..., xn L 20= n

20 40 60 80 100Xn

2´10-6

4´10-6

6´10-6

8´10-6

0.00001

pHx1 , ..., xn L 21= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

4´10-6

5´10-6

6´10-6

pHx1 , ..., xn L 22= n

20 40 60 80 100Xn

1´10-6

2´10-6

3´10-6

pHx1 , ..., xn L 23= n

20 40 60 80 100Xn

2.5´10-7

5´10-7

7.5´10-7

1´10-6

1.25´10-6

1.5´10-6

pHx1 , ..., xn L 24= n

Probs: 0.0400; 0.9065; 0.0535. Cards.: 55455; 12181375; 4540386.



Shannon: SCT

AEPCSTSCT

Probabilidad del CST

10 20 30 40 50

0.2

0.4

0.6

0.8

1

10 20 30 40

0.2

0.4

0.6

0.8

1

Probabilidades del conjunto de secuencias típicas, CSTMuy probables+Poco probables=Secuencias no–típicas



Shannon: SCT

AEPCSTSCT

Compresión a partir del AEP: Algoritmo

1 Ordenamos todas las secuencias de X n.2 En ese conjunto ordenado definimos un índice.3 Como |A(n)

ε | ≤ 2n(H(x)+ε), sólo necesitamosdn(H(x) + ε)e ≤ n(H(x) + ε) + 1 bits para codificar el CST.

4 En el conjunto A(n)ε = X n \ A(n)

ε , podemos “gastar” bits, yaque P(A

(n)ε ) < ε. Para codificarlo usamos tantos bits como

si quisiéramos codificar todo X n, es decir,dn log |X |e ≤ n log |X |+ 1 bits.

5 Para diferenciar ambos modos de codificaciónanteponemos un bit diferenciador.



Shannon: SCT

AEPCSTSCT


1 Ordenamos todas las secuencias de X n.

2 En ese conjunto ordenado definimos un índice.3 Como |A(n)









Shannon: SCT

AEPCSTSCT


1 Ordenamos todas las secuencias de X n.2 En ese conjunto ordenado definimos un índice.

3 Como |A(n)ε | ≤ 2n(H(x)+ε), sólo necesitamos

dn(H(x) + ε)e ≤ n(H(x) + ε) + 1 bits para codificar el CST.








Shannon: SCT

AEPCSTSCT


1 Ordenamos todas las secuencias de X n.2 En ese conjunto ordenado definimos un índice.3 Como |A(n)









Shannon: SCT

AEPCSTSCT

SCT: EnunciadoPrimer teorema de Shannon

Teorema (SCT)



Shannon: SCT

AEPCSTSCT


Teorema (SCT)

Sea (Xn)n∈N una secuencia de v.a.’s iid ∼ X sobre un alfabetofinito X



Shannon: SCT

AEPCSTSCT


Teorema (SCT)

Sea (Xn)n∈N una secuencia de v.a.’s iid ∼ X sobre un alfabetofinito X , y ε un número real > 0



Shannon: SCT

AEPCSTSCT


Teorema (SCT)

Sea (Xn)n∈N una secuencia de v.a.’s iid ∼ X sobre un alfabetofinito X , y ε un número real > 0. Entonces, para un nsuficientemente grande, existe un código que hacecorresponder uno a uno cada secuencia xn ∈ X n con unapalabra–código



Shannon: SCT

AEPCSTSCT


Teorema (SCT)

Sea (Xn)n∈N una secuencia de v.a.’s iid ∼ X sobre un alfabetofinito X , y ε un número real > 0. Entonces, para un nsuficientemente grande, existe un código que hacecorresponder uno a uno cada secuencia xn ∈ X n con unapalabra–código, tal que si `(X n) es la longitud de laspalabras–código, será



Shannon: SCT

AEPCSTSCT


Teorema (SCT)


EX

(1n`(X n)

)≤ H(X ) + ε.



Shannon: SCT

AEPCSTSCT


Teorema (SCT)


EX

(1n`(X n)

)≤ H(X ) + ε.

NOTA.



Shannon: SCT

AEPCSTSCT


Teorema (SCT)


EX

(1n`(X n)

)≤ H(X ) + ε.

NOTA. Nótese que 1n`(X

n) es la longitud de la palabra–códigopor carácter de xn.



Shannon: SCT

AEPCSTSCT

SCT: Demostración

Sea n suficientemente grande para que P(A(n)ε ) > 1− ε

EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=

∑xn∈A(n)

ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn)

+∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤

∑xn∈A(n)

ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2)

+∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

=

(n(H + ε) + 2) P(A(n)ε ) + (n log |X |+ 2) P(A

(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε )

+ (n log |X |+ 2) P(A(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε )︸︷︷︸≤1

+ (n log |X |+ 2) P(A(n)

ε )

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε )︸︷︷︸≤1

+ (n log |X |+ 2) P(A(n)

ε )︸︷︷︸<ε

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε )︸︷︷︸≤1

+ (n log |X |+ 2) P(A(n)

ε )︸︷︷︸<ε

≤

nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε )︸︷︷︸≤1

+ (n log |X |+ 2) P(A(n)

ε )︸︷︷︸<ε

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)

= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Demostración


EX (`(X n)) =∑

xn∈X n

p(xn)`(xn)

=∑

xn∈A(n)ε

p(xn)`(xn) +∑

xn∈A(n)ε

p(xn)`(xn)

≤∑

xn∈A(n)ε

p(xn) (n(H + ε) + 2) +∑

xn∈A(n)ε

p(xn) (n log |X |+ 2)

= (n(H + ε) + 2) P(A(n)ε )︸︷︷︸≤1

+ (n log |X |+ 2) P(A(n)

ε )︸︷︷︸<ε

≤ nH + n(ε+ ε log |X |+ 2(1 + ε)

n

)︸︷︷︸

ε′

= n(H + ε′)



Shannon: SCT

AEPCSTSCT

SCT: Ejemplo (I)

Tenemos una fuente con vas binarias iid ∼ X sobreX = {0,1}, con p = P(X = 1) = 0′2, luegoH(X ) = H(0′2,0′8) = h(0′2) = 0′7219, es la entropía de lafuente de bits.El CST A(n)

ε se alcanza para ε = 0′05 en n = 960, ya queP(A(n)

ε ) = 0′952063 > 1− ε = 0′95.Usaremos, según el algoritmo de la demostración,

dn(H + ε)e+ 1 = d960(0′7219 + 0′05)e+ 1 = 743 bits

para codificar cada secuencia del CST.Y dn log |X |e+ 1 = 961 bits para codificar las secuenciasno típicas.



Shannon: SCT

AEPCSTSCT

SCT: Ejemplo (I)

Tenemos una fuente con vas binarias iid ∼ X sobreX = {0,1}, con p = P(X = 1) = 0′2, luegoH(X ) = H(0′2,0′8) = h(0′2) = 0′7219, es la entropía de lafuente de bits.

El CST A(n)ε se alcanza para ε = 0′05 en n = 960, ya que

P(A(n)ε ) = 0′952063 > 1− ε = 0′95.

Usaremos, según el algoritmo de la demostración,

dn(H + ε)e+ 1 = d960(0′7219 + 0′05)e+ 1 = 743 bits




Shannon: SCT

AEPCSTSCT

SCT: Ejemplo (I)



ε ) = 0′952063 > 1− ε = 0′95.

Usaremos, según el algoritmo de la demostración,

dn(H + ε)e+ 1 = d960(0′7219 + 0′05)e+ 1 = 743 bits




Shannon: SCT

AEPCSTSCT

SCT: Ejemplo (I)




dn(H + ε)e+ 1 = d960(0′7219 + 0′05)e+ 1 = 743 bits

para codificar cada secuencia del CST.

Y dn log |X |e+ 1 = 961 bits para codificar las secuenciasno típicas.



Shannon: SCT

AEPCSTSCT

SCT: Ejemplo (I)




dn(H + ε)e+ 1 = d960(0′7219 + 0′05)e+ 1 = 743 bits




Shannon: SCT

AEPCSTSCT

SCT: Ejemplo (II)

La longitud media de las palabras–código será

`(X n) = P(A(n)ε )(dn(H + ε)e+ 1) + P(A

(n)ε )(dn log |X |e+ 1)

= 0′952063 · 743 + (1− 0′952063) · 961= 753′45

Lo que supone

1n`(X n) =

753′45960

= 0′784844

bits–código por cada bit–fuente, i.e., una ratio decompresión del 78 %.



Shannon: SCT

AEPCSTSCT

SCT: Unas notas

Observad que en este ejemplo no se cumple la acotacióndel SCT para ε = 0′05:`(X n) = 753′45 6≤ n(H + ε) = 741′051, pero sí para el ε′ dela demostración, ε′ = ε+ ε log |X |+ 2(1+ε)

n = 0′1021875,pues entonces es n(H + ε′) = 791′124.Si la entropía de la fuente binaria está próxima al máximode 1 bit, el teorema se cumple, pero el algoritmo puedeque no comprima. Por ejemplo, para p = P(X = 1) = 0,4 yε = 0′1 se alcanza el AEP para n = 23, pero la ratio decompresión es

1n`(X n) =

25′822223

= 1′12271,

es decir, ¡del 122 %!.



Shannon: SCT

AEPCSTSCT

SCT: Unas notas


n = 0′1021875,pues entonces es n(H + ε′) = 791′124.

Si la entropía de la fuente binaria está próxima al máximode 1 bit, el teorema se cumple, pero el algoritmo puedeque no comprima. Por ejemplo, para p = P(X = 1) = 0,4 yε = 0′1 se alcanza el AEP para n = 23, pero la ratio decompresión es

1n`(X n) =

25′822223

= 1′12271,




Shannon: SCT

AEPCSTSCT

SCT: Unas notas


n = 0′1021875,pues entonces es n(H + ε′) = 791′124.Si la entropía de la fuente binaria está próxima al máximode 1 bit, el teorema se cumple, pero el algoritmo puedeque no comprima. Por ejemplo, para p = P(X = 1) = 0,4 yε = 0′1 se alcanza el AEP para n = 23, pero la ratio decompresión es

1n`(X n) =

25′822223

= 1′12271,



Probabilidades, entropías y SCT - UdL...

Documents

Transcript of Probabilidades, entropías y SCT - UdL...