Katz Smoothing

[determinedCoder]

Hi everyone,

I'm new to this forum, so please bear with me :-)

I'm working on Katz smoothing with Good Turing discounting for a project of mine. If anyone else has worked with this algorithm, then I would greatly appreciate some advice.

Katz Smoothing for bigrams;

The counts are only discounted up to some K where K= 5 (GT discounting).


Pkatz (z | y) = C (y, z) / C (y) if C(y, z) > K



DC(y,z) / C(y) if 1 <= C(y,z) <= K



Ά(y) * P(z) if C(y, z) == 0


Where C are the normal counts and DC is the discounted count and;

Ά(y) = 1 - sum of all probabilities of bigrams starting with y / 1 - P(z)


I have some questions;


1. In Ά(y), what happens if the sum of all probabilities of bigrams starting with y sum to 1. This would happen if the counts of all these bigrams are above K, and hence not discounted. This would make Ά(y) = 0 and hence katz smoothing would return 0, invalidating the whole equation.
2. I'm also assuming that in the third equation above and in Ά(y), the P(z) returns the smoothed zero probability if the count of z is 0?


I would appreciate any help

Many Thanks