A forget gate has bias 10 and no other input, so f=σ(10). Is the carry
line lossless? Compute f, 1−f, the carry-line factor after 105 steps and
the memory horizon.
Hint
Use 1−σ(b)=σ(−b), and (I.10.5) for the horizon.
Solution
f=σ(10)=0.9999546 and 1−f=σ(−10)=4.539787×10−5.
After 105 steps the factor is f100000=0.010675: about one per cent of
the signal is left.
The horizon is −1/lnf=22026.97 steps. By (I.10.5) it lies just below
1/(1−f)=1+e10=22027.47.
The carry line is not lossless. A sigmoid gate can be pushed as close to one
as the bias allows, which sets a long but finite horizon. Only the 1997
design, with no forget gate at all, had a factor of exactly one, and
I.10.X03 shows what that cost.