Search This Blog

Tuesday, 4 November 2014

Information theory

Information theory is the study of the fundamental charac-teristics of information and its transmission and reception. As a discipline, information theory took its impetus from the ideas of Claude Shannon (see Shannon, Claude).

In his seminal paper “A Mathematical Theory of Com-munication” published in the Bell System Technical Journal in 1948, Shannon analyzed the redundancy inherent in any form of communication other than a series of purely ran-dom numbers. Because of this redundancy, the amount of information (expressed in binary bits) needed to convey a message will be less than the number in the original mes-sage. It is because of redundancy that data compression algorithms can be applied to text, graphics, and other types of files to be stored on disk or transmitted over a network (see data compression).

Shannon also analyzed the unpredictability or uncer-tainty of information as it is received—that is, the number of possibilities for the next bit or character. This is related to the number of possible symbols, but since all symbols are usually not equally likely, it is actually a sum of probabilities. Shan-non used the physics term entropy to refer to this measure. It is important because it makes it possible to analyze the prob-ability of error (caused by such things as “line noise”) in a communications circuit. Shannon’s basic formula is:

C = Blog2(1 + P / N)

where the channel capacity C is in bits per second, B is the bandwidth, P the signal power, and N the Gaussian noise power. Shannon found that if as long as the actual data trans-mission rate is less than the channel capacity C, an error-correcting code can be devised to ensure that any desired accuracy rate is achieved (see error correction). A related formula can also be used to find the lowest transmission power needed given a specified amount of noise.

The influence of Shannon and his disciples on comput-ing has been pervasive. Information theory provides the fundamental understanding needed for applications in data compression, signal analysis, data communication, and cryptography—as well as problems in other fields such as the analysis of genetic mutation or variation.

No comments:

Post a Comment