Measuring complexity
Some probability background
Basics of information theory
Some entropy theory
The Gibbs inequality
A simple physical example (gases)
Shannon’s communication theory
Application to Biology (analyzing genomes)
Some other measures
Some additional material
Examples using Bayes’ Theorem  Analog channels
A Maximum Entropy Principle
Application: Economics I (a Boltzmann Economy)
Application: Economics II (a power law)
Application to Physics (lasers)

Turing's deep 1937 paper made it clear that G¨odel's astonishing earlier results on arithmetic undecidability related in a very natural way to a class of computing automata, nonexistent at the time of Turing's paper, but destined to appear only a few years later, subsequently to proliferate as the ubiquitous stored-program computer of today. The appearance of computers, and the involvement of a large scientic community in

Is it possible to communicate reliably from one point
to another if we only have a noisy communication
channel? How can the information content of a random
variable be measured? This course will discuss the
remarkable theorems of Claude Shannon, starting from
the source coding theorem, which motivates the entropy
as the measure of information, and culminating in the
noisy channel coding theorem. Along the way we will
study simple examples of codes for data compression
and error correction.

THE recent development of various methods of modulation such as PCM and PPM which exchange
bandwidth for signal-to-noise ratio has intensified the interest in a general theory of communication. A basis for such a theory is contained in the important papers of Nyquist1 and Hartley2 on this subject. In the present paper we will extend the theory to include a number of new factors, in particular the effect of noise in the channel, and the savings possible due to the statistical structure of the original message and due to the nature of the final destination of the information.

Information may be defined as
the characteristics of the
output of a process, these being
informative about the process and the input.
This discipline independent definition
may be applied to all domains, from physics
to epistemology.
Hierarchies of processes, linked
together, provide a communication channel
between each of the
corresponding functions and layers
in the hierarchies.
Models of communication (Shannon),
perception, observation, belief, and
knowledge are suggested that are
consistent with this conceptual
framework of information