Information Theory and its applications in theory of computation
This set of lecture notes by Venkatesan Guruswami and Mahdi Cheraghchi addresses the intersection of information theory and theoretical computer science. The core topics to be covered in the lecture note include entropy, Kraft's inequality, source coding theorem, conditional entropy, and mutual information. It also covers KL-divergence, Chernoff bounds, data processing, and Fano's inequalities. Key concepts include AEP, universal source coding using the Lempel-Ziv algorithm, and proof of its optimality. It covers discrete channels and channel capacity, the Noisy Channel Coding Theorem, and how to construct capacity-achieving codes by concatenation and by polar codes. Additional topics: Bregman's theorem, Shearer's Lemma, graph entropy, and applications to optimal set disjointness lower bounds. This text offers a wide-ranging view of how the basic principles of information theory shed light on the construction of algorithms, and the establishment of bounds-on the complexity of problems in the field of theoretical computation.
Author(s): Venkatesan Guruswami and Mahdi Cheraghchi
NA Pages 
An Introduction to Information Theory and Applications
By F. Bavaud, J. C. Chappelier, and J. Kohlas—This long note contains a good survey of information theory and its applications. It introduces the basic ideas of uncertainty and information, then also the more practical extensions such as optimal coding schemes, followed by the theories underlying versions of stationary processes and Markov chains. Other challenges, as the note addresses, pertain to noisy transmission environments in coding. Highlighted here are several advanced topics that follow, including, importantly, error-correcting codes and cryptography. The resource will give both a theoretical background and a practical overview of how to encode, transmit, and secure information effectively. It is a very important guide for those who seek a deep understanding of information theory and how it relates to real problems of communication and data processing.
Author(s): F Bavaud, J C Chappelier and J Kohlas
293 Pages
Applied Digital Information theory
This note serves as a comprehensive guide to fundamental concepts in information theory and coding. This pdf provides discrete probability theory, information theory, and coding principles. Beginning with Shannon's measure of information, then delves into the efficient coding of information, the methodology of typical sequences is introduced, emphasizing the distinction between lossy and lossless source encoding. The text also discusses coding for noisy digital channels, block coding principles and tree and trellis coding principles.
Author(s): James L.Massey
153 Pages
An introduction to information Theory and Entropy
Om Carter-Introduction to information theory and entropy: It goes in deep to do some basic concepts of information theory, focusing on the concept of entropy and its applications. It does so by first investigating the measure of complexity and the elementary theories of probability before introducing some key ideas in information theory. It ranges from basic issues, such as entropy theory and the Gibbs inequality, up to Shannon's communication theory but also to practical applications in many diversified fields. Other topics dealt with are Bayes Theorem, analog channels, the Maximum Entropy Principle, and applications to biology and physics. The Kullback-Leibler information measure will be discussed in trying to cast light upon quantification of information and its relations with different fields of science. This book should be ideal for the general reader interested in information theory and its immense areas of application..
Author(s): Tom Carter
139 Pages
Information Theory Lecture Notes
Prof. Tsachy Weissman's lecture notes are an excellent summary of the core topics in the subject of information theory. The document initiates with a basic overview of entropy and relative entropy, followed by mutual information and asymptotic equipartition property. Further, it discusses communications theory, channel capacity, and the method of types. It also covers key topics such as typicality-conditioned and joint, lossy compression, and rate-distortion theory. The notes also include joint source-channel coding, where there is quite a good grasp of the principles and applications of information theory. These notes will be very helpful for students and professionals looking forward to structured, comprehensive knowledge about the subject.
Author(s): Prof. Tsachy Weissman
75 Pages
Information Theory for Data Communications and Processing
This is a wide-ranging text by Shlomo Shamai and Abdellatif Zaidi, covering both foundational and advanced topics in information theory applied to data communications and processing. It discusses basic issues, such as information bottleneck problems, unsupervised clustering via methods of the variational information bottleneck, and rate-distortion analysis. It proceeds to get into subjects of a higher level of difficulty: non-orthogonal eMBB and URLLC radio access, robust baseband compression techniques, and amplitude-constrained MIMO channels. Efficient algorithms have been derived for multicasting, content placement in cache networks, and the fundamental limits of caching. The title will be a ready reference for researchers and practitioners interested in the theory and practice of modern communication systems, comprehensively covering recent advancement efforts and applications in information theory.
Author(s): Shlomo Shamai, Abdellatif Zaidi
296 Pages
Information Theory and its applications in theory of computation
This set of lecture notes by Venkatesan Guruswami and Mahdi Cheraghchi addresses the intersection of information theory and theoretical computer science. The core topics to be covered in the lecture note include entropy, Kraft's inequality, source coding theorem, conditional entropy, and mutual information. It also covers KL-divergence, Chernoff bounds, data processing, and Fano's inequalities. Key concepts include AEP, universal source coding using the Lempel-Ziv algorithm, and proof of its optimality. It covers discrete channels and channel capacity, the Noisy Channel Coding Theorem, and how to construct capacity-achieving codes by concatenation and by polar codes. Additional topics: Bregman's theorem, Shearer's Lemma, graph entropy, and applications to optimal set disjointness lower bounds. This text offers a wide-ranging view of how the basic principles of information theory shed light on the construction of algorithms, and the establishment of bounds-on the complexity of problems in the field of theoretical computation.
Author(s): Venkatesan Guruswami and Mahdi Cheraghchi
NA Pages
