Shannon information deliberately only concerns syntactic information (information content). Other, more recent work, focuses on semantic information (information with meaning for a receiver).
> Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state.
Roughly speaking: The amount of computation or energy needed to perfectly reproduce a random source, such as a coin flip, is high, while the significance or meaning, for the average receiver, is low. Natural language text requires less computation to reproduce [1], but, for the average receiver, the significance is higher.
Hmm, but you could compress text to an equally random sequence. I.e. all minimal programs are by definition Kolmogorov random.
Also, what about crystalline forms, which are very orderly and require minimal computation to reproduce, but are equally insignificant for the average receiver?
> you could compress text to an equally random sequence
More or less correct. The key difference is that you could not compress a random coin flip sequence (and that a compressed text is meaningless until decompressed to original).
> all minimal programs are by definition Kolmogorov random
Compression provides an upper bound to K. Kolmogorov Randomness itself is not computable. AKA: You can't ever know if you have a minimal program.
The best approach that I've seen is a combination of Shannon information and Kolmogorov complexity. If an object has high Shannon information, then it is not crystalline. If it also has low Kolmogorov complexity then it is not random. This seems to characterize the sweet spot where meaningful information occurs. Kolmogorov called this quantity "randomness deficiency".
> Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state.
https://arxiv.org/abs/1806.08053
Roughly speaking: The amount of computation or energy needed to perfectly reproduce a random source, such as a coin flip, is high, while the significance or meaning, for the average receiver, is low. Natural language text requires less computation to reproduce [1], but, for the average receiver, the significance is higher.
[1] http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf