Shannon’s (fictional) Reverie . . .
Claude Shannon woke up one morning and said to himself, “Think of how our brains operate – it habituates to repeated stimuli but pays attention to a rare stimulus; things that are rare must carry a lot of information!” So, what do I know about quantifying rare or unlikely things? I know that things that are highly likely are highly probable; so unlikely-things or novelty can be thought of as the inverse of probability. But snap! Probability goes from 0 to 1; I need a “squashing” function around it so that the novelty measure does not blow up fast but at the same time, very low probability things (highly novel things) are highly weighted. How about . . .?
A bit of cleanup via taking expectations, probability density functions and some proper logarithms and we have Shannon’s famous equation for “information”,
When you reduce a (joint) probability density function to a scalar, there is a lot that you throw away; the “trick” is that the scalar that you come up with captures some aspect of reality that is *useful*. As the explosion of communication technologies in the past few decades shows, Shannon’s scalar sure did!
I am not claiming that this is how Shannon did his research but this is one way to approach new insights that you may have and their quantification.
Social and Other Networks:
In my recent blog on Social Networks, “What does ‘Emergent Properties in Network Dynamics’ have to do with Shopping?”, I noted the following: “Facebook connects us in a vast network – this is only a first step. The deep reason for the fascination with social networking can be understood from the shopper example. Shoppers are enmeshed in an ever-changing network of social interactions and preferences. Today in Retail Analytics, data are treated as isolated bits of information. In reality, data exist in *embedded* forms in preference and influence networks of the shopper as well as distributed in time and space.”
As you know, interest in understanding such Social Networks and controlling their “dynamics” via “influence functions” of nodes, etc., are at a fever pitch – advertising, retail and many other day-to-day eCommerce activities can benefit from a better conceptualization and quantification of social network dynamics.
We know that “coupling” in networks generate very interesting dynamics (see, Steven Strogatz, “Sync: The emerging science of spontaneous order”, 2003, for a very readable overview). Consider the ultimate of all networks – the brain. When we do “brain mapping”, interesting patterns arise. The brain mapping pictures show scans of a “depressed” and a “non-depressed” person. In the Depressed case, the brain regions are NOT coupled whereas in the non-depressed or Normal case, there is significantly more coupling and more uniform activity across the entire brain. Note however that if we looked at such a scan for an epileptic patient during a seizure, the scan will be all “lit up” showing nearly-complete coupling – that is a degenerate case!
Reprising the Reverie . . .
Coming back to healthy coupling and following Shannon’s (fictional) thinking process outlined in the first paragraph, what do I know about quantifying “coupling”? I know that when the underlying sources are coupled, their “stimulation” of the neocortex is uniform and they show up equally across the network at the same time, much like a single “wavefront” that reaches all the regions simultaneously. Imagine you are at a beach looking out to the sea and gentle waves are rolling in – let us say in parallel to the beach. If you are standing knee-deep in the water and look in a direction parallel to the beachfront (i.e., up or down the beach), the spatial frequency in your “look direction” (or the frequency of “corrugation”) is 0 cycles/meter! Much like the ocean waves, the single waverfront of stimulation is nearly “constant” across the distributed neocortex and its relevant strength can be thought of as the power at zero frequency.
I happen to know that power at zero frequency is called “Scale of Fluctuation” or “θ” in random field theory. From the previous work of Eric Vanmarcke (“Random Fields”, 1983, with an updated edition in 2010),
The result shown above is for a time series. For the brain mapping case, “θ” is 4π2g(0,0) of its 2-D normalized spectral density (same pattern follows for higher dimensional random fields). Calculations of θ for 1-D and 2-D cases are straight forward; ways to calculate “instantaneous” values of θ are also available using Kalman Filtering (introduced in my past publication, “Instantaneous Scale of Fluctuation Using Kalman-TFD & Applications in Machine Tool Monitoring”). Some curious properties of θ for 2nd order linear time invariant systems were also developed there. To recap the highlights –
From discrete-time linear time-invariant system principles, we know that constant damping ratio and undamped natural frequency contours in the z-plane are as shown on the left.
It is notable that for a second-order system, the constant θ contours shown on the right have remarkably simple geometric shapes. In fact, for θ = 1, the equation is quartic but very similar to a circle with origin at (0.5 + j0) and radius = 0.5!
Equation for θ = 1 contour is (x2 + y2) 2 + x2 + y2 – 2x = 0
There are more details in PG Madhavan, Theory and estimation techniques for Random Field Theory and "Theta" with practical applications: Instantaneous Scale ofFluctuation Using Kalman-TFD & Applications in Machine Tool Monitoring, SPIEProceedings, SPIE Vol. 3162, pp. 78-89, 1997.
Similar to Shannon’s scalar, H, θ reduces the joint probability density function to a scalar. Does θ capture some aspect of reality that is useful? The constant θ contours above seem to imply great significance as fundamental as natural frequency and damping – but at this time, such insights are not forthcoming!
Similar to Shannon’s scalar, H, θ reduces the joint probability density function to a scalar. Does θ capture some aspect of reality that is useful? Some real-world applications of θ from the past (see its use for machine tool chatter prediction) point to the following physical insights.
While highly speculative, previous studies and our “Shannon approach” suggest that θ is proportional to “coupling” and to “order” in a distributed node system whereas it is inversely related to “degrees of freedom (df)”. In the case of “df”, the concept is that more degree of freedom is a “dangerous precipice” for a distributed system where different parts are de-synchronized and they can spin off in different directions - pandemonium can ensue!
Large θ seems to indicate an ordered, widely-cooperative and well-functioning network; however, it is conceivable that a large θ may also indicate degenerate cases such as epileptic seizure or full-fledged chatter conditions (extreme cases of coupled sources and distributed action). An example is shown below.
Large θ on the left is a hallmark of “distributed order” whereas the large θ on the right, that of “locked-in order”. Low θ condition in the middle is visually indicative of disorder and the potential for degeneracy!
θ in Social Networks:
To help develop our intuition, let us consider some snapshots of geographcally distributed network maps. We have (1) Internet activity over continental United States, (2) LinkedIn infographic map and (3) Facebook social network map.
We notice strong coupling in the Eastern half of US among Internet nodes and similar features in the LinkedIn and Facebook networks. Our intuition is that the *bright spots* obvious in the Northeast US of the Internet map or the EU area in the Facebook map indicate more “correlated” activity. For simple network maps, we have developed primitive methods to estimate θ based on correlation functions.
Before we leave this blog, consider the brain map and the US map of the Internet. What we see in these pictures can be called “surface structure”, i.e., observed or measureable quantities. In the brain, the Surface Structure is created by activity deep within the brain (I am NOT referring to Chomskian linguistics model here). In the past, naïve physical modeling has conceptualized dipole oscillators in the “deep structure” of the brain giving rise to the Surface Structure as a starting point for theorizing. In the case of the brain, there are indeed Deep Structures (nuclei and ganglia and their dendritic potentials) giving rise to voltage variations on the scalp surface (earthquake tremors recorded on the surface of the earth and activity deep in the earth’s crust form a similar model).
Clearly, the Surface Structure of the Internet cannot be related to any actual Deep Structure in a physical model – there is no mechanical turk behind the Internet pulling the strings! However, even in such cases, conceptualizing observed activity as the resultant of implicit Deep Structure may be useful in developing analysis methods. The hope is that the physical model of network activity utilizing the concept of explicit or implicit deep structures with internal coupling will help advance our “analytics” tools for the extraction of patterns and information from spatially and temporally distributed networked systems.