Shannon’s (fictional) Reverie . . .
Claude Shannon woke up one morning and said to himself, “Think
of how our brains operate – it habituates to repeated stimuli but pays
attention to a rare stimulus; things that are rare must carry a lot of
information!” So, what do I know about quantifying rare or unlikely things? I
know that things that are highly likely are highly probable; so unlikely-things
or novelty can be thought of as the inverse of probability. But snap!
Probability goes from 0 to 1; I need a “squashing” function around it so that the
novelty measure does not blow up fast but at the same time, very low
probability things (highly novel things) are highly weighted. How about . . .?
A bit of cleanup via taking expectations, probability density functions and some proper logarithms and we have Shannon’s famous equation for “information”,
When you reduce a (joint) probability
density function to a scalar, there is a lot that you throw away; the “trick”
is that the scalar that you come up with captures some aspect of reality that
is *useful*. As the explosion of communication technologies in the past few
decades shows, Shannon’s scalar sure did!
I
am not claiming that this is how Shannon did his research but this is one way to approach new insights that you
may have and their quantification.
Social and Other
Networks:
In my recent blog on Social Networks, “What does ‘Emergent Properties
in Network Dynamics’ have to do with Shopping?”, I noted the following:
“Facebook connects us in a vast network – this is only a first step. The deep
reason for the fascination with social networking can be understood from the
shopper example. Shoppers are enmeshed in an ever-changing network of social
interactions and preferences. Today in Retail Analytics, data are treated as
isolated bits of information. In reality, data exist in *embedded* forms in
preference and influence networks of the shopper as well as distributed in time
and space.”
As you know, interest in understanding such Social
Networks and controlling their “dynamics” via “influence functions” of nodes,
etc., are at a fever pitch – advertising, retail and many other day-to-day eCommerce
activities can benefit from a better conceptualization and quantification of
social network dynamics.
We know that “coupling” in networks generate very interesting dynamics (see, Steven Strogatz, “Sync: The emerging science of spontaneous order”, 2003, for a very readable overview). Consider the ultimate of all networks – the brain. When we do “brain mapping”, interesting patterns arise. The brain mapping pictures show scans of a “depressed” and a “non-depressed” person. In the Depressed case, the brain regions are NOT coupled whereas in the non-depressed or Normal case, there is significantly more coupling and more uniform activity across the entire brain. Note however that if we looked at such a scan for an epileptic patient during a seizure, the scan will be all “lit up” showing nearly-complete coupling – that is a degenerate case!
Reprising the Reverie . . .
Coming back to healthy coupling and following Shannon’s
(fictional) thinking process outlined in the first paragraph, what do I know about quantifying “coupling”?
I know that when the underlying sources are coupled, their “stimulation” of the
neocortex is uniform and they show up equally across the network at the same
time, much like a single “wavefront” that
reaches all the regions simultaneously. Imagine you are at a beach looking
out to the sea and gentle waves are rolling in – let us say in parallel to the
beach. If you are standing knee-deep in the water and look in a direction
parallel to the beachfront (i.e., up or down the beach), the spatial frequency
in your “look direction” (or the frequency of “corrugation”) is 0 cycles/meter!
Much like the ocean waves, the single waverfront
of stimulation is nearly “constant” across the distributed neocortex and its relevant
strength can be thought of as the power at zero
frequency.
I happen to know
that power at zero frequency is called “Scale
of Fluctuation” or “θ”
in random field theory. From the previous work of Eric Vanmarcke (“Random
Fields”, 1983, with an updated edition in 2010),
The result shown above is for a time series. For the brain mapping
case, “θ” is 4π2g(0,0) of its 2-D normalized spectral density (same
pattern follows for higher dimensional random fields). Calculations of θ for
1-D and 2-D cases are straight forward; ways to calculate “instantaneous”
values of θ are also available using Kalman Filtering (introduced in my past publication,
“Instantaneous
Scale of Fluctuation Using Kalman-TFD & Applications in Machine Tool
Monitoring”). Some curious properties of θ for 2nd order linear time
invariant systems were also developed there. To recap the highlights –
From discrete-time linear time-invariant system principles, we know
that constant damping ratio and undamped natural frequency contours in the z-plane
are as shown on the left.
It is notable that for a second-order system, the constant θ contours
shown on the right have remarkably simple geometric shapes. In fact, for θ = 1,
the equation is quartic but very similar to a circle with origin at (0.5 + j0)
and radius = 0.5!
Equation for θ = 1 contour is (x2
+ y2) 2 + x2 + y2 – 2x = 0
There are more details in PG
Madhavan, Theory and estimation techniques for Random Field Theory and
"Theta" with practical applications: Instantaneous Scale ofFluctuation Using Kalman-TFD & Applications in Machine Tool Monitoring, SPIEProceedings, SPIE Vol. 3162, pp. 78-89, 1997.
Similar to Shannon’s scalar, H, θ reduces the joint probability density function to a scalar. Does θ capture some aspect of reality that is
useful? The constant θ contours
above seem to imply great significance as fundamental as natural frequency and
damping – but at this time, such insights
are not forthcoming!
Similar to Shannon’s scalar, H, θ reduces the joint probability density function to a scalar. Does θ capture some aspect of reality that is
useful? Some real-world applications of θ from the past (see its use for machine
tool chatter prediction) point to the following physical insights.
While highly speculative, previous studies and our
“Shannon approach” suggest that θ is
proportional to “coupling” and to “order” in a distributed node system whereas
it is inversely related to “degrees of freedom (df)”. In the case of “df”, the concept
is that more degree of freedom is a “dangerous precipice” for a distributed
system where different parts are de-synchronized and they can spin off in
different directions - pandemonium can ensue!
Large θ seems to indicate an ordered, widely-cooperative and
well-functioning network; however, it is conceivable that a large θ may also
indicate degenerate cases such as epileptic seizure or full-fledged chatter conditions
(extreme cases of coupled sources and distributed action). An example is shown
below.
Large θ on the left is a hallmark of “distributed order” whereas the large θ on the right, that of “locked-in order”. Low θ condition in
the middle is visually indicative of disorder and the potential for degeneracy!
θ in
Social Networks:
To help develop our
intuition, let us consider some snapshots of geographcally distributed network
maps. We have (1) Internet activity over continental United States, (2) LinkedIn
infographic map and (3) Facebook social network map.
We notice strong coupling in the Eastern half of US among Internet
nodes and similar features in the LinkedIn and Facebook networks. Our intuition
is that the *bright spots* obvious in the Northeast US of the Internet map or
the EU area in the Facebook map indicate more “correlated” activity. For simple
network maps, we have developed primitive methods to estimate θ based on
correlation functions.
Before we leave this blog, consider the brain map and the US map
of the Internet. What we see in these pictures can be called “surface
structure”, i.e., observed or measureable quantities. In the brain, the
Surface Structure is created by activity deep within the brain (I am NOT
referring to Chomskian linguistics model here). In the past, naïve physical
modeling has conceptualized dipole oscillators in the “deep structure” of the
brain giving rise to the Surface Structure as a starting point for theorizing.
In the case of the brain, there are indeed Deep Structures (nuclei and ganglia
and their dendritic potentials) giving rise to voltage variations on the scalp
surface (earthquake tremors recorded on the surface of the earth and activity
deep in the earth’s crust form a similar model).
Clearly, the Surface Structure of the Internet cannot be related
to any actual Deep Structure in a physical model – there is no mechanical turk behind the Internet
pulling the strings! However, even in such cases, conceptualizing observed
activity as the resultant of implicit
Deep Structure may be useful in developing analysis methods. The
hope is that the physical model of network activity utilizing the concept of explicit
or implicit deep structures with internal coupling will help advance our “analytics”
tools for the extraction of patterns and information from spatially and temporally
distributed networked systems.
www.JINinnovation.com
