Hyenas, lions and city lights – accurately measuring behaviour is rarely straightforward

“The spotted hyena is a vastly unappreciated, misrepresented beast.” I’m sure it is, you say, but what’s it doing in your analytics blog?

When I‘m not keeping up with the latest revolutions of the big data hype cycle†, you’ll find me reading popular science books. It’s amazing how often I’ll read about an idea from a seemingly left-field area of study that connects back into my own interests. It always pays to read widely, especially when you’re into analytics.


Spot the thief (image courtesy of lubye13 via Flickr)

Spot the thief (image courtesy of lubye13 via Flickr)

The above quote is from “Why Zebras Don’t Get Ulcers” by Robert Sapolsky, an expert in the differences between the stresses felt by animals and by humans. What really caught my attention was the section on how, and why, biologists got the wrong end of the stick for so long when it came to the poor old hyena. Sapolsky paints a picture of dawn rising over the Savannah, and a film camera zooming in on a skulking hyena stealing food from a recent lion kill, like the furtive born scavenger we know it to be.

Think again. Thanks to hand-me-down night vision scopes courtesy of Pentagon-surplus stores, the hours between daylight painted a very different story. Largely nocturnal, hyenas turn out to be fabulous hunters. Nimble and discreet they are far more effective than the big, slow and conspicuous lions, who the new observations unmask as the real thieves. When the light returns, what we are seeing is a disgruntled hyena trying to get its own kill back from the lions. Direct observations in only natural light were a partial measurement, which had given the wrong context. Limited seeing leads to unreliable believing, an important lesson for our data-driven future.

As if the gods were conspiring with me, I ran into another startling example in my next book¥ on the key images from science. Viewing the earth at night from space, the cities of the developed world shine out, highlighting the major areas of population density. We can see where all the people are by the lights! Again, not so fast – we can be deceived by a misreading of a “measurement”. Light is not, in general, a reliable indicator of population mass. The dark regions of Africa, Asia and South America contain the larger proportion of humanity. The observation of light is a measure of affluence, not people. It’s like our own local version of the missing mass in the Universe.

The secret is to collect enough of the big picture alongside your targeted measurements, to establish the full context. When I built a data-driven condition monitoring system to combat poor train reliability, it wasn’t enough just to measure data feeds from the suspect components. It needed additional feeds to establish the different operating states of the train, such as accelerating, braking, or coasting, as well as its location on the network. Only then could I have a broad enough picture of the real environment of my subject.
But as soon as I start to talk about trains and engineering, eyes glaze over and it all sounds very specific and technical. The beauty of the examples from popular science lay in their roots in everyday experience, which can be appreciated by all of us.

I still prefer lions, mind.

† sorry, I forgot – Gartner have decided big data is no longer interesting hype http://www.datanami.com/2015/08/26/why-gartner-dropped-big-data-off-the-hype-curve/
¥ Cosmic Imagery, John D. Barrow, W. W. Norton & Company

Nick Clarke

Nick Clarke

Following a career in academic research, I have worked in the commercial IT industry for about 15 ...

Latest Tweet from Nick Clarke

RT @Tessella: Read @AnalyticsCave Matt Jones' latest blog on @ForbesTech How Do We Avoid The Threat Of Supercharged Artificial Stupidity? h…

about 5 days ago

© Copyright 2018 Tessella
All rights reserved