Posted in All things Data, Personal Stories

What Humans can Learn from Machines


In my job, I work closely with all things Data. And the magical words you’d hear most likely after that is Artificial Intelligence and Machine Learning. We work with clients to help them use Machine Learning – gleaning insights from their data to gain a sustainable advantage in their business.

In other words, we help them discover what they do not know yet with the Data they already have. And that’s made possible with programming and building algorithms that can learn from the past to predict the future.  

The concept of teaching algorithms to learn from the past and replicating the future is a powerful one – and a lot of it has roots in observation of human behavior.

Look back at the history of how the various branches of Machine learning and AI evolved. Most of the thinking that contributed to this discipline was around making machines intelligent by mimicking inner workings of a human brain. Dig a little deeper and you would find branches like Neural networks & Reinforcement learning – entire paradigms of Machine learning inspired off human thinking processes.

After having worked in this industry quite a bit, and getting familiar with the inner workings of these algorithms, an insight that struck me was how much of the reverse is true.

Of course, many types of Algorithms have been taught to learn based on how Humans think and learn. However, there is a also lot that we Humans can learn from how algorithms get trained , tested and then and perform in the Real World.

Here are some examples:

You learn from what you Observe: Any machine learning algorithm you develop has this computationally intensive phase called the learning phase. You train the algorithm with a certain set of inputs and outputs – the machine picks up the patterns in the data to build a model of the world. Now, when you give it a new set of data to make a prediction – it generates an output based on the representation of the world that it has built.  Isn’t this how real life works?  Oftentimes we lament on our lack of ability to respond favorably to unexpected scenarios.  The reality is – you always learn from what you observe.

Generalizations from scant Data leads to Overfitting – Developing your life’s principles from scant data gives you an inaccurate representation of reality.  When models learn from too little data, then they fall into the peril of Overfitting. What that means is – they perform very well in test scenarios i.e. the environment where they have learnt, but fail miserably in the real world. In real life too, when you develop very strong viewpoints based on little data – it is quite certain that you might be wrong.  One observation of that in the workplace is how every person’s world view gets skewed by what they have seen in their previous roles and organizations – with learnings that might not be completely transferable. Hence, if you have a limited perspective and a new world before you – anticipate that you might be wrong. Look for new data that challenges your established beliefs, and that would help you be aware of the biases you have.

Exposing yourself to new Data enriches you to the next level: When a model does not give us good results – there are usually two ways of improving the accuracy. Either you feed the model new data – which is called ‘feature engineering’, or try a new way of looking at the data which is ‘Algorithm selection’.  Considering that you’ve done your homework right in the first place, in my experience – ‘Feature engineering’ (almost) always trumps ‘Algorithm selection’. The more relevant data you expose an algorithm to, the better it learns.  And the reason that happens is that more and varied data helps the algorithm develop an understanding of a wide variety of scenarios. In real life, the advice you hear is – get out of your comfort zone.  So, while the advice is to go ahead and do something that challenges you, what we are really saying is that expose yourself to a situation that you have not dealt with before.  More data helps you develop a worldview that is diverse and captures the intricacies that enable superior decision making. 

You need many models to map the complexity of the world: With one viewpoint, your understanding of reality is most likely biased. So, don’t depend too much on the opinions of those who are very similar to you. Research, ask questions – seek out diverse viewpoints. Pursue varied opinions because you achieve wisdom through a multiplicity of lenses. Otherwise, if all you know to use is a hammer – everything seems to look like a nail. Taking the parallel from machine learning, we observe that various models perform differently in different data dimensions, and a combination of models usually gives us superior results. So, the learning here is that if you want get a more accurate understanding of reality – think of multiple approaches for solving a problem. “Get a toolbox, not a hammer.”

The world is not Binary: One of my key instincts after years of management experience was to obsessively simplify messaging – get to the heart of the problem and find simple solutions. What I have realized over time is – the world is complex, and working with data and algorithms has helped me appreciate and embrace that complexity. For example, when we build machine learning models – say propensity to upgrade a product, there is usually no single data point that is overwhelmingly predictive of the outcome, but a combination of scores of signals or features that can accurately predict how a customer would behave. Similarly, machine learning also reveals that there can be hundreds of micro-segments in your data – customers with their own unique needs, wants and aspirations, which can be addressed uniquely. The world is not binary, even though we have strong instincts to view it so — ‘We are losing our jobs because immigrants are coming in and taking them’, ‘Equal pay for equal work will solve all women’s problems’. Binary answers are usually not accurate – and can sometimes be downright dangerous.

“Beware of simple ideas and simple solutions. History is full of visionaries who used simple utopian visions to justify terrible actions. Welcome complexity. Combine ideas. Compromise.”

In summary – as researchers and practitioners, we have built AI and Machine Learning systems by replicating the learning processes of human neurons and building patterns in the data that is fed to them. Unknowingly, we might have created a mirror image of real Life in these self-learning systems.  

One which powerful, dynamic and feeds not just from Human learnings, but also informs Humans on how to Learn!

Photo by Franki Chamaki on Unsplash

Posted in All things Data

A Letter of Recommendation – Algorithms


My fascination with Algorithms started when I was quite young. In my teens perhaps – when algorithms and the emerging world of computers seemed to be enticing and promising in equal measure.  My brush with them began in a high school computer class, when we were introduced to these archaic boxes of off-white bulky computers with a grey or black hard keyboards.

It was the early days, computers were a relatively new invention and being able to see one live in front of us was quite exciting. The first language we learnt was BASIC, and then graduated to more cognitively expensive ones like C and C++. You could make the computer do enchanting things, by giving it the most complex earth shattering instructions and then watch with pleasure as it bends over backwards to do your bidding. Indeed, how dramatic! 

And to add to that, these computers were primitive and heated up rather quickly so they needed enclosed and air-tight rooms with air conditioners in what was known as the “Computer department”. If you grew up in a small town with the harsh unforgiving Indian summer – spending time in there was quite a treat. Computer classes were the favorite even among students who didn’t fancy programming.

For myself – I must say that even though I was quite fascinated by the concept of programming, we never really hit it off.   I remember having read through the dense “Algorithms and Data Structures” book in my Engineering to capture any nuggets of wisdom that programming would bring. There was a promise , a connection to all the wonderful happenings in the Tech industry. Dramatic advances in technology  with a vision to transform the world. However, making loops in my head and if-then-else-break statements began to feel like a chore very soon.

Until, one day, almost ten years later – I discovered the magical world of Machine Learning. 

Machine learning, as a concept was a new paradigm where computers do not need to be programmed with explicit instructions about what needs to be done, but can be taught to learn purely by observation. And at the core of it is the concept of Learning. So first you train your algorithms to learn from the past, and with this knowledge of the past learnt primarily by observation, your algorithms can predict the future and take actions. All this may sound very mysterious but there is plain logic and and a lot of math behind all this.

My discovery of Machine Learning was not accidental. I started with reading books and spinning experiments of my own. And slowly, applying these experiments in many work projects exposed me to the inner workings of these digital beasts.  

And the more I knew – the more it astounded me. Imagine a machine that can observe how a system has performed in the past and develop a complete knowledge of the system from Day One. The power of this capability is mind boggling , and frightening in equal measure. 

Today, this is the core of what I do. And yet, the more I discover it, the more enamored I am by how much of this world fits into certain patterns, and how much of it can be discovered through pure math. It is also surprising to me how these algorithms reveal our hidden beliefs and desires, some of them which we might not be aware of ourselves. 

There are very many fears on what this means for our future, and where this technology will take us. And it also raises provocative questions.

  • What can we learn from these super powerful algorithms?
  • What are the benefits and limitations of using these technologies?
  • How do we leverage these technologies without succumbing to the inherent biases they come with ?
  • What are the key challenges we would face — as strategists, programmers, individuals, society and humankind in general?

There are many versions of answers to these. I am hoping to discover my own answers through these pages.

Photo by Kevin Ku on Unsplash

Posted in All things Data

Shopping “Experiences”!!



You have walked into your favorite music store across the street. Just as you pick up a Justin Bieber album off the shelf, digital trackers embedded in the walls identify the album with its RFID tag. A Plasma panel at the store begins to play the latest video of the pop star. As you walk towards the screen to watch this video, sensors in the store track your movements, and a high definition Prosilica camera captures your picture, feeding it to complex algorithms that morph these images. Within seconds, you are pleasantly surprised to see yourself inside the TV screen in the video, gyrating right beside the teen pop star!

Delighted with the experience, you begin to explore the rest of the store. As you move down the aisles, biometric iris recognition software detects your digital identity, and instantly downloads your past purchase and publicly available social networking history into the store’s central database. Eye motion sensors identify which shelf you are looking at, and LCD panels display the message – You “liked” Enrique’s Insomniac on Facebook, would you like to check out his new album “Euphoria” ? Your friend Maria has already bought it, and 59 buddies flashed it on Twitter!!

Well, if you thought all this was a page out of a science fiction book, then think again. This is not a futuristic wish anymore, and sophisticated technologies like these are already being deployed in retail stores right now. With the advent of Web 2.0 and advanced digital electronics, consumer retail has morphed dramatically. Leapfrogging into the future, innovation in IT is now making the customer the center of this “Digital Universe”.

The new age consumer with deep pockets and short attention spans is now more individualistic and much harder to please. With razor thin margins and a consuming need for differentiation, retailers are increasingly pushed towards cutting-edge innovation. Retailing giants like Wal-Mart, Carrefour and Tesco have already boarded the bandwagon, and the others have no choice but to follow. The key is to involve consumers at an emotional and sensory level, and shoppers are now spoilt for choice.

Retailers are now leveraging IT to enhance the end user experience, reduce costs, manage growth and capture multicultural markets. IT innovation is being used to have a consumer centric view to solve problems, achieve creative differentiation and provide complete solutions to task oriented shoppers. In this era of Do-It-for-me marketing, consumers demand retailers to speed up their transactions, driving changes in store concepts, and providing faster and convenient transaction processing and payment.

“Ethnography is the new core competence” remarked Andrew Jones in his book “Innovation acid test”. This has never been more true than now! . Working closely with market research, “knowledge activists” now explore the innovative aspects of any business proposition by cognizing customer intelligence. Corporations compete to identify and “Catch a wave”, creating opportunities to generate growth and fantastic financial performance. Digital immersive technologies like Augmented Reality are being used to integrate branding and entertainment with customer experiences, offering services that resonates with the target customer.

When you look for avenues of innovation in retail, the opportunities are endless. Starting from the use of RFID to measure inventory, warehousing, distribution operations and the supply chain, to providing an enhanced shopping experience for product and brand differentiation, IT can be used as an enabler at every step. All that we need is an imagination to sell the dream!

Augmented technology can be used in storefronts to entice shoppers. Interactive Ads will allow shoppers to “try out” something even before they walk into the store. Large screen kiosks installed at store entrances will empower shoppers to search or quickly browse through the product selection. Digital signage can be used to deliver personalized messages to shoppers, using RFID tags to identify which items they are carrying through the store. Brands stocked in that store can advertise in these digital signs, creating new revenue streams. Mobile coupons are already being used by Starbucks and Target to enhance repeat business. Customers are sent personalized messages alerting them of offers at the store. These services can be customized with location based services, and serve as key drivers of store traffic.

Indeed, IT will now allow us to peer into the living rooms of our customers, data mining their inner desires, and making it available on the supermarket shelves. The IT revolution captures the social entrepreneurship and individualistic spirit of these times. Bill Bryson, a best-selling American author has aptly captured the essence of this era – “We used to build civilizations. Now we build shopping malls”!!