Anomaly Detection based on Prediction - A Step Closer to General Artificial Intelligence







Anomaly detection refers to the problem of finding patterns that do not conform to expected behavior [1]. In the last article "Understanding Neocortex to Create Intelligence", we explored how applications based on the workings of neocortex create intelligence. Pattern recognition along with prediction makes human brains the ultimate intelligent machines. Prediction help humans to detect anomalies in the environment. Before every action is taken, neocortex predicts the outcome. If there is a deviation from the expected outcome, neocortex detects anomalies, and will take necessary steps to handle them. A system which claims to be intelligent, should have anomaly detection in place.

Recent findings using research on neocortex have made it possible to create applications that does anomaly detection. Numenta’s NuPIC using Hierarchical Temporal Memory (HTM) framework is able to do inference and prediction, and hence anomaly detection. HTM accurately predicts anomalies in real world applications such as server anomalies, stock anomalies and geospatial tracking.

All the anomaly detection techniques such as simple threshold, statistical, distance based or supervised methods do not exploit prediction, and hence need a lot of manual intervention. Also those techniques don’t have the capability of neocortex to do continuous learning on streaming data.

Frameworks like HTM are taking anomaly detection closer or even surpassing human like capabilities. This is indeed a step closer to General Artificial Intelligence.

REFERENCES:

[1] Chandola V., Banerjee, A. and Kumar A., 2009.  "Anomaly detection: A survey." ACM computing surveys (CSUR), 41(3), p.15.

[2] J. Hawkins, S. Ahmad, and D. Dubinsky. (2014) The Science of Anomaly Detection [Online technical report]. Redwood City, CA: Numenta, Inc. Available: http://numenta.com/#technology

Comments

Post a Comment

Popular posts from this blog

Can language models reason?

Image
  Illustration by Rajashree Rajadhyax Large language models are an impressive technology that excels at providing answers based on vast knowledge of the world. However, they often struggle with reasoning and logic-based questions. To overcome this limitation, the Chain-of-Thought (CoT) method was introduced. This technique emulates how humans solve problems by reasoning step-by-step. Just as we break down complex problems into smaller, manageable parts for analysis and logical resolution, CoT guides language models to adopt a similar approach. By mimicking this natural, structured problem-solving process, CoT enhances the models’ ability to handle logic-driven tasks with greater accuracy and reliability. This approach, however, requires more time and computational effort because it involves processing more tokens . Tokens are the building blocks of text, like words or parts of words, that the model uses to understand and generate responses. Since the Chain-of-Thought method involv...
Read more

AI implementation myths

Image
  Visionary leaders recognize the importance of adopting cutting-edge technologies to enhance their business operations. This is a bold move, especially in the early stages when few businesses have embraced it. Generative AI is currently at such a juncture. Although AI has experienced several setbacks over the years, recent advancements in foundational models like large language models (LLMs) have reignited interest and investment in the field. Generative AI never fails to impress those who see it in action. At Cere Labs, my company, we have developed a Gen AI framework, and whenever I demonstrate it, the feedback is unanimously positive, with many stating it is "useful." However, despite this enthusiasm, there is hesitation when it comes to integrating this technology into business practices. Addressing certain misconceptions and concerns could help leaders more easily adopt Generative AI to enhance their operations. With my 8 years of experience assisting businesses in inco...
Read more

Homework 2.0: The science behind homework and why it still matters in the AI age!

Image
  Illustration by Rajashree Rajadhyax I recently came across an article about how teachers are coming up with methods to distinguish between homework done by students and that done by AI. This effort by teachers is not without cause; a recent case in Massachusetts saw a student penalized for using AI in their history paper, highlighting the need for such measures. I found it intriguing and couldn’t help but reflect on this shifting trend.  Homework! Does that word bring back memories? The race to finish it as fast as possible so you could finally go play! Holidays would’ve been so much better without it, don’t you think? I know I felt that way, and I’m pretty sure you did too. And let’s not forget those classmates with endless excuses for not doing it. The most common one? “I wasn’t feeling well!” 🤒 Some would even throw in a few coughs just to make it believable 😏. I was that kid on the opposite end. Just before a holiday, I’d be sneaking in some homework during lunch break...
Read more