By Dr. Rustom Kanga

Studies have shown that an operator watching 2 monitors misses 45 % of the action after 10 minutes and after 22 minutes he misses 95% of the action.  This is in a relatively empty scene.  In a busy scene it takes even less time for an operator to miss action on a screen.  What are the chances that an operator watching hundreds and sometimes thousands of screens would observe anything of use?

Organizations requiring security have therefore realized that the very large sums of money that they have spent on cameras and control rooms provides them with little real security.  The video recorders they have installed are useful after an event but have little value in preventing an incident.

Hence there has now been the growing realization that for organizations to get value from their security systems they need to add ‘intelligence’ such that the operator will be told when an incident happens.  The system should be able to tell him that a person has fallen down on Camera 235 or that a bag appears to have been abandoned on Camera 650.

Now that astute users of security are recognizing that without intelligence their investments in security equipment are underutilized at best and at worst useless, there is a burgeoning industry of providers of such intelligence.  Suppliers who provide nothing more than simple features such as database access tout themselves as providing intelligence, causing significant confusion in the minds of the users.  How can one tell the difference between hundreds and even thousands of suppliers who all purport to add intelligence to security systems?

At iOmniscient, as suppliers of Intelligent Surveillance we had the same problem.  We had products which were extremely intelligent -- being able to do things that humans could not do and others which were relatively simple products involving simple programming, using techniques that are practiced in many universities and available in the public domain.  And since the company’s products covered the whole range of capabilities in the Intelligent Surveillance field we came up with an Intelligence Rating system for our own products which can of course be used more universally by users to define the intelligence of their own systems.

CATEGORIZATION OF ‘INTELLIGENT’ TECHNOLOGIES

To start with there are two major classes of Intelligent Technologies -- those involving Detection and those involving Identification.  Detection involves observing a scene and understanding what is happening in the scene (e.g., if someone has jumped over a fence).  Identification involves knowing the identity of the person (i.e., knowing that Jack Smith is the person who jumped over the fence).  Note that both Detection and Identification can be used for objects other than humans.  Identification for instance can be extended to vehicles using number plate recognition systems.

DETECTION TECHNOLOGIES

The measurement of human IQ is based on a normalized system.  The average person in the population has an IQ rating of 100.  That means 50% of the population has a rating higher than 100 and the remainder has a rating below that.  A similar rating system was used on ‘intelligent’ security products.  Those algorithms involving enormous complexity mimicking the human brain were classified as requiring a high IQ while other techniques that were simple to implement and were deemed to have a low IQ rating.

The products were rated as having an IQ from as low as 60 to as high as 180.  As one would expect in looking at the market there are many suppliers who offer the lower IQ capabilities.  As the IQ rating increases the number of players in the market decreases rapidly.

At the lowest level of the scale the algorithms are based on Video Motion Detection (VMD).  VMD in its simplest form consists of comparing the pixels on one image with those on the next.  If there is a difference this means that there was some change in the scene usually interpreted as motion.  Such systems rated at IQ 60 are widely available but of course as the IQ rating implies they are of little value as they are prone to false alarms.  Not every change is due to motion.  Light variations, reflections on water, shadows and a host of other changes cause the change in pixels making such systems useful only to those who want to claim their systems have intelligence while not expecting them to do anything practical.

At the next level of sophistication systems can be set up to group the pixel changes between images as ‘blobs’ and to then track the movement of the blobs across the screen (IQ 100).  The characteristics of the blobs (such as size or shape) can also be analyzed and the system can therefore differentiate between people and small animals or between cars and trucks.  Even within an IQ level there can be many variations in technology.  It can be quite simple to track a single person in a relatively empty scene.  As the scene gets more crowded the algorithm has to cope with blobs that merge and split.  Tracking a particular person through a crowded scene can be a very complex task and not all companies who can supply a technology with this rating can necessarily cope with such scenes effectively.

Using this level of technology it is also possible to detect left objects in a relatively empty scene as this just involves noting when a blob has split and where one part of the old blob remains stationary.

The next level of capability (IQ 110) involves being able to clearly define the item that is being tracked (say people) and to be able to count them accurately.  As with IQ 100 there can be wide variations in how well suppliers do counting.  It depends on their ability to cope with blobs that split and merge.  A good algorithm that has been professionally set up can be very accurate.

Once blobs have been tracked and counted the next level of sophistication involves understanding what the blobs are actually doing.  This is Behaviour Analysis (IQ 115).  At this level of technology one can detect loitering, running and slips and falls.  Behaviour can be culturally dependent and hence not all behaviours are easy to detect by a system.  For instance I observed a system designed to detect fighting.  It worked well in the U.K.  However, it collapsed in Italy as it easily mistook two exuberant Italians meeting after a long separation as fighting.  At this level there are less than a handful of companies that can offer a comprehensive set of behaviour detection algorithms.

As one gets to IQ 120 the technology moves from detecting and observing individuals to operating in crowded environments.  Crowd Management at this level can provide capabilities such as determining how many people are in a very crowded scene at a given time (as opposed to counting individuals passing through a gate).

Finally one gets to IQ 140 which allows the detection of abandoned objects (or removed objects) in a crowded scene*.  This is useful in an environment such as an airport where luggage may be left unattended or in museums or warehouses where items may be stolen.  Such systems have to cope with very long detection times (if the detection time is too low there would be thousands of nuisance alarms from passengers placing their bags down momentarily).

And with the long detection times the system has to cope with significant obscuration (where the object is obscured for a significant period of time by passersby).

Finally at IQ 180 the system has the capability to do everything that can be done at IQ 140 does, but it does it even when the object may be invisible to the human eye because the object is tiny and with little contrast.

Often systems will do one or the other of these detections on a camera scene.  A system that can do all of them at the same time has been defined as IQ Infinity.

Figure 1.  IQ Rating of Intelligent Surveillance Capabilities  (Source: iOmniscient Corporation)

RECOGNITION: GOING BEYOND DETECTION

Having detected an object in the scene, the next step is to understand what it is.  At the simplest level one can differentiate a person from small animals or vehicles on the basis of size and aspect ratio.

This area becomes more complex when it becomes necessary to differentiate between two objects that may have similar size and shape.  The conventional approach for achieving this was to use Neural Networks.  Essentially this involved the system being shown multiple instances of an object and then doing a match against these.  For instance, if one wanted to recognize a dog, one would feed the system with thousands (or even millions) of images of different dogs from different angles.  When an object was seen its image would be matched against those of dogs in the data base and if there was a match one would know it was a dog.  Neural networks are useful for simple matching.  Unfortunately for complex matching it involves massive amounts of computing power and storage.  I call this the memory approach -- to attempt to recall what a dog would look like based on our memory of it.

The alternative approach involves using reasoning rather than memory.  For instance in the above example one could tell that the object seen was a dog from its characteristics such as its wagging tail, its floppy years or the way it moved or behaved.  This approach using reasoning is enormously powerful as it uses little computing power and storage.  Moreover, it mimics the way human actually think and hence the system can use criteria that a human might use for recognizing a particular object.

Such technologies are used to enhance products at various levels of intelligence.  For instance, with an IQ 140 level system installed in an airport, the user may wish to detect abandoned luggage.  However, in most airports he would also detect abandoned trolleys/luggage cards.  An intelligent system can not only detect a left object but discriminate between a bag and a trolley/ hand cart.  Further, it can tell if the luggage cart is empty or loaded.

GENERAL CHARACTERISTICS THAT ALL SYSTEMS SHOULD HAVE

All systems, no matter what their intelligence level, need to have certain core characteristics.  Several of these features are architectural though some do require a level of intelligence.  Architectural characteristics include openness (the ability to take inputs from any camera, operate on any computer and interface with any Video Recording system) scalability (the ability to grow from a single camera to thousands of cameras) and distribution (the ability to place the intelligence either centrally or remotely in a network.

Intelligence characteristics include the ability to avoid nuisance alarms caused by extraneous factors such as light changes, water reflections or shadows, the ability to understand perspective and the ability to know if all the cameras can see properly and are operational (specially if the system is prone to sabotage, vandalism or just bad weather).

A very significant general characteristic of a system that would differentiate a smart system from one that is less so is the ability to ‘jump to Event’.  Most systems have the ability to set a ‘pre-alarm’ time and to go back a short period before the alarm.  But what if one is looking at a crowded airport and sees 10 suspicious bags?  A sophisticated Jump to Event system can allow the User to focus on any one of those bags and with a single click of a button, see when it was brought in and by whom.

IDENTIFICATION TECHNOLOGIES

Video based identification technologies fall into two classes today -- those focused on identifying humans and those used in identifying objects such as vehicles.

Facial Recognition technologies have been around for a while.  Today techniques for one-on-one recognition are fairly widely available and they can be used for checking if a person matches his photograph in a passport.  This is interesting but does not necessarily improve productivity significantly as a human looking at the passport could do the same task in as much time.  The more sophisticated systems are focused on identifying people in a many-to-many environment.  As such they would attempt to recognize a number of terror suspects mingling in a crowded scene.  Unfortunately ‘many-to-many’ systems have not proven to be sufficiently accurate as yet.

Vehicle Identification systems can read License Plates.  The more sophisticated systems can also detect the vehicle make or model based on the emblem or the grill.  The most sophisticated systems may use multiple images to gain accuracy using statistical models.

CONVERGENCE OF DETECTION AND IDENTIFICATION

Traditionally Detection and Identification needed different types of cameras placed differently.  Identification required a close up view of the object.  Detection requires a far off view so that the object can be viewed in the context of its surroundings.  The latest innovations involve being able to do detection and identification on the same cameras and this technology will be described in the next issue.

SO, WHAT IS THE IQ OF YOUR SYSTEM?

As Users one needs to ask oneself as to what their system is required to do.  Are you, as a user, happy to have a system that is only useful after the fact to review disasters or do you prefer to have one that can help you to pre-empt disasters.  What types of events are important to you?  Can your system help you to avoid these events or in the worst case tell you instantaneously if one occurs?  The IQ Rating system described will help you to assess the level of intelligence your system has and needs.

Dr. Rustom Kanga is the founder and CEO of iOmniscient Corporation (www.iomniscient.com).

For more information, please send your e-mails to swm@infothe.com.

ⓒ2007 www.SecurityWorldMag.com. All rights reserved.