People watching

How facial recognition is evolving into a major industry as it helps to control stadium crowds

Human beings are exceptionally good at spotting a familiar face in a crowd. We are adept because, at its most primitive level, the skill of being able to recognise a friend or a foe is one that improves the chances of our survival.

An individual’s identity is verified by facial
recognition technology

Now scientists are trying to create technology which does the same job, in order to improve security at large events. CCTV cameras, data storage and computing power are relatively cheap so they hope to combine them all into a system that automates the process of people-watching. They want to check automatically the identity of each person who passes by and receive an alert when the face of a suspect is spotted.

The first significant trial of the technology at a major sports event was at the 2001 Super Bowl in Tampa Bay, Florida, USA, where software surreptitiously scanned everyone passing through the turnstiles. Probable matches with the faces of known criminals were displayed on the screens in a police control room.

Afterwards local police claimed the system had pinpointed 19 people with criminal records out of a crowd of 100,000. It's not clear how many "false positives" were made in the Florida trial but face recognition technology was not used again at the 2002 Super Bowl.

Put simply, it wasn't good enough at sorting the bad guys from the good guys. "If you have your settings low enough to capture people, it captures way too many people," the head of stadium security was reported as saying. "If you set it too high, it doesn't capture anyone."

It was more than two years before face recognition was tried again at a large sports event. A system was launched at the stadium of PSV Eindhoven football club in the Netherlands at the end of 2003. Two cameras were pointed at a section of the crowd where hooligans were known to congregate. "They were cameras with pan/tilt and zoom functionality so we could zoom in on their faces," said Arnoud van Zuijlen of LogicaCMG which was in charge of the project.

Facial recognition technology is rapidly
developing into a big industry

The cameras scanned the crowd at five matches in the next three months, capturing images of more than 100 faces at each game and comparing them with pictures of known troublemakers that had been taken in the recent past by the police. "Approximately 85 per cent of all identifications were correct, being either a correct match on a person in the police database or a no-match because the person was not in the database," said van Zuijlen.

After the trial the system was abandoned due to political reasons because the organisations involved, including the club and the police, could not agree which should pay for the technology to control hooliganism. But it had been a useful experiment and it taught the technologists three key factors needed to improve the accuracy of face recognition: lighting, camera position and the possible obscuring of faces by clothing such as hats and scarves.

"While lighting can be improved in stadia and camera positioning can be massively improved with a little planning, the issue of clothing is quite a challenge," said a spokesman for LogicaCMG. The company has since tested it again in the more controllable environments of airports and shopping malls.

The third major trial of face recognition software at a major sports venue began at Bern Arena, Switzerland, in November 2005. It's a joint project between SC Bern ice-hockey club, Broncos Security and Unisys. One hundred fans volunteered to have their faces scanned into a database for a three-part test.

The first part is when people insert their entry tickets into the automatic turnstile and a camera tries to recognise the volunteers. The second test is held during the matches and a rotating camera tries to pick out the volunteers' faces from the crowd. Finally, a mobile camera with a wireless link to the database again tries to spot the volunteers as the crowd leaves the stadium.

The purpose of the pilot project is to establish the identification hit rate, to find out how many images security officers can simultaneously view on their monitors and how instructions can be issued to the security personnel on the spot. All results as well as errors are being documented.

Facial recognition is used for two fundamentally different tasks: verification and identification. Typical verification jobs ensure that people really are who they claim to be before allowing them access. The technology compares a current image to images volunteered for a database so the success rates are good.

On the other hand, identification jobs attempt to match unknown individuals from surveillance CCTV with images from passports or criminal records. This is more difficult and, in practice, officials often use facial-recognition technologies only to narrow the list of potential matches before passing the job on to a human analyst.

Current facial-recognition technologies use one or more of four basic methods. Appearance-based methods measure the similarities of two or more images. Rule-based methods analyse facial components, such as the eyes, nose and mouth, to measure their relationship between images. Feature-based methods analyse the characteristics of facial features, including edge qualities, shape and skin colour. Texture-based analysis examines the different texture patterns of faces.

The technology is still in its infancy but financial analysts expect it to blossom. Consultants Frost & Sullivan say the global market that was worth USD 186m in 2005 will grow by 27.5 per cent annually to reach USD 1.2bn in 2012.

Face recognition is just one part of the biometrics bandwagon and it is one of the hardest to perfect. Others that are more accurate and widespread include fingerprint and iris recognition. They are already in place and being used for border control. Also under development is gait analysis, automatically recognising people by the way they walk. Even preliminary work in ear recognition has begun.

Voice recognition is being applied to telecoms systems, giving access to accredited users whose speech pattern matches a pre-recorded template. The next big challenge is teaching computers to lip read. If that works then not only will the technology be able to match human ability to spot a face in a distant crowd but it will also be able to understand what the person is saying.