House of Commons spam fiasco highlights perils of poor content filtering

Up to 5% of incoming email may be incorrectly classified or quarantined: ‘Nobody should accept incorrectly quarantined email as a trade-off for spam-free working,’ says Nexor

13 February 2003 – Nexor is warning companies that over-zealous anti-spam practices are creating black holes for legitimate email that has been incorrectly classified and quarantined. The problem, known as false positives, is often accepted as a compromise for spam-free email. However, Nexor suggests the level of false positives is reaching unacceptable levels.

The trouble comes from basic technology being unable to differentiate between spam and legitimate emails that may contain blacklisted phrases or foreign words, says secure messaging company Nexor. The issue was highlighted recently when the anti-spam solution used by the House of Commons incorrectly quarantined emails relating to the Sexual Offences Bill and bilingual English-and-Welsh content from the Welsh nationalist party, Plaid Cymru.

'There is a real lack of industry research into this problem. However, when using basic content-filtering technologies, it is our opinion that approximately 5% of incoming email is incorrectly classified or quarantined. Of course, anti-spam solutions are an important part of any IT infrastructure, but would we accept 5% of our telephone calls being dropped or misrouted?' says Liz Thomas, VP Marketing at Nexor. 'Email administrators can, of course, retrieve quarantined email, but the question must be asked; how many important, or time-sensitive, emails are being overlooked and what is the effect on UK business? On top of this, organisations must consider the human resource costs of manually reviewing quarantined emails and eliminating false positives.'

Most anti-spam products use content-filtering techniques that rely on basic keyword searches. Messages are then blocked based on pre-determined rules.

'Take the healthcare sector for example,' continues Thomas. 'Using basic keyword searches, you would have trouble distinguishing between the word "breast" used in the context of a medical report, and subsequently, within a spam mail. Organisations need to look at more sophisticated ways of controlling emails and not rely on out-of-the-box keyword search functionality. For example, knowledge management software should be used to classify the content of emails based upon the natural language concepts contained within them. Such technologies apply sophisticated pattern-matching techniques and neural network technology to understand the context of information contained in an email. This enables spam email control to go beyond keywords and identify concepts in the information, then route the email appropriately.'

While testing such technology, Nexor’s messaging products were able to recognise foreign languages (including Welsh) and to distinguish the use of the word 'breast' in a medical report and in a piece of spam email.

In addition to this type of contextual analysis, organisations need to take more control of their email systems and use security policies to determine where messages originate and how they are subsequently routed. Nexor technology uses originator and recipient validation checks and policy-based authorisation checks to limit the amount of spam accepted. Other techniques, such as real-time black hole lists, also help reduce the level of spam. Nexor advises any organisation looking to minimise its levels of spam to implement a number of technologies as part of an overall solution. Only by applying a combined approach will organisations be successful in the battle against spam.

For further information, please info [at] nexor [dot] com (subject: Nexor%20Press%20Release) (contact Wendy Draper)