When solving the fragmentation problem the solution is basically rule based. The IDS maps each host on the target network to a specific packet reassembly policy. For example, Windows has a different reassembly policy then FreeBSD, which is the result of interpretations made by the developers of the protocol stack in regards to the RFCs. When packets destined for a specific host are fragmented, the IDS is able to reassemble them in the same fashion as the host. The problem described above is simplified from the typical real world example. It should be clear that the IDS requires more knowledge about the systems it's protecting -- information that could potentially be learned.

Anomaly detection

Consider a more complex issue where intelligence is crucial to security and traffic flows. Typically a noticeable change in network traffic, an anomaly, indicates a security threat. Your FTP server has an outgoing session on port 3164 or your Web server is accepting very large client requests. How does your IDS learn to distinguish between normal traffic and anomalies?

Large client requests are normal for some Web servers and people do run FTP servers on port 3164. Security professionals can hard code the expected network activity; however, this is tedious and not practical for dynamic environments. Instead, your IDS learns what the typical network flow is like and then when the flow changes the IDS alerts you. Then you can give the system feedback as to whether this anomaly is interesting or not. Interesting anomalies will encourage the IDS to identify future interesting anomalies, while allowing the IDS to ignore non-interesting ones. This idea is not new in the AI or the intrusion detection development community and is typically referred to as anomaly detection.

Example of intelligence at work

Here we attempt to demystify the concept of AI by writing a Snort preprocessor that can adapt (that is, a learning preprocessor). It will be a proof of concept to begin with, showing some benefits of intelligence in an IDS, and it will also highlight features of Snort that some people may not have had a reason to tinker with. What is Snort? Below is an explanation that was taken directly from the Snort project Web site:

Snort is an open source network intrusion prevention and detection system utilizing a rule-driven language, which combines the benefits of signature, protocol and anomaly based inspection methods.

There are two key elements that make Snort an ideal choice in this article. For one, it's open source. Also, it's, well, it's open source. So maybe that's just one, but itís a big one! Even if you are not a developer it allows you to leverage developers to produce customizations to Snort.

Snort Traffic flow

Snort passively captures all traffic from a given source using a packet capture module based typically on either WinPcap or LibPcap depending on the given operating system. The traffic then passes through a decoding module responsible for identifying the link level protocols and working its way up the network layers (e.g. Ethernet->IP-> TCP) all the while organizing the packets into data structures for further processing.

At this point Snort may generate alerts for malformed packets. Next, the traffic passes through the preprocessors. For now it suffices to say that the preprocessors identify suspicious traffic or traffic that requires special processing, but we will come back to them later. For example the Frag3 preprocessor identifies fragmented packets and reassembles them according to the destination host (discussed earlier).

Next, the traffic is forwarded through the detection engine, which is the portion of Snort that most people identify as the IDS. The detection engine applies predefined rules to the traffic and generates alerts based on those rules. The rules file is the portion that most administrators update on a regular basis as new vulnerabilities are discovered.

Lastly, output plug-ins dictate how alerts and logs are written. For example, a flat file with an associated tcpdump log. By default there are two customizable portions of Snort -- the rules file and the preprocessors. Here we are interested in preprocessors.

Snort Preprocessors

A Snort preprocessor is essentially some C code that is applied to every packet that is decoded before it is passed on to the detection engine. If Snort is not working the way you like it, or if there is missing functionality, developers or security professionals can take it upon themselves to write in that missing functionality. An example of a Snort preprocessor is spp_fnord.c by Dragos Ruiu.

When exploit code is sent to a machine in the form of shellcode, it typically contains a long series of "no operation, commands" (NOPs). This is called a NOP sled. In a buffer overflow attack the NOP sled supplies the exploit developer with a fudge factor when they figure out what address they should be jumping to in order to execute their shellcode. If the jump instruction lands anywhere in the NOP section of the buffer containing the shellcode the machine will execute the NOPs one after the other, which does nothing, until it reaches and executes the shellcode.