Logical Defensive Software

Signature-based IDS misses what it has no rule for. Behavioral ML catches the unknown, but it is expensive and slow. What I'm building is an attempt to get both: a fast rule-based path for what we already know, and a slow behavioral path that learns new patterns and promotes them down to the fast path. A system that gets cheaper and faster the longer it runs.

The name comes from Cyberpunk 2077. Logical Defensive Software there is not just a set of rules and monitoring probes stopping cybercriminals from reaching your system: it is something working on a deeper, logical level. I'm trying to bring that idea to life with embedded ML micromodels.

The Linux at the source

Using Linux and doing low level development on the Linux Kernel lets me use the full power of Open Source software: the ability to read, understand and modify source code, or build on top of it. I already had projects using sysctl directly from Golang to deploy single-binary solutions on top of the Kernel, without the whole GNU stack, and it worked amazing. Now I'm trying to use the same knowledge to go deeper, but also across a wider surface.

SNORT

In the early 2000s I was testing Prelude IDS and then Snort to detect malicious actions against Linux servers I was administrating. Those detectors worked generally fine. They used rule-based filtering to validate whether a packet sent to your machine is malicious or safe to process.

Those days are gone. Signature matching alone is not enough anymore: it catches only what it already has a rule for, and misses novel attacks by design. But the muscle memory remained. That same muscle memory pushed me to combine it with the ML knowledge I picked up slightly later, around 2013-2015, when I was building first Neural Networks used effectively in meteorology. Why not merge the two. So I did.

My version of LDS

LOD.drawio.png

In this diagram of LDS the colors mean:

  • Blue: in-LDS-scope components
  • Purple: Kernel-level component
  • Green: system services (web server, DNS, SSH, NTP, TFTP, RSync, SMTP, SMB)
  • Yellow: logical source the high-level data comes from

I split my LDS into two processing paths: fast and slow.

The fast path

The fast path is rule-based, going through the Rule processing module. It collects data from Netfilter, log analyzers, and the ethernet device (raw traffic TAP). This path is a simple rule-to-action mapping. It looks at the input params, validates whether a rule exists, and if it does, sends the matched rule as an event to the anomaly detector. The anomaly detector looks at other parameters, correlates the data using a small CPU-bound micromodel, and generates a Netfilter blocking/dropping rule (ban).

Fast means cheap and hot. This is the path that catches what the system already understands.

The long path

The long path is where real detection happens. Here the behavioral processing module is used. It has two subcomponents: a pattern matcher and a temporal analyzer.

The pattern matcher is a simple pattern-finding tool. When it sees a deviation in traffic it can backfeed its data to the temporal analyzer. Think of it as an old-school IDS that met a treadmill and dropped two thirds of its weight: only the engine left. On top of that IDS-like method I'm adding a micromodel, a basic neural network, to analyze patterns across more variables than a rigid pattern DB can assume. Rigid patterns false-positive and false-negative a lot; the model is there to soften that.

The temporal analyzer is a streaming processor. I'm experimenting with a small SNN here. SNN is normally used for audio, video or sensor data, but the idea is that a log or event stream is also a temporal signal: successive lines are the incoming data, and the timing between events carries meaning. So the neurons can be tuned to spike on a specific behavior over time. Example: a port scan followed by two failed SSH passwords within tenths of a second. A static classifier looking at one line in isolation would miss that; the temporal one should not.

Temporal analysis and/or pattern matcher alerts go to the behavioral classifier. There another ML micromodel decides whether the proof of malicious behavior is solid enough to push to the anomaly detector, to be fused with other platform data, or whether it should be dropped at that point.

This is also where the platform can self-optimize. The model can decide a situation is so common that it should become a permanent rule in the rule classifier: promoted to the fast path. This lets the platform learn and adjust, jumping from expensive slow processing to a generated, specific rule on the fast, cheap track that just blocks what meets the requirement.

Promotion is the interesting and the dangerous part, and I'm treating it carefully. A rule promoted too eagerly poisons the fast path: now you have a fast, deterministic rule blocking legit traffic, without the model context that could have nuanced it. So generated rules are meant to carry their origin, a confidence, and a TTL, kept separate from hand-written ones, and the fast path should report its hits back to the slow path so the model does not go blind to a pattern it promoted and cannot notice when that pattern goes stale.

The anomaly detector gets data from the behavioral classifier (trigger), the rule classifier (trigger), the service metrics monitor (argument), the log analyzer (argument), the process monitor (argument) and the system consistency monitor (argument). It uses an ML micromodel to fuse all of this and decide whether a rule should be pushed to Netfilter to block the connection.

Not only network monitoring

The design is meant to look beyond the network. Service metrics, log analysis, process state and system consistency are there to judge whether the machine can actually serve a request: whether it has the resources, whether it is in a consistent state, or whether it has already been affected by malware or intrusion. If not, the request can be rejected to keep the system from collapsing.

That is the intent of the architecture, not a claim that it does all of this today. Cutting access to a machine that is already compromised, or shedding load before it falls over, is where I want this to go.

The progress

I'm at the beginning of the path: gathering components, doing PoCs, clarifying rules, collecting data for ML training. This will be a long process. But I'm experimenting with it and I hope to share the progress as it goes.