• 
In the previous article on this topic, we discussed:
Now in this article, we’ll discuss how Curiefense solves this problem.
Curiefense approaches bot management from multiple angles, and uses a series of filters to block malicious bots. They are:
These are applied in a multi-stage sequence. To increase performance, low-overhead methods are applied first. Only traffic that passes the initial filtering is subjected to the more computationally demanding methods. It is very difficult for any forms of automated traffic to pass through the entire sequence.
Curiefense consumes threat intelligence feeds, such as lists of IPs and ASNs currently being used by threat actors. Incoming requests from known-hostile sources can be identified and blocked.
Of the various mechanisms Curiefense uses, this is the simplest. It obviously won’t detect hackers who are using advanced tactics (like abusing cellular gateways to access ‘clean’ IPs).
But it will eliminate a large amount of easily-detected hostile requests with minimal processing. For example, if your service is getting traffic from an IP on the Spamhaus DROP list, there’s no reason to waste extra CPU cycles on analyzing it.
Many bot attacks require a large number of requests to be sent to the targeted system. Common examples are stuffing credentials into login forms, payment card validation, and other types of brute-force attacks.
Curiefense can be configured to count requests that match specific characteristics (e.g., requests from the same traffic source, or that have a specific header, and so on). A traffic source can be blocked when it submits too many requests within a configured time.
This mechanism doesn’t attempt to identify bots by their characteristics; rather, it blocks hostile traffic (whether bot or human) based on the behavior of the sender. In practice, this winds up filtering mostly bot traffic, since a lot of bot-based attacks are volumetric.
Curiefense admins can configure the system to reject traffic based on a variety of characteristics. Common examples are Tor browser users, anonymous proxy users, requests originating from a public cloud IP, and so on.
Depending on the ACLs that are chosen, typically a large portion of the excluded requests are sent by bots.
This is another behavioral mechanism.
Many applications have a natural flow to the requests that the server receives. For example, when someone visits a page in a web application, the server might receive a number of GET requests. Then as the user interacts with the page, a POST request is sent.
Often, hostile bots will not follow this sequence. For example, a bot might attempt an ATO (account takeover) by going to a login page and submitting a number of POST calls, without previously sending any GETs.
Curiefense can be configured to enforce request sequences within a session. Any bot (or for that matter, human) who submits out-of-sequence requests can be blocked.
A common way for detecting bots is to verify that the visitor is using a legitimate browser (Chrome, Firefox, Safari, etc.) instead of a headless browser or emulator.
Threat actors know this, of course. As a result, many modern bots masquerade as human visitors using legitimate browsers. Unfortunately, many commercial security solutions cannot detect that this is being done.
Curiefense offers optional browser verification capabilities that go beyond the techniques used by most commercial solutions. For example, it injects subtle errors into the browser environment, and sees how the ‘browser’ reacts. Curiefense knows the exceptions that are thrown by legitimate browsers (with the claimed version, screen resolution, width, etc.). Other reactions are anomalous, and indicate that this ‘visitor’ is probably a bot.
Mobile/native apps have no browser environment to verify. However, they present a different opportunity for client verification.
Curiefense offers an optional SDK for iOS and Android apps. The apps are rebuilt and published with the SDK embedded. In use, the SDK signs the application, authenticates the device, hardens all communication, and verifies user identity.
This provides a reliable, secure mechanism to verify that the packets are originating from a legitimate user, and not from an emulator or other bot.
Curiefense offers an optional Biometric Behavioral Analysis capability. It uses Machine Learning and UEBA (User and Entity Behavioral Analytics) to construct behavioral profiles of legitimate users and how they interact with the protected applications. Attempts at anomalous usage can then be detected and blocked.
This is similar to the Session Flow Control described earlier, but much more extensive and sophisticated. Instead of relying upon an admin to define a profile, Curiefense does it automatically, based upon a much wider range of parameters. They include:
Threat actors have no access to this data, or the profiles that are created from them. This makes the profiling extremely resistant to reverse-engineering. Note too that the only way for a threat actor to avoid this form of filtering is to avoid any activities that a legitimate user would not do. In other words, for hostile actors to avoid being blocked, they cannot perform any hostile actions.
In the early days of CAPTCHA and then reCAPTCHA, these technologies provided useful benefits to organizations on the web. However, reCAPTCHA is no longer the best way for automatically blocking hostile bot traffic. These two articles have illustrated the differences between reCAPTCHA and Curiefense’s bot filtering.
Curiefense detects bots without relying on any interactions with a human visitor. All of the techniques it uses are accomplished in a few milliseconds, and much of the processing (such as the browser verification) only occurs once, at the beginning of a session.
Of course, threat actors will continue to refine their techniques and attack tools. At the same time, we’re continuing to improve Curiefense. We have a variety of new features on the way, and more are planned beyond those.
If you are a security enthusiast, devops engineers, etc. and want to know more about Curiefense, or if you’d like to file bugs or feature requests, feel free to reach out via our GitHub. We’d love to hear from you!
