Investigating website fraud – the bad and the ugly

In my last post, I talked about the Koobface virus, how it is an example of a business logic flaw, and how difficult it can be to find business logic flaws on websites. BTW – I should know. It used to be my job to find, investigate, and fix the business logic flaws on eBay. I’m sure you’d agree that is not a trivial task.

Let’s look at the next step of addressing business logic flaws: investigation. Once the business logic flaw has been identified (usually after days of customers reporting the problem), it’s critical to determine how the attack is being carried out (and, if possible, by who).

Here’s a typical investigation technique by someone trying to determine the exact method of the perpetrators of a business logic attack:

1. Grab all of the weblogs for the website during the impacted time period

2. Sort through the weblogs to find the exact transactions that represent the attack

3. Correlate the data in the attack transactions to determine patterns

The above list may seem simple, as usual, there are hidden complexities. First, grabbing the weblogs for the website can be a daunting task. Because of load balancing, you have to get ALL of the weblogs – it’s not enough to just grab the ones where you think the data might be – a load balancer will scatter user sessions across all of the files. And the size of the logs can be massive – I know some sites that have terabytes of logs per day – compressed. Dealing with a terabyte of data is no small feat.

Next, sorting through the logs can be challenging. Again, because of the load balancers, the transactions for a session are scattered. This means that if you find one part of a session in one file, you’ll need to find the rest of the session, and it could often be in any other file. Once you have all of the transactions for the session you have to reassemble them into the correct order so that they make sense. This is even more challenging when webservers clocks are not synchronized.

Finally, you have all of the sessions for the bad behavior re-assembled, now you have to look through them to determine a fingerprint. What is it about each session that is similar to the other bad sessions, but dissimilar to good activity on the site.

Needless to say, doing all of the above by hand is an extremely time consuming task. The investigation step of addressing a business logic flaw can take DAYS by itself. And, similar to the identification phase, while the investigation is happening customers are continuing to be impacted.