Volatility - cmdscan buggy?

I tweeted last week that I was impressed with a new command in Volatility called cmdscan.  The command is designed to extract command shell history.  I had run it on a variety of new and old RAM dumps and appeared to get slightly random results, often interspersed with obviously correct history.

In my tweet I made the comment that the command was good but a bit buggy. 

An example of my issues are in the image below:-

You can see that the upper part of the results seem to display erroneous results whereas the lower portion is very obviously a series of recovered commands. 

To my pleasant surprise Michael Ligh himself dropped me a line asking for more details which I duly provided only to discover that I should have not been such an ass but have checked the code before making the comment,  Turns out it is doing exactly what it should.  I thank Mike for his gracious response and explanation which I re-print here:-

MHL

...so in short, cmdscan is to consoles as psscan is to pslist. In other words, the consoles plugin (not sure if you tried that one) will find active/running console sessions (like pslist will only find active processes) and not only print command history but full input/output buffers. The cmdscan plugin, on the other hand, will scan through memory using pattern matching and try to brute force with sanity checks etc - the advantage being that it can not only find histories from active/running processes but also closed consoles that have been partially deallocated or overwritten (similar to how psscan carves and finds terminated processes).

If you take a look at the command history structure:

http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/malware/cmdhistory.py#44

You'll see there's a CommandBucket member which is an array of pointers (to command structures). The CommandCount member tells you how many pointers in the CommandBucket are valid. However, if the command history structs belong to closed/terminated processes, then we cannot rely on CommandCount. It could be 0 although there are still valid pointers in the CommandBucket array. Or vice versa - it could be 40 although there are only 10 valid pointers in the array - not even continuous, it could be slots 0, 4, 5, 10, 11, 12, 18, etc.

So cmdscan ignores the CommandCount member and treats CommandBucket as an array of 50 pointers, because 50 is the max history on most systems. If a pointer points to a valid location (i.e. somewhere allocated and not paged) and looks like it might be at lest some unicode characters, then its printed to the terminal.

If you look at your "Screen Shot 2012-09-21 at 15.27.59.png" image, it says CommandCount is 15. You see slots 0-14 are are valid but it goes on to print slot 18, 25, 32, 39, and 46 anyway just in case CommandCount isn't accurate. The consoles command would trust CommandCount and only print slots 0-14.

After looking at the screen shots, I'd say the plugin is working as expected. So if you do get a chance to look over the code, its pretty well commented and you should be able to figure out why it seemed buggy.

Thanks again to MHL and the Volatility team for such a useful toolset.

Advanced Open Source Intelligence Gathering

 The Internet contains a vast amount of information about people that may be of interest to us.  Police and other Agencies may want to know more about a suspect, a company may want to research the background of a senior candidate or understand the ‘exposure’ of their company or key employees.  Much can be gathered if you know how to exploit online resources.

 
This course focuses on the investigators ability to gather information on people, groups or companies from the Internet in a truly advanced manner.  Rather than just using ‘advanced’ Google searches and other web sites we will be leveraging the tools available to look ‘under the surface’ of the internet, accessing data gleaned by understanding database API’s used by the likes of Twitter, Facebook and others, ‘dark net’ data collection methods and other areas rarely taught.  

The 4 day course is completely hands-on and will teach a range of skills from staying anonymous, bouncing data around the world, setting up false online identities, extracting data using API's, using Patervas awesome Maltego and graphing and visualizing data both historical and in real-time.

We have already begun seeding the Internet with the false identities of subjects that we will be investigating on the course.  The final exam will pit your new skills against the online world as you work to discover all you can about a person, their friends and what they are planning to do!

Other Open Source courses are available, but not like this!

The course will include a 6 month license for Maltego Case File, 6 months VPN access, an encrypted 
hard drive, a large number of software tools and course manual.

The 4 day course is £1800 + VAT 

Nick Furneaux (me!) teaches Law Enforcement agencies all over the world and this is the first time that corporate students have been accepted.

To inquire further please contact me here

Syllabus

Day 1

Understanding the law – what can you do?

Setting up your tool kit

            Encryption of data

            To cache or not to cache

            Benefits of using Virtual Machines

            Adding magic to Firefox

Bouncing anonymously round the world – Proxies and VPN’s

Setting up your own false identities

Day 2

Maltego Case File usage

Aggressive searching – only search the part of the web you need to

            Lots of useful sites to bookmark and try

Searching through maps

            Using social media to ‘see’ an area

What can a web site tell us?

            Who owns it and where are they?

            Blowing a web site apart – mapping a web site in real time

            What did it used to say? - Finding deleted data on the Internet

            Finding hidden links

            Finding documents

            The wonder of Metadata!

Maltego V3!

Google Hacking 101

Understanding email – identification and tracking

Day 3

Finding forums, blogs, websites, IRC entries

            Working with IRC clients

Are you or you organization leaking?

            Using your skills to understand your own vulnerabilities

            Checking if hackers have released your/corporate information

Exploiting Social Networking

Mapping Social Networking accounts and followers

Following the network – don’t forget the family!

Extracting data from Twitter via API

Extracting data from Facebook via API

Facebook ‘naughtyness’

Graphing Twitter data LIVE

Day 4

Geo location possibilities (Where are they, or are they where they say they are?)

EXIF data extraction

Plane and Ship mapping

Enumerating Geo-Coordinates using API

Finding people using public records

Being a bit more aggressive to get IP’s

101 Social Networking – why not just call and ask what you want to know!

Final exam – Full online search and enumeration of a named subject.  Course grade based on details located. (Open book)

Course certificated and graded.

TOTAL COST - £1850 + VAT

Firewire fun with Thunderbolt

Since 2006 when Adam Boileau released his research on exploiting machines using Firewire, we have had fun unlocking locked computers and imaging RAM from the same.  With the release of Thunderbolt (TB) I wondered if the same issues surrounding Direct Memory Access (DMA) exists with that implementation.  Turns out it does.  The interesting thing about this is that allowing DMA provides much of the cool functionality that TB provides however this also provides an attack vector for physical access in the same way as Firewire.  As this is due to an implementation in the hardware layer the OS remains blissfully unaware of whats going on.

Reading a blog on the subject this week there were many comments about it being a lame duck attack as physical access is needed, however, many in our community know that gaining access to a machine is often possible.

If a computer is dead then file level access is simple, however a Windows or Mac that is booted but password locked has always been a problem, however with TB now appearing on all Macs this could be rather a useful technique.  It is notable that Lion appears to turn off DMA in certain circumstances but more work needs to be done to understand this fully.

Enter 'Inception', a very nice proof of concept tool from the Break n Enter blog.  Some work has been done in this area and it seems to work pretty well in certain situations. I won't bother re-blogging everything, but I strongly recommend reading the page I linked to above and also the video which shows the extraction of RAM and the pwning of the FileVault password (loving the music too).  Big shout out to them for the tool and the work.

I'll try and spend some time on this in the next few weeks and let you know how I get on.

Skype IP addresses - in the clear

The security forums and blogosphere have been buzzing for the past few days with an 'undocumented feature' of Skype, the ability to discover the internal and external IP addresses of any Skype account currently logged in.  I don't mean people on your buddy list - I mean ANYONE!

Knowledge of this is critical if you use Skype in any situations where your location needs to remain secure or simply if you are interested in personal privacy.

I've tested this and it does what it says on the tin.  I was able to extract the external and internal IP's of a friend in the US to within a few miles of his house, a buddy in Asia to within a few streets and my own to just a few miles down the road.  More concerningly the internal IP combined with the internet facing address provides the basis for a direct probe and then attack of any individual on Skype's global address book.

The details seem to have come initially from Russian hackers and appeared on PasteBin on April 26th but there is a site which will do it all for you.  I won't copy the whole thing as there is a perl script to assist with parsing the logs but here is the gist:-

http://pastebin.com/rBu4jDm8

1. Downloading this patched version of Skype 5.5:
http://skype-open-source.blogspot.com/2012/03/skype55-deobfuscated-released.html

2. Turn on debug-log file creation via adding a few registry keys.
https://github.com/skypeopensource/skypeopensource/wiki/skype-3.x-4.x-5.x-enable-logging

3. Make "add a Skype contact" action, but not send add request, just click on user, to view his vcard(general info about user). This will be enough.

4. Take look in the log of the desired skypename.
The record will be like this for real user ip: -r195.100.213.25:31101
And like this for user internal network card ip: -l172.10.5.17

21:16:45.818 T # 3668 PresenceManager: aїљ noticing skypetestuser1 0x3e54a539a91a19fc-s-s65.55.223.23 :40013-r195 .100.213.25:31101-l172 .10.5.17:22960 23d23109 82f328ff

5. Catch user via whois service.
http://nic.ru/whois/?query=195.100.213.25

This is help you to get info about skype user: City, Country, Internet provider and internal user ip-address. 

I don't want to overstate this, but this is a big deal.

There is also a web site now if you don't want to bother with the log route - http://skype-ip-finder.tk/, just type in your targets Skype name and bingo, the IP's are even helpfully linked to!  If they are not currently online it does not seem to provide the last known address, only if they are currently online.  Please be cautious with this URL, I have not tested it for a browser payload etc and wouldn't be surprised if something nasty awaits!  However, using it on a VM would be advisable.

Also if you are going to try the patched Skype be 'super' cautious and also some users have reported having their Skype accounts terminated.

I appreciate that Skype is both free and P2P meaning that IP's are often visible when in a conversation, file transfer etc but at least you are in a conversation with a 'known' person.  This technique can be used by and against, anyone with a Skype account, regardless of whether they are a buddy.

I hope that Skype take a serious look at this, simply proxying contact requests would likely solve it which wouldn't be awfully hard for them.  I for one really appreciate the Skype service and use it daily, however, I live in nice, reasonably safe England, not one of the many Countries where it is used for secure comms, free from Government intervention.  For them alone, this needs to be solved.

Visualizing Online Investigations - LIVE

This is my 3rd blog post on data visualization, its becoming a bit of a hobby if Im honest.  Its really good fun!  Aside from fun, I am beginning to believe that there is a significant future in enabling investigators and juries alike to be able to ‘see’ data in a way that is meaningful and useful.  In my last post I outlined how Facebook chat was graphed for an abuse case and I had many interesting emails on the subject.

There is a lot of work to do but I decided to move on to a more challenging area, visualizing online data in a LIVE setting.  It seemed that there were 2 areas worth looking at, Twitter and investigating web sites.

For both of the examples below I used the free graphing tool Gephi with a variety of plugins.

Twitter

I'm sure no one reading this needs to have an explanation of Twitter, however, there are areas where an investigator may want to use Twitter to understand how an event was panning out live.  An example would be the Police monitoring the ring leaders of a riot or a journalist looking for the movers and shakers in the development of a news event. 

An example of the latter came up when I was playing early on with live mapping of Twitter feeds.  I had set a filter to intercept all #syria hashtags during the bombardment of the Syrian city of Homs.  As the tweets hit 3000 a pattern began to exist in the spherical graph, a cluster of someone who was a tweeter being heavily retweeted.  Zooming into the graph gave me his username.  A bit of research indicated that this guy was IN homs at the time tweeting what he was seeing in real time.  If I was a journalist, I would be wanting to talk to this guy.

Using Gephi with a plugin written specifically for Twitter data I started working with different filters and displays.  The plugin taps into the global Twitter feed and applies the filter to decide what to capture.  Eventually, I got it sorted and I have posted a slightly less serious example on Youtube with ‘appropriate’ music.  I was working on it when I heard that Whitney Houston had sadly died.  I quickly started a Twitter capture with hashtags associated with the singer and started a video screen capture.  It is fascinating to watch the Tweets arrive and clusters begin to take shape.  Initially the busy tweeters were the news outlets such as CNN, but these were quickly replaced with ‘people’, some of which were very popular to retweet.

This is definitely a capability that many investigators should examine.  Check out the Whitney video or watch it on YouTube - http://www.youtube.com/watch?v=E70smI9hY_I.

Internet Investigations

For any investigator, whether it be Police, Corporate investigator, Social Engineer or Journalist the ability to understand the web presence of their subject can be invaluable.  Being able to simply browse to their targets web site and see what links exist, what services are in use, who handles their credit cards, whether they use analytics, so many different aspects.

Again using Gephi along with an http plugin I set Firefox up to proxy through the plugin and started recording.  Using Firefox I then browsed to the web site of OccupyWallSt.org and navigated through its pages.  The results can be seen (with appropriate music again!) below or at YouTube - http://www.youtube.com/watch?v=oXgEEznpyvg.

Forensic visualization is probably best used to see data in a clearer way from results gleaned from a disk or RAM dump etc.  However, these live feeds provide a fascinating view of the world or an investigation tool that should not be overlooked.

Forensic visualization Part 2 - Court Case

Visualization gone serious

I blogged some weeks back on research I was doing around visualization of forensic data which was well received with some very interesting comments from readers (both of you!).  However, the week after the posting I was asked to be involved in a prosecution of a man who was accused of various forms of grooming, sexual assault, voyeurism etc of several teenage girls in his community centre.

The case has now concluded and the man received 4 years prison, so a good result, however I wont name the case as I refer to the victims and they deserve as much anonymity as possible.

The case revolved around a large amount of Facebook chat between the accused and the girls, and between the girls themselves.  Some of the chat was quite damning and on the face of it, it was clear that he was trying to talk the girls, one in particular, out of coming forward with what had been happening using emotional blackmail.

His defense on the Facebook chats was that the girls had logged in as him and had chats between themselves, implicating him in wrongdoing. 

I was asked to consider the workings of Facebook, could they log in at the same time as him on a different computer, would he have a record on his own machine and what were the ‘relationships’ between the parties involved.

The word, relationships, got me thinking, could we visualize the data to ‘see’ the relationships and would it be easier for a jury to understand and interpret?  Now, it is easy to map out Facebook ‘Friends’, the excellent Facebook Visualizer as well as the Facebook transform in Maltego will help with that task, but that doesn't really help us understand the activity that exists between those people.  Although Im not much of a Facebook user I have load of buddies on Skype but some of them I haven't spoken to in years.  Just because the accused and Girls A,B, and C were on each others Facebook lists and the fact that there was some chat doesn't ‘a relationship make’!

I used IEF 4(Internet Evidence Finder) to carve all the Facebook chats and fragments out of the 4 hard drives, it even did a great job on the accused’s Mac hard drive and I was left with 4 CSV files with thousands and thousands of chats.  Now to make some sense of it.

I tidied up the CSV’s, removing some of the metadata that I didn't need and essentially just left the FROM, TO and the CHAT columns.  Next I imported this data into Maltego as an Edge weighted graph.  I expected this to cluster the chats around the person who made them and it worked better than expected.

Fig 1 shows the recovered chats on the accused’s computer and who he was talking to.  Each orange dot is a person he has chatted with and the surrounding green dots are each individual chat.  The primary cluster, centre left, is the accused with all his chats; being his machine we would expect this to be the largest cluster.  As we can see there are many chats to many different people, however, our eye is quickly drawn to the 2nd largest cluster on the centre right.  This is a person he talks to more than anyone.  Rolling our mouse over the orange dot in the centre of the cluster, surprise, surprise, it is our 13 year old Girl B.  The 3rd largest, at the bottom, is his best friend, but top right, Girl A. 

Fig 1

This graph gives us an excellent tool, aside from just numbers and statistics as to who was important to him in a Facebook setting.  The question, was this just a girl or girls with a crush, that it was one way traffic, is quashed by this graph, Girl B and Girl A are the 1st and 3rd most frequently communicated with persons on his extensive Facebook buddy list.

Encouraged by the success I did the same process on the machine of Girl B.  This time, as there were many different chat partners I also removed the chats that only existed once or twice, the boy at school saying Hi, a friend inviting to a party etc, but which were not repeated with that person.  The results in Fig 2 are fascinating:-

Fig 2

The primary cluster is of course Girl B herself, but no prize for guessing which cluster is the accused??  You’ve got it, the 1st next biggest cluster top left, in fact their chats are almost twice as many as any other person.  Remember we are talking about a teenage girl here with lots of people to chat too and he was chatting with her more than twice as much as her best friends at school.

I then moved on to looking at the relationships with all those involved.  I again used Maltego and imported all the chats from all the machines but removed the actual chat.  This provided a link graph between the Girls and the accused and their friends, also showing connections between those friends.  I will not present that graph as it includes the names of the persons involved but it showed the accused front and centre with chat connections with all the girls involved and showed the connections between those girls and their friends. 

I felt this was very useful to a jury and so included it in my report to the prosecution barrister.  It went on to form part of the jury pack so I can say that my graphs have made it to Court.  Sadly, I was not called to give evidence on this occasion as the defense agreed all our findings and signed a statement to that effect.  Shame really as I was looking forward to presenting this data in open Court and judging the reaction from a jury.  Not that I am expecting wild applause and fist pumping whooping but it would be interesting all the same.

So far I’ve been using Maltego but have been given heads up of other free tools that might do the same job.  The primary tool is Gephi, thanks @danmcquillan for the tip, a superb, free graphing application for Windows or Mac which supports many different output graphs.  So far Im liking it, it takes a little more work pre-application as you need to define your Nodes and Edges for it to successfully graph the links.  I’ve also had problems with the Preview and output elements which keep crashing, I need to pop a message on the forums really.


A Bump on the Node


Just for your information, the visualization industry seems to be dominated by research groups in Universities ‘visualizing’ everything that moves and then posting them on Youtube with no information about how it was done except the message ‘Arn’t we clever!’. 

However, if you want to learn about it you appear to need the brain the size of planet, a doctorate in statistics and a student card.  It is a very difficult area to start learning as a beginner.  For example, search Google for - What are Nodes and Edges.  Go on, try it.  The top link is Wikipedia that presents you with a series of equations that make up graphing theory.  Its a nightmare.

Anyway, for those of you out there with a shriveled 40-something brain like me, a Node is an element such as the person on my graphs and the Edges are the links between them. 

Eg

I am Nick Furneaux.
My friends are Ed, Toby and Chris
I talk to Ed and Toby
I never talk to Chris

The Nodes are:-

Nick
Ed
Toby
Chris

The Edges are:-

Nick - Ed
Nick -Toby

The graph would show links between me and Ed and Toby but Chris would be an unlinked orphan node floating around the graph on his own.  Sorry Chris.

Clear?  Good.

Hear endeth the lesson!

Evidence visualisation

I've been doing a load of research on trying to easily visualize digital forensic data with the hope that patterns, frequencies and clusters would stand out easily.  There are already excellent tools that do a great job for primarily email such as NUIX and Intella, but these are pretty expensive beasts.  You can also look at software such as I2's Analyst Notebook but now we are talking stratospheric money, out of my league.

My mind was focused when a friend at the Met Police introduced me to a new tool call Bulk Extractor from Simson Garfinkle which scans across an image and extracts data strings, very quickly, based on a plugin structure.  I set out to run Bulk Extractor against a RAM image and had tremendous results.  The tool will extract email addresses, URL's, search terms, Credit card numbers, telephone numbers and others, and does so with aplomb.  The tool generates a list of text files which can be analyzed with the Bulk Extractor Viewer. You can run it against disk images, phone memory dumps and RAM. This is great, but when faced with a list of 10,000+ URLS where do you start.  This is where some visualisation help really comes in.

After alot of looking around I came back to a tool I have used many times, Maltego.  Maltego is primarily used for the enumeration of Internet data, connecting IP's, WHOIS, email and domain information to enable the mapping of an online infrastructure.  It also enables the importing and graphing of text/csv files.

I ran Bulk Extractor against an old 512meg RAM dump and amongst other things it extracted URL links between over 3000 IP addresses.  Normally I would move on quietly(!), however, I tidied up the columns in Excel and imported into Maltego, mapping the  URL address columns.  This is what I saw:-

Each little cluster represents URL's linking to a central URL in the hub.  A quick look shows the most popular URL's at the top with many links.  Straight away the list of 3,000 is somewhat more manageable if we are interested in popular links.

Zooming down we see:-

Although a tad tricky to see there are little links between the nodes with URL addresses linking to the primary URL.  We simply draw around a cluster and then we see:-

Although the URLS linking in are hard to see, believe me they are there, showing all the URLS that link to the central Mozilla.org URL.  How cool is that?

Next I thought IP addresses would be fun, except we had over 10000 entries from the one RAM dump.  However, it mapped very well:-

Again there are some very obvious clusters which may be of interest.  Scrolling in we see a very definite structure:-

Scrolling in further we see all the interconnected IP's with a very interesting structure with clusters grouped together into super-clusters.

Further again and we see the individual addresses:-

Now we can see each individual connected IP and their port numbers.  Now Maltego really comes into its own.  We select the centre of the cluster and select the Transform to reverse look up the domain and TLD.  As if by magic the graph redraws this cluster and we get:-

We now can see that all of these IP's are referencing back to Yahoo.com and it is a very popular cluster in the RAM dump.

Being able to 'see' data in this way can help the investigator to quickly zone in on the important areas, seeing, if you like, the wood for the trees.

I'm now doing work on mapping outputs from Volatility and will blog again in a few days.

Cheers

Nick Furneaux