Transform Tuesday++ : Facebook,SOA,SPF and Shodan integration!

Hey guys,

Transform Tuesday is here again! Sure it might be a Wednesday, but unfortunately the Fremont data center that two of our Linodes are hosted on went down yesterday (http://status.linode.com/2010/11/fremont-power-issues-rfo.html) !

None the less, there is a small possibility that it's Tuesday somewhere in the world, much like the fabled "it's 5 o'clock somewhere in the world".

This installment has a bunch of new transforms and I'm going to show off some that @achillean did integrating both Shodan and Exploit-db! Our transforms show integration with the Facebook graphAPI as well as some new DNS transforms (SPF and SOA).

Lets get to the good stuff!

Facebook GraphAPI:
I'd like to preface this by saying that we are not trying to break the terms of use of Facebook and we think that we completely abide by the principles listed on http://developers.facebook.com/policy/:

Create a great user experience

• Build social and engaging applications
• Give users choice and control
• Help users share expressive and relevant content

Be trustworthy

• Respect privacy
• Don't mislead, confuse, defraud, or surprise users
• Don't spam - encourage authentic communications

However if we are asked to take down the transforms we will of course.

What are the Facebook transforms?

• toFacebookObject - search Facebook via the graphAPI and return the results. Think of this as your Facebook search engine transform!
• toFacebookAffiliation- convert the above objects to to a Facebook affiliation with the profile picture and a link to the profile.
  
• toPhrase - try and extract the phrase from the Facebook object (what the status message was).
• toPersonFromProfile/toPerson - extracts the person from the Facebook object that made the post so you can use this with the normal people searching transforms.
• toEntitiesNER - take the phrase from the Facebook object and try and extract terms/locations from this.
• toFacebookObjectPerson - simply search Facebook for this person's name.
• toEntitiesNERTwitter - technically not dealing with Facebook, but also allows the same functionality as the above toEntitiesNER transform - but on tweets!

A quick example:
So let's take a look at the phrase 'TSA' on Facebook. Simply drag a phrase onto your graph, set your slider to 255 (3rd notch) and run the 'toFacebookObject' transform. You should see results like this:

Next you can take all of these entities to other entities with the use of Named Entity Recognition (NER) via the 'toEntitiesNER' transform. This will try and extract things like Companies, Locations, and other entity types from the messages. NER is not perfect, as you will notice things like 'Pat Downs' as a person. Keep in mind that NER is very difficult to do! However you can immediately get some results from what is being said in the public Facebook space, such as:


From the screenshot above you can see that the term has connections to the US, deals with two agencies and Mr 'Pat Downs' - someone I think many people can now relate to at this point in time!

Other things you can do is take each of the facebookObjects to Person entities so that you can then perform other searches on these people or identify people commenting a lot on the phrase you originally searched for. To do this you can simply select all the facebookObjects as before and run the 'toPersonFromFacebook' transform:


So why are these Facebook transforms useful:

• Tracking spam: you can use a phrase that you know is used in spam, take this to facebookObjects, then take each of these to a phrase ('toPhrase' transform), and then again search Facebook for these phrases, rinse and repeat until you have identified all the spammers.
• Tracking what is said about a specific term (and who says it the most) as well as how often they are talking about and who they are. You can also identify locations/companies/other useful information by taking these objects and performing Named Entity Recognition on them.
  
• If it was possible to identify friends of an individual (think the typeahead bug) you could identify the spheres of influence around people on Facebook that you have found via your graphAPI queries.

How to get these transforms:

Entity: http://ctas.paterva.com/TDSTransforms/GraphAPI/facebookObject.mtz
Seed: https://cetas.paterva.com/TDS/runner/showseed/SocialMedia


SPF/SOA Transforms:
Recently the topic of spam came up in the office and why SPF(txt) records were never implemented - they seem to be a viable means to stopping spam. We looked at the implementation a bit and noticed some very cool things, such as:

• Admins are lazy and want the ability to move their mail servers around so they give their entire IP range in the SPF records
• SPF records often include other SPF records which show other domains relating to the one you are interested

Secondly a transform RT has always wanted has been one that looks at the SOA records for domains to get the zone's administrative email address and the primary name server (where the zone was created - this is not necessarily one of the current nameservers). These transforms often provides information that's not found in the normal enumeration process.

We have developed two transforms specifically aimed at these:

• DomainToSOAInformation
• DomainToSPFInformation

Hereby some examples of using these transforms:

SPF Transform:
Compare the NS of pentagon.mil (left) to the NS found in the SOA record (right):


SOA Transform:
Quickly and easily identify Google's netblocks from their SPF records:




These transforms have been added to the standard infrastructure seed which can be found at: https://cetas.paterva.com/TDS/runner/showseed/Infrastructure

Shodan:
This week there has been a lot of coverage of the Shodan transforms, developed on the TDS. The transforms essentially allow the integration with the fantastic shodanhq.com as well as exploitdb.com.

The transforms are as follows:

• searchExploitDB - Search the Exploit DB archive's exploit descriptions.
• getHostProfile - Returns the list of banners for the given IPv4 as well as general host information (hostname, location, etc.).
• searchShodanDomain - Search the Shodan database for information on the given domain name.
• searchShodanNetblock -Searches Shodan for hosts contained in the given netblock.
  
• searchShodan - Use the Shodan search engine to locate computers.

Some examples:
Identify hosts belonging to google.com:

• Drag the domain google.com onto the graph and run the searchShodanDomain transform or run the searchShodanNetblock on one of the netblocks found with the SPF transforms (see earlier):
  

• Verify these results by running the getHostProfile on one of the returned IP addresses:

• Search for host responses with the word 'scada' in them by dragging the phrase 'scada' onto the graph and running the 'searchShodan' transform:

• Identify Vulnerabilities that have 'scada' in the name or description by using the same phrase and running the 'searchExploitDB' transform:

Overall these transforms are awesome, and it is great to see people building (and releasing) transforms via the TDS! Hopefully we can see improvements on these such as:

• Ability to search the returned banners against exploitdb
• Ability to search the builtwith.com results against exploitdb
  
• Exploits with a link to where one can find the specific exploit
  

Where can I get the shodan transforms?
The Shodan transforms can be found at http://maltego.shodanhq.com/

Finally, apologies for the Goliath of a blogpost. When I started it this morning it didn't seem like that much, but it's grown quite a bit. Special thanks to the Shodan guys for developing some awesome transforms.

Damn the man. Save the Empire.
-AM

Tweet