This instrument is used to gather numerous intelligence sources for hosts. Hostintel is written in a modular trend so new intelligence sources could be simply added.
Hosts are recognized by FQDN host identify, Domain, or IP handle. This instrument solely helps IPv4 for the time being. The output is in CSV format and despatched to STDOUT so the information could be saved or piped into one other program. Since the output is in CSV format, spreadsheets corresponding to Excel or database programs will simply be capable of import the information.
I created a brief introduction for this instrument on YouTube: https://youtu.be/aYK0gILDA6w
This works with Python v2, nevertheless it must also work with Python v3. If you discover it doesn’t work with Python v3 please put up a difficulty.
$ python hostintel.py -h utilization: hostintel.py [-h] [-a] [-d] [-v] [-p] [-s] [-c] [-t] [-o] [-i] [-r] ConfigurationFile InputFile Modular utility to search for host intelligence data. Outputs CSV to STDOUT. This utility won't output data till it has completed all of the enter. positional arguments: ConfigurationFile Configuration file InputFile Input file, one host per line (IP, area, or FQDN host identify) optionally available arguments: -h, --help present this assist message and exit -a, --all Perform All Lookups. -d, --dns DNS Lookup. -v, --virustotal VirusTotal Lookup. -p, --passivetotal PassiveTotal Lookup. -s, --shodan Shodan Lookup. -c, --censys Censys Lookup. -t, --threatcrowd ThreatCrowd Lookup. -o, --otx OTX by AlienVault Lookup. -i, --isc Internet Storm Center DShield Lookup. -r, --carriagereturn Use carriage returns with new traces on csv.
First, be certain your configuration file is right in your laptop/set up. Add your API keys and usernames as applicable within the configuration file. Python and Pip are required to run this instrument. There are modules that have to be put in from GitHub, so ensure the git command is obtainable out of your command line. Git is straightforward to put in for any platform. Next, set up the python necessities (run this every time you git pull this repository too):
$ pip set up -r necessities.txt
There have been some issues with the inventory model of Python on Mac OSX (http://stackoverflow.com/questions/31649390/python-requests-ssl-handshake-failure). You could have to put in the safety portion of the requests library with the next command:
$ pip set up requests[security]
Lastly, I’m a fan of virtualenv for Python. To make a custom-made native set up of Python to run this instrument, I like to recommend you learn: http://docs.python-guide.org/en/latest/dev/virtualenvs/
$ python hostintel.py myconfigfile.conf myhosts.txt -a > myoutput.csv
You ought to be capable of import myoutput.csv into any database or spreadsheet program.
Note that relying in your community, your API key limits, and the information you’re trying to find, this script can run for a really very long time! Use every module sparingly! In return for the lengthy wait, you save your self from having to drag this knowledge manually.
There is a few pattern knowledge within the “sampledata” listing. The IPs, domains, and hosts have been picked at random and on no account is supposed to focus on any group or particular person. Running this instrument on the pattern knowledge works within the following means:
Small Hosts List:
$ python hostintel.py native/config.conf sampledata/smalllist.txt -a > sampledata/smalllist.csv *** Processing 126.96.36.199 *** *** Processing 188.8.131.52 *** *** Processing 192.168.1.1 *** *** Processing 10.0.0.1 *** *** Processing google.com *** *** Processing 184.108.40.206 *** *** Writing Output ***
Larger Hosts List:
$ python hostintel.py native/config.conf sampledata/largerlist.txt -a > sampledata/largerlist.csv *** Processing 220.127.116.11 *** *** Processing 18.104.22.168 *** *** Processing 22.214.171.124 *** *** Processing 126.96.36.199 *** *** Processing 188.8.131.52 *** *** Processing 184.108.40.206 *** *** Processing 220.127.116.11 *** *** Processing 18.104.22.168 *** *** Processing 22.214.171.124 *** ... *** Processing 126.96.36.199 *** *** Processing 188.8.131.52 *** *** Processing 184.108.40.206 *** *** Processing 220.127.116.11 *** *** Processing 18.104.22.168 *** *** Processing 22.214.171.124 *** *** Processing 126.96.36.199 *** *** Processing 188.8.131.52 *** *** Writing Output ***
You can get API keys on the websites under in your configuration file.
- GeoLite2 (No community I/O required)
- DNS (Network I/O required)
- VirusTotal (Public API key and community I/O required, throttled when applicable)
- PassiveTotal (API key, username, and community I/O required)
- Shodan (API key and community I/O required)
- Censys (API key, username, and community I/O required)
- ThreatCrowd (Network I/O required, throttled when applicable)
- OTX by AlienVault (API key and community I/O required)
- Internet Storm Center (Network I/O required)
- The GeoIP2 Python library
- The Python DNS library
- The VirusTotal Python library
- The Shodan Python library
- The Censys Python library
- The PassiveTotal Python library
- The ThreatCrowd Python library
- The OTX Python Library
- The Internet Storm Center DShield Python Library
Crude notes can be found here.