Crawlic is a Web recon tool (find temporary files, parse robots.txt, search folders, google dorks and search domains hosted on same server)
Automatic GIT/SVN clone (using dvcs-ripper) if .git or .svn folder if found.
Latest change 21/12/2015 : crawlic.py Fix path errors
Crawlic Helper
Requirements :
+ Python 2.7.x
+ git/svn ripper needs LWP.pm library (Original dvcs ripper: https://github.com/kost/dvcs-ripper)
+ pholcidae library
Configuration :
– Change user-agent : Edit user_agent.lst, one user agent per line
– Change folders to find : Edit folders.lst, one directory per line
– Change files to scan : Edit extensions.lst, one file extension per line
– Change dorks list : Edit dorks.lst, one dork per line
– Change google dorks list : Edit google_dorks, one dork per line, use %s as target url
Installation:
git clone https://github.com/Ganapati/Crawlic cd Crawlic pip install pholcidae ./crawlic.py -h (for helper)
Source: https://github.com/Ganapati