Online Website Information Gathering Tools

Regardless of the color of their hats, cybersecurity experts start their works in the same way: "by gathering as much information as they can about the target."

The information collected is used directly in determining how the attack will be carried out and in planning each stage of the attack process. Therefore, the most crucial stage of security tests or a cyber attack is the information gathering stage.

Why Is The Most Important Stage?

We would like to tell you a story about the security test we performed for one of our customers. 

According to the agreement between us,  the only information that we received from our customer was only the domain name of their company. Let's call it s4e-customer.com.

On the first day of the penetration testing, we only collected passive information;

• Learned the IP addresses,
• Examined IP whois information,
• Looked at other websites working on the same IP addresses,
• Reviewed the web archive histories,
• Collected email addresses from open sources related to the s4e-customer.com,
• Examined cached pages from Google,
• Found other sites that using the same Google Analytics code,
• Have listed the employees' social media accounts,
• Again using search engines, we downloaded static files and extracted information from meta tags,
• Detected data leak related to email addresses

We are standing here to keep the list from getting longer, and this is enough for our story.

After having this data, we moved active information gathering stage;

• Scanned the all web urls (crawling)
• Tried to detect subdomains,
• Examined javascript in web applications,
• Scanned TCP ports,
• Scanned UDP ports,
• And more, but again, that's enough for the story.

Now, look at what we are doing by just collecting information. We did not run any exploitation code, detect zero-day vulnerabilities, just gathered data, and the result was:

• First of all, we saw that the remote desktop port (RDP) is accessible for remote management of an IP address belonging to our customer.
• We have seen that the s4e-customer.com page has a 'partners section’ in the web archive history. This information was not available on their up to date  website.
• We determined that an employee in the company providing database support is also a former employee of s4e-customer.com.
• Our customer was using a template for email addresses like {name}. {first later of last name} [email protected]. By using this information, we predicted the old email address (username) for the former employee.
• We searched this email through the deep web, examined the previously leaked passwords related to the email (this is not as complicated as you think, at https://haveibeenpwned.com, you can see if your email and password has been compromised. We access the leaked password information.)
• And we got a password.
• Remember the remote administration interface?
• Afterwards, we tried to connect RDP by using the former employee's predicted username and leaked password.  (name.s,  the password we detected)
• And, we are in.

Isn’t it thrilling?

We just gathered information. There are no unique methods, no advanced techniques. We merely searched for the right thing in the right place and combined the information we got.

Note: Our customer was informed at every stage, and their approval was obtained at the necessary steps.

Note 2: Our marketing experts kill us if we do not write this :)

If your company needs a security test, please contact us. 

Or wait, we will contact you ;) 

Just a joke, click here and fill out the pentest request form.

Here is the some information gathering techniques.

Information Gathering Techniques

The technique of collecting information for an asset (IP address, domain name, website, email, etc.) is classified into two: active information collection and passive information collection.

Passive Information Gathering

All transactions made without leaving a trace on the relevant asset are called passive information collection (whois, website history, searching through leaked data). This type of information collection does not leave any traces on the relevant system, so that it can  not be detected. To illustrate, you can query the whois information of a domain name or IP address from many different places. This query information cannot be detected by the system administrator of the domain name or IP address.

Passive Information Gathering Can Be Processed Through:

• Who-is information
• Search engine results
• Website history
• Searching on data leak
• Links that are available for everyone (documents, files)

Active Information Gathering

The methods that leave traces (port scanning, directory scanning, DNS queries for sub-domain name detection) are known as active scanning. If you want to get information about the ports running on an IP address, you should scan for the open ports. For this, you have to send specific packets to the IP address, which can be detected by those who manage the relevant system.

What is detected here is the port scanning event and scanner IP addresses. The system administrator can only see the port scanning event and the scanner IP addresses. Generally, live systems generate many logs, therefore system administrators can overlook active information gathering scans. Additionally, it can be pretty difficult to distinguish between a genuine user browsing a website and a person accessing the website for information gathering purposes.

Active Information Gathering Can Be Processed Through:

• Port scanning
• Back-end web architecture details
• Directory scanning
• DNS queries
• Un-updated apps' vulnerabilities

Let's end with the following quote from the Chinese Sun Tzu, one of the most famous commanders who ever lived:

"It is said that if you know your enemies and know yourself, you will not be imperiled in a hundred battles; if you do not know your enemies but do know yourself, you will win one and lose one; if you do not know your enemies nor yourself, you will be imperiled in every single battle."

Information Gathering Tools Contains These Checks

• DNS MX Record Lookup

  You must have an MX record to send or receive e-mail from your domain's email addresses. You can check if your MX records are set correctly with this tool.

• DNS NS Record Lookup

  NS or Nameserver records point to a DNS server for domain and subdomains. So those who want to access your domain or subdomains are directed to a DNS server that holds IP-domain matching. Check your NS records with NS Record lookup tool.

• DNS ANY Record Query

  An ANY DNS query is used to get all DNS records available for a specific domain name. Let's check what DNS records are answered to ANY DNS query for your domain. Make an ANY DNS request with this tool.

• DNS TXT Record Lookup

  TXT records are used to keep text values related to your domain name. This type of record usually used for, proof for ownership of domain (google, yandex verification), Sender Policy Framework (SPF) records or DKIM. Check your TXT records with TXT record lookup tool.

• Top 10 UDP Ports Scanner

  You need to know which services and ports are accessible over the internet. Do not forget UDP protoco ! Check your top 10 udp open ports with online port scanning tools.

• DNS Zone Transfer Checker

  DNS servers share zones using AXFR protocol. If it's misconfigured, attackers can get all DNS information related to your domain. You can use Security for Everyone’s DNS vulnerability scanner to scan your DNS zone transfer.

• TCP Full Port Scan

  Use this service if you want to see scan all port. If changes are made regularly on the server, it is a great advantage to use full port tcp scan.

• DNS A Record Lookup

  Any system or anyone who wants to access your domain has to resolve your A records. These records point to IPv4 addresses. You can make an online DNS A record lookup, and check IPv4 address of your domain.

• Allowed HTTP Methods

  You can learn which HTTP methods are used for supporting your website with this tool.

• AAAA Record Lookup Tool

  A DNS AAAA Record (Address for IPv6) records hold IPv6 address or addresses related to your domain. Get your domain IPv6 address with AAAA record lookup tool.

• DNS CNAME Record Lookup

  CNAME (Canonical Name) is a type of DNS record that is used as an alias for another domain.

• Subdomain Finder Online

  Subdomains often address different sections of a website (blog, e-mail, admin panel or another application). Each subdomain could be a new attack vector for you.

• SSL/TLS Supported Cipher

  Check your SSL/TLS configuration for supported ciphers. Do not use weak ciphers. Also learning supported SSL cipher and making cross check with supported ones by security devices can be very important.

• Send Ping Online

  ICMP protocol is used to check whether the system is alive or for debugging. You can send PING packages to the target system by using PING tool.

• Top 10 TCP Ports Scanner

  You need to know which services and ports are accessible over the internet. Because an attacker can easily learn. Check your top 10 tcp open ports with online port scanning tools.

• Technology Identifier

  How much do you think a person can retrieve information about the technologies you use on your website? Your web servers, JavaScript libraries, analytical codes, programming language, operating system are only a few of them.

• Email Harvester

  How much do you think a person can retrieve information about the e-mail accounts related to your company or website? Check our tool for how much information can be obtained about your e-mails.

• Leaked Token-API Key Scanner

  An API key is a unique identifier serves as a authentication token. Attackers can use your leaked API keys by impersonating you and access your private data.

• Backup Files Scanner

  Backup files are critical files generally forgotten in somewhere while in development process. Check your system for backup files which can lead information leaks about your services.

• Asset Blacklist Checker

  Due to some misconfigurations, your asset may not reach the target. Check if your IP address or domain is blacklisted or not.

• Version Control System Scanner

  Version control systems may lead security vulnerabilities. Check if you have one.

• Log File Scanner

  Critical information can be compromised if log files are accessed by anyone.

• Other Files Scanner

  Critical information can be compromised if these files are accessed by anyone.

• Panel Scanner

  Almost every application, user interface comes with a user, admin panel. Sometimes panels includes some security vulnerabilities. Wtih our tool, you can check your publicly available panels.

• API Endpoint Scanner

  API endpoints that do not have an authentication mechanism can cause many private data to be leaked by attackers.

• Config File Scanner

  Critical information can be compromised if config files are accessed by anyone.

• IP Whois Lookup Tool

  Simple and fast IP Whois lookup tool.

• E-mail Service Detector

  Check the email service or spamfilter that is used for a domain.

• Web Application Firewall (WAF) Detection from DNS Records Scanner

  In order to bypass the WAF, it is important to determine the WAF used in the system first.

• Telerik File Upload Detection Scanner

  Attackers can exploit your web application if your Telerik framework File Upload page is accessible to everyone.

• Online Security.txt File Scanner

  When security risks in web services are discovered by independent security researchers, this file defines the channels to disclose them properly.

• Online Robots.txt File Scanner

  Robots.txt file can expose something sensitive such as the path of an administration panel.

• Online Private Key Scanner

  If access permission to the private key file is configured incorrectly, anybody who steals the key can log into everything you have access to.

• Online Trace.axd File Scanner

  ASP.NET's includes a powerful mechanism for detailed request tracing called Trace.axd and it can also be used by attackers to gain information about requests and responses to the application.

• Elmah.axd File Scanner

  If ELMAH is not properly configured elmah.axd file can allows attackers to gain information about the application.

• Domain Whois Lookup Tool

  Simple and fast Domain Whois lookup tool.

• Online Apache Server Status Disclosure Scanner

  It is possible to obtain an overview of the remote Apache web server's activity and performance by requesting the URL '/server-status'.

• Favicon Detection Scanner

  Your favicon should be a visual representation of your website's brand, in order to help users quickly identify your site when they scan through search results.

• Email Extractor

  You can extract emails from your website.

• Web Application Firewall (WAF) Detection Scanner

  In order to bypass the WAF, it is important to determine the WAF used in the system first.