OSINT Introduction: Open Source Intelligence Gathering Techniques

Intelligence Without Permission

OSINT is a methodology for gathering intelligence from publicly available sources without requiring authorization from the target. Unlike covert operations that require classified access, OSINT leverages information the target has already made public — websites, social media, public records, technical data, and any other legally accessible sources. This intelligence-driven approach forms the foundation of reconnaissance, threat intelligence, and investigative research.

OSINT Sources

The scope of OSINT sources is vast. Every piece of publicly available information is a potential data point.

Search Engines

Google, Bing, DuckDuckGo, and specialized search engines are the most accessible OSINT tools. Advanced search operators (Google dorking) unlock powerful filtering capabilities:

Combining operators creates powerful queries: site:target.com filetype:xlsx might find leaked spreadsheets, while site:linkedin.com "target company" "software engineer" can identify employees.

Social Media

Social media platforms are rich OSINT sources. LinkedIn reveals organizational structures, employee roles, and technologies used. Twitter (X) can expose real-time information, personal details, and internal discussions. Facebook, Instagram, and Reddit provide additional personal and organizational data.

Tools like Sherlock can search for a username across hundreds of social media platforms simultaneously, revealing accounts linked to the same identity.

WHOIS and DNS Data

WHOIS records contain registration information for domain names: registrant name, email address, phone number, and physical address. While privacy services (WHOIS privacy) mask this information, historical WHOIS data may still reveal details.

DNS enumeration reveals the infrastructure behind a domain. Records like A, MX, TXT, and NS provide IP addresses, mail servers, verification records, and nameservers. DNS dumpster and SecurityTrails offer free DNS intelligence.

Shodan

Shodan is a search engine for internet-connected devices. Unlike Google, which indexes web pages, Shodan indexes banners from services running on devices — web servers, cameras, routers, industrial control systems, and more. A Shodan search can reveal exposed databases, unsecured webcams, and outdated software versions.

Pastebin and Data Leak Sites

Pastebin and similar services are used to share text snippets publicly. Attackers sometimes post stolen data, credentials, or internal documents. Monitoring these sources provides early warning of data breaches.

Essential OSINT Tools

Recon-ng

Recon-ng is a powerful reconnaissance framework with a modular architecture similar to Metasploit. It includes modules for domain enumeration, contact discovery, geolocation, and reporting. Modules can be chained into automated workflows, making Recon-ng efficient for large-scale reconnaissance.

theHarvester

theHarvester specializes in gathering email addresses, subdomains, IPs, and virtual hosts from public sources. It queries search engines, PGP key servers, and SHODAN to build a comprehensive picture of a target's public footprint.

theHarvester -d example.com -b google,linkedin,bing

Maltego

Maltego provides graphical link analysis, displaying relationships between entities — people, domains, IPs, email addresses, and organizations — as an interactive graph. Its transforms automate data collection from various sources, and the visual representation helps investigators identify connections that text-based tools might miss.

Google Dorking

Google dorking uses advanced search operators to find specific information indexed by Google. Common dork queries include:

The Google Hacking Database (GHDB) maintained by Exploit-DB catalogs thousands of dork queries organized by category.

The OSINT Methodology

Effective OSINT follows a structured methodology to ensure thoroughness and avoid missing critical information.

1. Define Objectives

What are you trying to find? Specific objectives might include identifying the technology stack of a target, finding employee email addresses, locating exposed sensitive documents, or mapping network infrastructure.

2. Identify Sources

Based on your objectives, determine which sources are most likely to yield relevant data. For technology stack analysis, use Shodan and BuiltWith. For employee information, use LinkedIn and theHarvester. For leaked credentials, use DeHashed and Have I Been Pwned.

3. Collect Data

Systematically gather data from identified sources. Use automated tools where appropriate, but do not rely on automation alone — manual exploration often reveals unexpected findings.

4. Analyze and Correlate

Raw data is not intelligence until it is analyzed. Correlate data points from different sources. An email address found on LinkedIn combined with a password from a breach database combined with a login portal found by Google dorking becomes actionable intelligence.

5. Report

Document findings clearly with evidence. Include the methodology used, data sources, discovered information, and recommendations. Reports should be factual and avoid speculation.

{@visual osint-investigation-workflow}

Ethical and Legal Considerations

OSINT is legal because it uses publicly available information. However, there are important boundaries:

Building an OSINT Toolkit

A well-organized OSINT toolkit covers multiple data categories:

| Category | Recommended Tools |

|----------|------------------|

| Search | Google, Bing, Yandex, DuckDuckGo |

| DNS | dig, nslookup, SecurityTrails, DNSDumpster |

| Social Media | Sherlock, Twint (Twitter), LinkedIn scraping |

| People | Pipl, Spokeo, DeHashed |

| Technical | Shodan, Censys, BuiltWith, Wappalyzer |

| Frameworks | Recon-ng, Maltego, SpiderFoot |

| Automation | Python (requests, beautifulsoup4, selenium) |

Start with a few tools and master them before expanding. OSINT is a skill that improves with practice — the more investigations you conduct, the better you become at finding hidden information efficiently.

References

{@ref mitre-attack-reconnaissance}

{@ref nist-sp800-115}