etshx-deepJS

Professional Deep JavaScript Analysis & Secret Extraction Tool

Overview

etshx-deepJS is a professional, multi-mode deep JavaScript analysis tool designed for security researchers, bug bounty hunters, and penetration testers. It crawls web applications, collects JavaScript files (both live and archived via Wayback Machine), and performs deep analysis to extract:

Secrets & Credentials — API keys, access tokens, private keys, database connection strings
API Endpoints — REST paths, GraphQL endpoints, WebSocket URLs, with HTTP method detection
Sensitive URLs — Admin panels, debug endpoints, internal IPs, exposed config files
Security Risks — eval() usage, innerHTML assignments, CORS wildcard origins
Embedded Data — JWT claims (decoded), base64-encoded secrets, hex keys, high-entropy strings

The tool uses 220+ regex patterns organized across 21 categories, combined with a multi-stage validation pipeline that includes Shannon entropy analysis, context-aware confidence scoring, and false-positive filtering to deliver accurate, actionable results.

Features

Analysis Engine

220+ regex patterns covering cloud providers (AWS, GCP, Azure), VCS tokens (GitHub, GitLab), chat platforms (Slack, Discord, Telegram), payment processors (Stripe, Square, PayPal), databases (MongoDB, PostgreSQL, Redis), authentication (JWT, OAuth, bearer tokens), private keys (RSA, EC, OpenSSH, PGP), and 50+ third-party services
Deep structural analysis — parses variable assignments, object properties, function calls, comments, and environment variable references to find secrets that regex alone would miss
JWT decoding — automatically decodes JWT tokens and extracts sensitive claims (passwords, API keys, PII)
Base64 auto-decode — decodes base64 strings and checks for embedded secrets (connection strings, PEM keys, credential patterns)
String concatenation reassembly — reassembles split secrets like "part1" + "part2" and analyzes the combined value
Hex-encoded secret detection — flags hex strings near sensitive context keywords (encrypt, cipher, AES, HMAC, etc.)
High-entropy string detection — catch-all for unknown secret formats using Shannon entropy thresholds
Source map detection — identifies sourceMappingURL references and optionally fetches/analyzes source maps
Webpack bundle detection — identifies webpack bundles, chunk references, and module IDs

Collection Engine

Recursive page crawling — follows links up to configurable depth, extracting <script src> tags and inline JavaScript
Wayback Machine integration — collects archived JS files via the CDX API, useful for finding secrets in older deployments
Subdomain enumeration — passive enumeration via 6 sources: crt.sh, HackerTarget, Wayback Machine, RapidDNS, AlienVault OTX, Anubis; plus optional external tools (subfinder, assetfinder, findomain)
Live subdomain checking — filters subdomains to those with active web servers before crawling
Cross-JS reference extraction — finds webpack chunk references, dynamic import() calls, and require() references within JS files
Rate limiting — configurable per-domain rate limiting to avoid overwhelming targets
Proxy support — full HTTP/HTTPS proxy support for routing through Burp Suite, etc.

Validation & False-Positive Reduction

Shannon entropy scoring — high-entropy values are more likely to be real secrets
Context-aware validation — analyzes surrounding code for confidence-boosting keywords (secret, token, key, password) and confidence-reducing keywords (example, test, placeholder)
Placeholder filtering — filters out known example values (your_api_key, changeme, test123, documentation example keys)
Code construct filtering — filters out JavaScript keywords and syntax that aren't secrets
Library artifact filtering — filters version strings, CSS selectors, MIME types, file extensions, CDN paths, npm package names, color hex codes, etc.
Format validation — pattern-specific format checks for AWS keys, GitHub tokens, Stripe keys, JWTs, Slack tokens, and more
Confidence scoring — each finding gets a numeric confidence score (0.0–1.0) and label (very_high → very_low) based on format validity, entropy, context keywords, and diversity

Reporting

JSON report — full structured report with all findings, endpoints, URLs, metadata, and validation details
CSV reports — separate findings and endpoints CSV files for easy import into spreadsheets or SIEM tools
HTML report — styled, dark-themed interactive HTML report with severity badges, sortable tables, and summary cards
Text report — plain-text report with no ANSI colors, suitable for piping or CI/CD pipelines
Terminal output — colorful, formatted real-time output with progress bars and severity highlighting

Installation

Prerequisites

Python 3.7+
pip package manager

Install Dependencies

pip install -r requirements.txt

The only hard dependency is requests. The tool also uses urllib3 (bundled with requests).

Optional External Tools (for subdomain enumeration)

These are optional but enhance subdomain discovery when using -d (domain enumeration) mode:

# Subfinder (recommended)
go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest

# Assetfinder
go install -v github.com/tomnomnom/assetfinder@latest

# Findomain
# See: https://github.com/findomain/findomain

If none of these are installed, the tool still works using its built-in passive sources (crt.sh, HackerTarget, Wayback, RapidDNS, OTX, Anubis).

Usage

Quick Start

# Analyze a single target URL (crawls for JS files)
python3 etshx-deepJS.py -u https://example.com

# Enumerate subdomains and analyze JS from all of them
python3 etshx-deepJS.py -d example.com

# Analyze subdomains from a file
python3 etshx-deepJS.py -sf subdomains.txt

# Analyze JS/page URLs from a file
python3 etshx-deepJS.py -uf urls.txt

# Analyze a single JS file directly
python3 etshx-deepJS.py -j https://example.com/app.js

All Options

TARGET MODES (choose one):
  -u, --url URL              Single target URL to crawl and analyze
  -d, --domain DOMAIN        Root domain for subdomain enumeration + JS analysis
  -sf, --subdomain-file FILE File containing subdomains (one per line) to crawl for JS
  -uf, --url-file FILE       File containing URLs (JS files or pages) to analyze
  -j, --js URL               Direct JS file URL to analyze

COLLECTION OPTIONS:
  -t, --threads INT          Number of concurrent threads (default: 10)
  -to, --timeout INT         Request timeout in seconds (default: 15)
  --rate-limit FLOAT         Minimum seconds between requests to same domain (default: 0.3)
  --proxy URL                HTTP/HTTPS proxy URL (e.g., http://127.0.0.1:8080)
  --max-depth INT            Maximum crawl depth for page links (default: 2)
  --wayback                  Also collect JS files from Wayback Machine
  --no-ssl-verify            Disable SSL certificate verification

SUBDOMAIN ENUMERATION OPTIONS:
  --sources SRC [SRC ...]    Sources to use: crtsh, hackertarget, wayback, rapiddns, otx, anubis
  --no-external              Don't use external tools (subfinder, assetfinder, findomain)
  --check-live               Check which subdomains have active web servers (default: on)
  --no-check-live            Skip live subdomain check (analyze all subdomains)

ANALYSIS OPTIONS:
  --fetch-source-maps        Fetch and analyze source maps
  --include-filtered         Include false-positive-filtered findings in output
  --min-severity LEVEL       Minimum severity: critical, high, medium, low, info
  --min-confidence LEVEL     Minimum confidence: very_high, high, medium, low, very_low

OUTPUT OPTIONS:
  -o, --output DIR           Output directory for reports (default: ./output/<target>_<timestamp>)
  -f, --format FMT [FMT ..]  Output formats: json, csv, html, txt (default: json csv html)
  --no-color                 Disable colored terminal output
  -q, --quiet                Quiet mode: suppress detailed findings output

Examples

# Full domain scan with Wayback Machine collection and all output formats
python3 etshx-deepJS.py -d example.com --wayback -f json csv html txt

# Route through Burp Suite proxy
python3 etshx-deepJS.py -u https://example.com --proxy http://127.0.0.1:8080 --no-ssl-verify

# Only show critical and high severity findings with high+ confidence
python3 etshx-deepJS.py -d example.com --min-severity high --min-confidence high

# Include false-positive-filtered findings for manual review
python3 etshx-deepJS.py -u https://example.com --include-filtered

# Fetch and analyze source maps for deeper analysis
python3 etshx-deepJS.py -u https://example.com --fetch-source-maps

# Use specific subdomain enumeration sources only
python3 etshx-deepJS.py -d example.com --sources crtsh hackertarget

# Disable live check and analyze all subdomains (faster)
python3 etshx-deepJS.py -d example.com --no-check-live

# Output to a specific directory
python3 etshx-deepJS.py -u https://example.com -o ./my-results

# Quiet mode for CI/CD pipelines
python3 etshx-deepJS.py -j https://example.com/app.js -q -f json -o ./ci-output

Operating Modes

Mode 1: Single Target URL (`-u`)

Crawls a single URL, follows links up to --max-depth depth, collects all <script src> JS files and inline JavaScript, then performs deep analysis. Optionally collects from Wayback Machine with --wayback.

Mode 2: Domain Enumeration (`-d`)

Enumerates subdomains for the given root domain using passive sources (crt.sh, HackerTarget, Wayback, RapidDNS, OTX, Anubis) and optional external tools (subfinder, assetfinder, findomain). Filters to live subdomains, then crawls each for JS files and analyzes them collectively.

Mode 3: Subdomain File (`-sf`)

Reads a list of subdomains from a file (one per line), optionally checks which are live, crawls each for JS files, and analyzes them collectively. Accepts bare domains (sub.example.com) or full URLs (https://sub.example.com).

Mode 4: URL File (`-uf`)

Reads a list of URLs from a file (one per line). URLs ending in .js/.mjs/.jsx are fetched directly as JS files. Other URLs are crawled as pages for JS extraction. Both are analyzed together.

Mode 5: Direct JS URL (`-j`)

Fetches a single JS file URL directly and analyzes it. Fastest mode for analyzing a known JS file.

Pattern Categories

Category	Description	Patterns
Cloud Provider Secrets	AWS, GCP, Azure, DigitalOcean, Heroku, Linode, Cloudflare, Vault	20
Version Control Secrets	GitHub, GitLab, Bitbucket, NPM, DockerHub tokens	17
Chat/Communication Secrets	Slack, Discord, Telegram, Twilio, SendGrid, Mailgun, Mailchimp	20
Payment Secrets	Stripe, Square, PayPal, Coinbase, Braintree	9
Database Secrets	MongoDB, PostgreSQL, MySQL, Redis, MSSQL, Elasticsearch, Firebase, Supabase, Neo4j	17
Authentication Secrets	JWT, OAuth, bearer tokens, API keys, client secrets	9
Private Keys	RSA, EC, DSA, OpenSSH, PGP, PKCS12, JWK	10
Generic Secrets	Passwords, secrets, tokens, keys, credentials in assignments	9
Infrastructure	S3 buckets, GCS, Azure Blob, internal IPs, Docker/K8s secrets, Grafana	13
API Endpoints	fetch, axios, XHR, jQuery, Angular, GraphQL, WebSocket, gRPC, REST paths	20
Sensitive URLs	Admin panels, Jenkins, Grafana, Kibana, Swagger, Actuator, debug, .git, .env	10
Third-Party Service Secrets	Google Maps, Mapbox, Algolia, Sentry, Datadog, Shopify, HubSpot, Airtable, Notion, Vercel, Netlify, Auth0, Okta, Firebase, OneSignal, and 20+ more	40
Debug/Source Map Info	Source map references, webpack, React/Vue DevTools, debugger statements, console logging	6
Sensitive Data	Email addresses, credit cards (Visa/MC/Amex/Discover), SSN, IBAN, encryption keys, OAuth config, CORS, eval, innerHTML, git/FTP/proxy URLs with credentials	20

Project Structure

etshx-deepJS/
├── etshx-deepJS.py        # Main CLI entry point (5 modes, argparse, pipeline orchestration)
├── requirements.txt        # Python dependencies
├── README.md              # This file
├── steps.json             # Build progress tracker
├── core/                  # Core analysis modules
│   ├── __init__.py
│   ├── patterns.py        # 220+ regex patterns (21 categories, severity, confidence, entropy)
│   ├── analyzer.py        # Deep analysis engine (pattern scan, deep analysis, JWT/base64/hex/concat decode, aggregation)
│   ├── validator.py       # False-positive reduction (entropy, context, placeholder/format/library filtering, confidence scoring)
│   ├── collector.py       # JS collection (page crawler, Wayback Machine, JS file fetcher, batch fetch)
│   ├── subdomain.py       # Subdomain enumeration (6 passive sources + 3 external tools, live check)
│   └── reporter.py        # Output reporting (terminal, JSON, CSV, HTML, text)
├── utils/                 # Utility modules
│   ├── __init__.py
│   ├── http.py            # HTTP client (session, retries, rate limiting, proxy, batch fetch)
│   └── file.py            # File I/O (URL/subdomain parsing, CSV/JSON writing, TLD extraction, dedup)
└── output/                # Default output directory for generated reports

Output Formats

Each scan generates reports in the output/ directory (or a custom directory with -o):

File	Format	Description
`report.json`	JSON	Full structured report with findings, endpoints, URLs, metadata, and validation details
`findings.csv`	CSV	Tabular findings (severity, category, pattern, value, source, line, confidence, entropy)
`endpoints.csv`	CSV	Tabular endpoints (URL, method, source, context)
`report.html`	HTML	Styled dark-themed interactive report with summary cards and tables
`report.txt`	Text	Plain-text report with no ANSI colors

How It Works

┌─────────────────────────────────────────────────────────────────┐
│                        etshx-deepJS Pipeline                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  1. COLLECTION                                                   │
│     ┌──────────────┐   ┌──────────────┐   ┌──────────────┐     │
│     │ Crawl Pages  │──▶│ Extract <script src> + inline JS       │
│     │ (recursive)  │   │ Follow links up to max-depth            │
│     └──────────────┘   └──────────────┘   └──────────────┘     │
│     ┌──────────────┐   ┌──────────────┐                         │
│     │ Wayback Mach.│   │ Subdomain    │  (optional, -d mode)    │
│     │ CDX API      │   │ Enumeration  │                         │
│     └──────────────┘   └──────────────┘                         │
│                          │                                       │
│                          ▼                                       │
│  2. FETCH JS FILES (batch, threaded, rate-limited)              │
│                          │                                       │
│                          ▼                                       │
│  3. DEEP ANALYSIS                                                │
│     ┌─────────────────────────────────────────────────────┐     │
│     │ Pattern Scan     — 220+ regex patterns               │     │
│     │ Deep Analysis    — var assignments, obj props,       │     │
│     │                     comments, env vars               │     │
│     │ JWT Decoding     — decode tokens, extract claims     │     │
│     │ Base64 Decoding  — decode + check for secrets        │     │
│     │ String Concat    — reassemble split secrets          │     │
│     │ Hex Detection    — hex keys near sensitive context   │     │
│     │ Entropy Analysis — high-entropy catch-all            │     │
│     │ Endpoint Extract — fetch/axios/XHR/jQuery/GraphQL/WS │     │
│     └─────────────────────────────────────────────────────┘     │
│                          │                                       │
│                          ▼                                       │
│  4. VALIDATION PIPELINE                                          │
│     ┌─────────────────────────────────────────────────────┐     │
│     │ Placeholder Filter  — known example/test values      │     │
│     │ Code Construct Filter — JS keywords, not secrets     │     │
│     │ Library Artifact Filter — versions, CSS, MIME, etc.  │     │
│     │ Format Validation  — pattern-specific format checks  │     │
│     │ Context Analysis   — keyword proximity scoring       │     │
│     │ Entropy Scoring    — Shannon entropy thresholds      │     │
│     │ Confidence Scoring — 0.0–1.0 with label             │     │
│     │ Deduplication      — same value + pattern + source   │     │
│     └─────────────────────────────────────────────────────┘     │
│                          │                                       │
│                          ▼                                       │
│  5. REPORTING (JSON, CSV, HTML, Text, Terminal)                 │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Severity Levels

Level	Description	Examples
Critical	Immediately exploitable secrets	AWS keys, Stripe live keys, private keys, MongoDB URIs, GitHub PATs
High	Likely exploitable secrets	API keys, database URIs, payment keys, Slack/Discord tokens, JWT secrets
Medium	Potentially sensitive info	Endpoints, Firebase URLs, Supabase URLs, analytics keys, webhook URLs
Low	Informational	OAuth client IDs, public keys, debug info, internal IPs, email addresses
Info	Contextual info	Source map references, React/Vue DevTools, webpack artifacts

Confidence Levels

Each finding is assigned a confidence score based on format validation, entropy, context keywords, and diversity:

Label	Score Range	Meaning
very_high	0.80–1.00	Very likely a real secret (valid format, high entropy, sensitive context)
high	0.65–0.79	Likely a real secret
medium	0.45–0.64	Possible secret — manual review recommended
low	0.25–0.44	Unlikely to be a real secret
very_low	0.00–0.24	Almost certainly a false positive

Use --min-severity and --min-confidence to filter output to only the most relevant findings.

Environment Variables

Variable	Description
`NO_COLOR`	Set to `1` to disable colored terminal output (same as `--no-color`)
`ETSHX_DEBUG`	Set to `1` to print full stack traces on errors

Disclaimer

This tool is intended for authorized security testing only. You must have explicit permission to test the targets you scan. The authors are not responsible for any misuse or damage caused by this tool. Always follow responsible disclosure practices when reporting vulnerabilities.

Author

etshx

License

This project is licensed under the MIT License.

etshx-deepJS v1.0.0 :: Deep JavaScript Analysis Tool

Navigation Menu

Repository navigation

About

Repository files navigation

etshx-deepJS

Overview

Features

Analysis Engine

Collection Engine

Validation & False-Positive Reduction

Reporting

Installation

Prerequisites

Install Dependencies

Optional External Tools (for subdomain enumeration)

Usage

Quick Start

All Options

Examples

Operating Modes

Mode 1: Single Target URL (`-u`)

Mode 2: Domain Enumeration (`-d`)

Mode 3: Subdomain File (`-sf`)

Mode 4: URL File (`-uf`)

Mode 5: Direct JS URL (`-j`)

Pattern Categories

Project Structure

Output Formats

How It Works

Severity Levels

Confidence Levels

Environment Variables

Disclaimer

Author

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
__pycache__		__pycache__
core		core
utils		utils
README.md		README.md
etshx-deepJS.py		etshx-deepJS.py
requirements.txt		requirements.txt
steps.json		steps.json

Latest commit

History

Folders and files

Repository files navigation

etshx-deepJS

Overview

Features

Analysis Engine

Collection Engine

Validation & False-Positive Reduction

Reporting

Installation

Prerequisites

Install Dependencies

Optional External Tools (for subdomain enumeration)

Usage

Quick Start

All Options

Examples

Operating Modes

Mode 1: Single Target URL (-u)

Mode 2: Domain Enumeration (-d)

Mode 3: Subdomain File (-sf)

Mode 4: URL File (-uf)

Mode 5: Direct JS URL (-j)

Pattern Categories

Project Structure

Output Formats

How It Works

Severity Levels

Confidence Levels

Environment Variables

Disclaimer

Author

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Mode 1: Single Target URL (`-u`)

Mode 2: Domain Enumeration (`-d`)

Mode 3: Subdomain File (`-sf`)

Mode 4: URL File (`-uf`)

Mode 5: Direct JS URL (`-j`)