ScrapeForge Parameters

Complete parameter reference for enterprise web scraping

Enterprise Features

ScrapeForge includes advanced features like JavaScript rendering, residential proxies, and comprehensive data extraction. Credit consumption varies based on features used.

Core Parameters

Essential ScrapeForge Parameters

Parameter	Type	Required	Description
`url`	string	Required	The target URL to scrape. Must be a valid HTTP/HTTPS URL with proper encoding. Example:`"https://example.com/products"`
`render_js`	boolean	Optional	Execute JavaScript on the page using a real browser engine (Chromium). Example:`true`
`wait_for`	string	Optional	CSS selector or XPath to wait for before considering the page loaded. Example:`"#product-list", "//div[@class=\"content\"]"`
`wait_time`	integer	Optional	Maximum time to wait for the wait_for element in seconds. Example:`10`
`user_agent`	string	Optional	Custom User-Agent string to use for the request. Example:`"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"`
`proxy_type`	string	Optional	Type of proxy to use for the request. Example:`"residential", "datacenter", "mobile"`
`proxy_country`	string	Optional	Target country for proxy location using ISO 3166-1 alpha-2 codes. Example:`"US", "GB", "DE"`

Advanced Parameters

Advanced Extraction & Rendering Options

Parameter	Type	Required	Description
`extract_links`	boolean	Optional	Extract all links found on the page with their text and attributes. Example:`true`
`extract_images`	boolean	Optional	Extract all images with their URLs, alt text, and dimensions. Example:`true`
`extract_schema`	boolean	Optional	Extract structured data (JSON-LD, Microdata, RDFa) from the page. Example:`true`
`extract_meta`	boolean	Optional	Extract meta tags, Open Graph, and Twitter Card data. Example:`true`
`custom_headers`	object	Optional	Custom HTTP headers to include with the request. Example:`{"Authorization": "Bearer token", "X-Custom": "value"}`
`cookies`	object	Optional	Cookies to include with the request. Example:`{"session_id": "abc123", "user_pref": "value"}`
`screenshot`	boolean	Optional	Capture a screenshot of the page (requires render_js=true). Example:`true`
`screenshot_options`	object	Optional	Screenshot configuration options. Example:`{"format": "png", "quality": 90, "full_page": true}`

Bulk Scraping Parameters

Bulk Processing Options

Parameter	Type	Required	Description
`urls`	array	Required	Array of URLs to scrape in bulk (max 100 URLs per request). Example:`["https://site1.com", "https://site2.com", "https://site3.com"]`
`concurrent_requests`	integer	Optional	Number of URLs to process simultaneously. Example:`5`
`retry_failed`	boolean	Optional	Automatically retry failed requests with different proxies. Example:`true`
`max_retries`	integer	Optional	Maximum number of retry attempts for failed requests. Example:`3`
`callback_url`	string	Optional	Webhook URL to receive results asynchronously. Example:`"https://your-api.com/webhook"`

Bulk Scraping Limits

Bulk requests are limited to 100 URLs per request. For larger datasets, use multiple requests or contact support for enterprise solutions.

Credit Consumption

Base Costs

Basic scraping:

3 credits

JavaScript rendering:

+5 credits

Residential proxy:

+2 credits

Mobile proxy:

+4 credits

Feature Costs

Link extraction:

+1 credit

Image extraction:

+1 credit

Schema extraction:

+2 credits

Screenshot capture:

+3 credits

Response Format

ScrapeForge Response Fields

Field	Type	Description
`content`	string	The raw HTML content of the scraped page. Example:`"<html><head>...</head><body>...</body></html>"`
`text_content`	string	Plain text content extracted from HTML (JavaScript-rendered if applicable). Example:`"Welcome to our store. Browse our latest products..."`
`links`	array	Array of link objects with URL, text, and attributes (if extract_links=true). Example:`[{"url": "...", "text": "...", "rel": "...", "target": "..."}]`
`images`	array	Array of image objects with src, alt, and dimensions (if extract_images=true). Example:`[{"src": "...", "alt": "...", "width": 800, "height": 600}]`
`schema_data`	array	Structured data found on the page (if extract_schema=true). Example:`[{"@type": "Product", "name": "...", "price": "..."}]`
`meta_data`	object	Meta tags, Open Graph, and Twitter Card data (if extract_meta=true). Example:`{"title": "...", "description": "...", "og:image": "..."}`
`screenshot_url`	string	URL to the captured screenshot (if screenshot=true). Example:`"https://cdn.searchhive.com/screenshots/abc123.png"`
`load_time`	float	Total time taken to load and process the page in seconds. Example:`2.847`
`status_code`	integer	HTTP status code returned by the target server. Example:`200`
`final_url`	string	Final URL after following redirects. Example:`"https://example.com/products"`
`credits_used`	integer	Number of API credits consumed by this request. Example:`7`

Example Requests

Basic Request

Simple scraping without JavaScript rendering for static content:

Basic ScrapeForge Request

curl -X POST https://www.searchhive.dev/api/v1/scrapeforge \
  -H "Authorization": "Bearer: sk_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "render_js": false,
    "extract_links": true
  }'

Advanced Request

Full-featured scraping with all extraction options and JavaScript rendering:

Advanced ScrapeForge Request

Technical Details

JavaScript Engine

• Chromium-based rendering

• Full ES6+ support

• DOM manipulation handling

• AJAX/Fetch request processing

• Custom wait conditions

Proxy Network

• 100M+ residential IPs

• 200+ countries available

• Automatic IP rotation

• High-speed datacenter options

• Mobile carrier networks

Security Features

• TLS 1.3 encryption

• Browser fingerprint masking

• Anti-bot detection bypass

• Request signature hiding

• Behavioral mimicking

Parameter Best Practices

Optimization Tips

Use render_js selectively:

Only enable for dynamic content to save credits and time.

Set appropriate wait_for:

Use specific selectors for critical elements.

Choose right proxy type:

Residential for high-protection sites, datacenter for speed.

Batch similar requests:

Use bulk endpoint for multiple URLs from same domain.

Common Issues

Incorrect wait_for selectors:

Test selectors in browser dev tools first.

Too short wait_time:

Allow sufficient time for slow-loading content.

Missing required headers:

Some sites require specific headers for access.

Invalid custom headers:

Ensure header names and values are properly formatted.