Octoparse Proxy Setup: Custom Lists and Switch Interval

Octoparse has three places that mention proxies, and only two of them touch your scraping traffic.

The login-screen proxy catches people out. The help center says its settings "only apply during the login process" and exist for companies that restrict outbound requests. Putting your scraping proxy there changes nothing about how a task reaches the target site.

Inside Anti-blocking you choose between Use Octoparse proxies, the built-in residential pool billed by traffic, and Use my own proxies, which opens a Configure dialog for your list. The rest of this guide is about the second option, plus the cases where the first one is the only choice.

Octoparse's docs describe the scope of custom proxies as "deliberately narrow", and every item on the list matters:

A few proxy vendors publish Octoparse guides that show a host:port:username:password line. Octoparse's own pages say authenticated proxies "are not currently supported", so we don't build on that format. If you try it anyway and it appears to work, confirm the exit IP before trusting it, because a silently ignored proxy and a working one look identical in the task editor.

With no credentials, the practical answer is IP whitelisting: the proxy provider accepts any connection that comes from an address you've registered. Our explainer on IP whitelisting for proxies covers the trade-offs; for Octoparse it's simply the only route that fits the dialog.

The gateway has to see a connection from a whitelisted address, and that address is your network's public IP, not the 192.168.x.x shown in your Wi-Fi settings.

1. On the computer that will run Octoparse, open https://api.ipify.org in a browser. The number it shows is your public IP.
2. Sign in to your SparkProxy dashboard and add that IP to the whitelist.
3. Turn off any VPN on that computer before you check, or you'll whitelist the VPN's address.

Plan sizes set how many addresses you can register. On 30 October 2026 the sparkproxy.io pricing section listed the entry plan at $75/month (USD, excluding VAT/GST) with 100 threads and 5 whitelisted IPs, rising to 25 whitelisted IPs on the 1,000-thread plan.

What a missing whitelist looks like: we sent requests to gateway.sparkproxy.io from a build server that was not on any whitelist, using curl 8.15.0 on 30 October 2026. Both proxy ports answered the HTTPS request with the same line:

curl: (56) CONNECT tunnel failed, response 407

A plain-HTTP request returned HTTP/1.1 407 Proxy Authentication Required. Inside Octoparse you won't see that text; the page just fails to load. So test from a terminal on the same computer first. Windows 10 and 11 ship curl, so this works in Command Prompt or PowerShell:

curl -sS -x http://gateway.sparkproxy.io:11000 https://api.ipify.org

An IP address that isn't yours means the whitelist works. The 407 line means it doesn't, usually because your public IP changed. Home broadband connections often get a new address after a router restart, and a scheduled overnight run then fails for no visible reason. If that keeps happening, re-check api.ipify.org whenever runs start failing, or run Octoparse from a machine with a fixed address.

To see the rotation behaviour of both ports before you touch Octoparse, this Python script asks each port for its exit IP three times. It sends no credentials, exactly as Octoparse will:

import time

import requests

GATEWAY = "gateway.sparkproxy.io"
PORTS = {"Random Rotation": 11000, "Sticky Sessions": 11002}


def exit_ip(port: int) -> str:
    proxy = f"http://{GATEWAY}:{port}"
    resp = requests.get(
        "https://api.ipify.org?format=json",
        proxies={"http": proxy, "https": proxy},
        timeout=20,
    )
    resp.raise_for_status()
    return resp.json()["ip"]


for label, port in PORTS.items():
    seen = []
    for _ in range(3):
        try:
            seen.append(exit_ip(port))
        except requests.exceptions.ProxyError:
            seen.append("refused: is this computer's public IP whitelisted?")
            break
        time.sleep(2)
    print(f"{label} (port {port}): {seen}")

Expect different addresses on 11000 and one repeated address on 11002. That pattern is the whole basis of the next two sections.

SparkProxy's gateway listens on port 11000 for HTTP/HTTPS with Random Rotation, 11002 for Sticky Sessions, and 13000 for SOCKS5. Octoparse is HTTP-only, so only the first two belong in the dialog.

1. Open the task, then Task Settings > Anti-blocking.
2. Tick Access website via proxies, then Use my own proxies, then Configure.
3. Enter one line: gateway.sparkproxy.io:11000 for independent pages, or gateway.sparkproxy.io:11002 for anything with a login or click-through pagination.
4. Confirm, then click Save in the upper-left corner.
5. Start a Local run against a page that prints your IP, such as https://api.ipify.org, before pointing the task at the real target.

Octoparse documents the format as IP:port and every example it gives is numeric. We have not verified whether the current dialog accepts a hostname. If it rejects gateway.sparkproxy.io, run nslookup gateway.sparkproxy.io in a terminal and paste the address it returns with the same port. We confirmed on 30 October 2026 that the resolved address answers on 11000 and 11002 exactly as the hostname does. The catch is that a pasted address won't follow the gateway if its IP ever changes, so repeat the lookup when connections start failing.

Step 5 is not optional. The Octoparse docs warn that a successful connection test "does not prove the target accepts the IP", and the reverse trap also exists: a typo in the list can leave the task running on your home connection, which is exactly the IP that was blocked.

This is where most Octoparse proxy setups go wrong. There are two independent clocks.

Octoparse's clock. Per its docs, when a task starts Octoparse uses a proxy from your list, and "if you set a rotation interval, it switches proxies at that interval while the task runs. Custom proxies are selected from your list." Switching pauses when the task is paused. The docs don't publish the allowed interval values or their unit, so read them in your own dialog rather than trusting a number from a blog.

The gateway's clock. A rotating gateway is one address that hands out many exit IPs behind it. On port 11000, each new connection can leave from a different IP. On port 11002, Sticky Sessions hold one exit IP for five minutes. Octoparse can't see any of this; to Octoparse your entire pool is a single list entry.

Put together, the outcomes look like this:

The practical rule: with a gateway, the port is your rotation setting and the Octoparse interval is close to irrelevant. Leave it unset if your version allows that, or set it long. Choosing a good interval for a list of static IPs is its own question, worked through in our guide to picking a proxy rotation interval.

One side effect deserves attention. Octoparse documents that "when configured, Octoparse can switch a User-Agent or clear Cookies when the IP changes." We infer from the docs that "the IP changes" means Octoparse switching list entries, since it has no view of the gateway's exits. So on port 11000 the exit IP changes far more often than Octoparse thinks, and its cookie-clearing rule never lines up with the real IP changes.

The symptom is specific, and once you've seen it you'll recognise it. The first page or two extract cleanly. Then one of these happens:

• the browser view shows the site's login page in the middle of the run;
• Click to Paginate lands back on page 1, so rows from page 1 appear again in the export;
• the site shows a verification page or "unusual activity" notice after a click, not on the first load;
• detail pages open, but the content is a different region's version of the page.

All four have the same root: the site tied your session (a login cookie, a search token, a cart, a pagination cursor) to the IP that created it, and the next request arrived from a different IP. Octoparse's own rotation table says it plainly for logged-in pages: keep the IP stable, because "an IP change can invalidate a Cookie session or trigger a security check." Its troubleshooting page lists "IP change" as a check when a login reappears.

Fixes, in the order we'd try them:

1. Move to port 11002. Sticky Sessions keep one exit IP for five minutes, so a login followed by a few pages of pagination happens on one address. This fixes most cases on its own. Background on how sticky routing works is in what a sticky session proxy is.
2. Turn off "clear Cookies when the IP changes" for login tasks. Clearing removes the session, and Octoparse warns that a login-required site may send the task back to its sign-in page.
3. Keep each logged-in stretch under five minutes. A task that paginates for twenty minutes will cross several sticky windows. Split it: one task collects listing URLs, a second task opens those URLs as a URL list. URL-list tasks don't depend on a page cursor, so an IP change between items costs nothing.
4. Use Auto Login cookies. Octoparse (from version 8.6.8) can save the cookies from a sign-in you perform once. When the session drops, refresh the saved cookies instead of building login clicks into the workflow.
5. Use Checkpoint resume after a break. Octoparse 10.2 can resume a supported interrupted local run from saved progress. It validates the checkpoint first and starts fresh if the task changed; scheduled local runs always start fresh.

What a proxy can't fix: an account the site has flagged, a session that expires on a timer regardless of IP, or a site that simply forbids automated access. If the login page returns even on a sticky IP inside a five-minute window, the cause is the session itself, not the network. Our guide to scraping websites behind a login goes deeper on session handling.

Where the task runs decides which IP the target sees, and Octoparse's settings are not shared between the two.

The Public IP Pool is a region picker, not a proxy service. Octoparse's Cloud IP pools page listed US, UK, Germany, Japan, Australia, France, China, Canada, Brazil, South Korea and United Arab Emirates, plus Default, and states that cloud runs "can use different cloud nodes and IPs within the selected region." So a logged-in Cloud task can also meet the mid-task IP problem above, with no sticky port to fall back on.

This explains a common complaint: "it works locally and fails in the cloud". Locally the task ran on your custom proxy or home IP; in the cloud it ran on a shared Octoparse address your list never touched. Test the two environments separately and compare the screenshot and row count, as Octoparse's troubleshooting page recommends.

Work down this list in order. The early checks take seconds and rule out problems no proxy can solve.

Two honest caveats. SparkProxy's pool is datacenter IPs, and some sites block datacenter networks outright; on those, rotation through any datacenter pool won't help, and Octoparse's residential built-in proxies are the tool for the job. And rate limits are about behaviour as much as address. If a site blocks you after twenty fast pages, adding a wait between page loads often does more than another hundred IPs. Our guide on how to avoid getting your proxy blocked covers the non-IP signals.

A custom-proxy task is a local task, so treat the computer as part of the pipeline.

Scheduling. Octoparse supports local schedules for "cases where Cloud Extraction does not work or where the target website requires a local IP address". Set Maximum runtime in whole minutes so a stuck run is stopped before the next one starts; the docs say it stops the task whether it's running or paused. Disable sleep on the computer, and remember the whitelist: if the public IP changes overnight, every scheduled run fails at the first page.

Storage. Local runs save collected data on the computer. Set an export (CSV or a database) at the end of each run so a later re-run doesn't leave you merging partial files by hand.

Concurrency. SparkProxy plans are sized by concurrent threads, with unlimited bandwidth included, so the number that matters is how many connections Octoparse holds open, not gigabytes. A browser opens several connections per page, and Octoparse's Boost mode runs subtasks in parallel. Measure instead of guessing: start your heaviest task, then run this in PowerShell on the same computer.

(Get-NetTCPConnection -RemotePort 11000,11002 -State Established -ErrorAction SilentlyContinue).Count

Keep the peak comfortably under your plan's thread count. Our explainer on concurrent connections in proxies shows how the count works.

Cost. The two proxy routes inside Octoparse bill on different units. Octoparse's built-in proxies cost $3 per GB of page-loading traffic (USD, Octoparse docs, read 30 October 2026), need at least $3 in credits, and are non-refundable; Octoparse's own testing puts 1 GB at roughly 500 pages, which it notes varies with images, scripts and retries. By that estimate, 50,000 pages a month would be about 100 GB, or about $300. SparkProxy starts from $75/month for 100 threads with unlimited bandwidth, so a heavier page doesn't move the bill. There's a 24-hour free trial with no credit card, enough to run the exit-IP script and one real task, and 3, 6 and 12 month terms take 5%, 10% and 15% off. The trade-off is the one in the table above: the flat-rate route only works for Local runs.

When pagination silently resets, the run still "completes" and the export just contains repeats. This short script reads an Octoparse CSV export and reports duplicate rows and the first row where a repeat begins, which is usually where the IP changed.

import csv
import sys
from collections import Counter

path = sys.argv[1]
key = sys.argv[2] if len(sys.argv) > 2 else "Title"

with open(path, newline="", encoding="utf-8-sig") as f:
    rows = list(csv.DictReader(f))

if not rows:
    sys.exit("export is empty")
if key not in rows[0]:
    sys.exit(f"column {key!r} not found; columns are {list(rows[0])}")

values = [row[key].strip() for row in rows]
counts = Counter(v for v in values if v)
repeated = {v: n for v, n in counts.items() if n > 1}
print(f"{len(rows)} rows, {len(counts)} unique {key} values, {len(repeated)} repeated")

first_seen = {}
for i, value in enumerate(values):
    if value and value in first_seen:
        print(f"row {i + 2} repeats row {first_seen[value] + 2}: likely restart point")
        break
    first_seen.setdefault(value, i)

Run it as python check_export.py export.csv "Product URL", naming whichever column should be unique. Row numbers match a spreadsheet, with the header as row 1. If the first repeat lands at a page boundary, look at the run log around that time for a login page or a proxy switch.

If the task must run unattended in the cloud, your own proxy list is off the table. That leaves three paths:

1. Octoparse Cloud with built-in proxies, billed per GB as above.
2. Octoparse Cloud with the Public IP Pool set to a region, when the problem is location rather than blocking.
3. Move that fetch to an API. For jobs that are really "download these URLs", SparkProxy's Scraping API runs the request on its side. Each request is independent: the docs state that "cookies and login state do not persist across separate API requests", so this suits URL lists and numbered page URLs, not click-through pagination behind a login.

A minimal version for numbered pages, using the API's JSON envelope. The envelope's body is base64 and has to be decoded before you look for anything in the HTML:

import base64

import requests

API = "https://scrape.sparkproxy.io/api/v1"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}
LISTING = "https://www.sparkproxy.io/blog?page={page}"


def fetch(page: int) -> str:
    params = {
        "url": LISTING.format(page=page),
        "render_js": "false",
        "proxy_type": "premium",
        "json_response": "true",
    }
    resp = requests.get(API, params=params, headers=HEADERS, timeout=90)
    resp.raise_for_status()
    envelope = resp.json()
    print(page, envelope["status_code"], envelope["credits_used"], "credits")
    return base64.b64decode(envelope["body"]).decode("utf-8", errors="replace")


for page in range(1, 4):
    html = fetch(page)
    if "<article" not in html:
        print(f"page {page}: no articles found, stopping")
        break

Pricing from the API docs: render_js defaults to true and costs 5 credits, so setting it to false makes each call a 1-credit plain fetch; country_code adds 5 credits. New accounts get 1,000 free credits. Whether this beats a managed proxy setup depends on the job, which our comparison of a scraping API versus self-managed proxies works through.