GuidesReliabilitySynthetic Monitoring

End-to-End Synthetic Monitoring: HTTP Checks, JS Rendering, and Browser Journeys

DevHelm Team/Jul 18, 2026/17 min read

Synthetic monitoring is simple in theory: run scripted checks against production on a schedule so you learn about broken paths before customers do. In practice, most teams stop at HTTP 200 and call it done. End-to-end monitoring means stacking layers: probe the wire, inspect what the HTML actually contains, and only then escalate to a real browser when the user journey depends on JavaScript.

This guide builds that stack with runnable Python. You will probe five public targets, classify which of them are static HTML versus JS shells, scrape blog titles from a live SSR listing, prove why an empty SPA root fools a status check, and know when to hand off to a browser journey. For the category definition and tool comparison, start with what synthetic monitoring is and the 2026 tools roundup. This page is the how-to.

Prerequisites

Python 3.11+ with a virtualenv
httpx (or requests) and beautifulsoup4 for Layer A/B
Optional: Playwright Python + Chromium for the Layer C snippet (Playwright setup)
No paid accounts. All targets below are public.

python -m venv .venv
# Windows: .venvScriptsactivate
# macOS/Linux: source .venv/bin/activate
pip install httpx beautifulsoup4
# optional later:
# pip install playwright && playwright install chromium

Create a working directory and keep each script as its own file so you can promote them into CI one layer at a time.

The three layers of an end-to-end check

Layer	What it runs	Catches	Misses	Relative cost
A. HTTP / API	GET/POST, assert status, headers, body, latency	Outages, TLS failures, empty/wrong payloads, slow APIs	Dead buttons, broken client JS, post-login UI	Cheap
B. Rendered HTML	Parse HTML after fetch, or after a headless render	Missing listing cards, wrong titles, SPA shells with no content	Multi-step clicks, auth redirects, canvas/WebGL	Medium
C. Browser journey	Headless Chromium through login/checkout	The failures users actually hit	Nothing Layer A/B already covered, if you over-use it	Expensive

End-to-end monitoring (sometimes called e2e monitoring) is not "run Playwright on everything." It is matching the layer to the failure mode. API and uptime checks give you breadth. Browser checks give you depth on the few journeys that pay the bills. Headless browser monitoring is Layer C specifically; this guide starts at Layer A so you do not pay browser prices for problems a JSON assert would catch.

A useful rule of thumb: if a failure would show up in curl with a clear status or body problem, keep it on Layer A. If the failure only appears after the client bundle runs, you need Layer B or C.

Get posts like this in your inbox

Engineering insights on monitoring, incidents, and uptime. Bi-weekly, 2-minute read.

Layer A: Probe five public domains with HTTP

Pick targets that exercise different response shapes. The set below is deliberate:

https://example.com — tiny static HTML document
https://github.com — large application HTML
https://www.ft.com — often bot-gated (403 from datacenter IPs)
https://en.wikipedia.org/wiki/Main_Page — server-rendered encyclopedia HTML
https://httpbin.org/get — JSON API, not a page

# probe_http.py
from __future__ import annotations

import time
from dataclasses import dataclass

import httpx

TARGETS = [
    "https://example.com",
    "https://github.com",
    "https://www.ft.com",
    "https://en.wikipedia.org/wiki/Main_Page",
    "https://httpbin.org/get",
]

HEADERS = {
    "User-Agent": "DevHelmSyntheticGuide/1.0 (+https://devhelm.io/blog)",
    "Accept": "text/html,application/json;q=0.9,*/*;q=0.8",
}


@dataclass
class ProbeResult:
    url: str
    status: int | None
    elapsed_ms: int
    content_type: str
    body_len: int
    error: str | None = None


def probe(url: str, client: httpx.Client) -> ProbeResult:
    started = time.perf_counter()
    try:
        response = client.get(url, follow_redirects=True)
        elapsed_ms = int((time.perf_counter() - started) * 1000)
        return ProbeResult(
            url=url,
            status=response.status_code,
            elapsed_ms=elapsed_ms,
            content_type=response.headers.get("content-type", ""),
            body_len=len(response.content),
        )
    except httpx.HTTPError as exc:
        elapsed_ms = int((time.perf_counter() - started) * 1000)
        return ProbeResult(
            url=url,
            status=None,
            elapsed_ms=elapsed_ms,
            content_type="",
            body_len=0,
            error=type(exc).__name__,
        )


def main() -> None:
    with httpx.Client(headers=HEADERS, timeout=20.0) as client:
        for url in TARGETS:
            result = probe(url, client)
            print(
                f"{result.status!s:>4}  {result.elapsed_ms:>5}ms  "
                f"{result.body_len:>7}B  {result.content_type[:40]:<40}  "
                f"{result.error or ''}  {result.url}"
            )


if __name__ == "__main__":
    main()

Run it:

python probe_http.py

A representative run from typical cloud or datacenter egress looks like this (numbers move; the shapes matter):

 200    120ms      648B  text/html                                https://example.com
 200    190ms   500000B+ text/html; charset=utf-8                 https://github.com
 403     80ms    12000B  text/html                                https://www.ft.com
 200    240ms   100000B+ text/html; charset=UTF-8                 https://en.wikipedia.org/wiki/Main_Page
 200    500ms      300B  application/json                         https://httpbin.org/get

Three lessons show up immediately.

Status code is not the product. example.com returning 200 with a few hundred bytes of HTML is fine. httpbin.org/get returning 200 with JSON is fine only if you assert a field (for example "url" present). GitHub returning 200 does not mean your authenticated dashboard works.

Bot and WAF responses are first-class signals. ft.com returning 403 from many server IPs is not "the Financial Times is down." It is "this probe path is blocked." Synthetic monitoring from datacenters will see this class of result. Treat 403 and 429 as classified outcomes (expected for that target, or a CDN misconfig on your site), not as a binary up/down.

Latency budgets belong on the probe. A 200 that takes 8 seconds is an incident for many APIs. Record elapsed_ms next to status from day one. Persist the metric even when the assert passes so you can spot slow regressions before they turn into timeouts.

Assert on the body, not only the status

def assert_httpbin(response: httpx.Response) -> None:
    assert response.status_code == 200
    assert "application/json" in response.headers.get("content-type", "")
    payload = response.json()
    assert payload.get("url") == "https://httpbin.org/get"
    assert "headers" in payload
    assert response.elapsed.total_seconds() < 3.0


def assert_example(response: httpx.Response) -> None:
    assert response.status_code == 200
    text = response.text
    assert "Example Domain" in text
    assert len(text) > 100

Wire those asserts into the probe loop so a wrong body fails the process:

ASSERTIONS = {
    "https://example.com": assert_example,
    "https://httpbin.org/get": assert_httpbin,
}

with httpx.Client(headers=HEADERS, timeout=20.0) as client:
    for url, assert_fn in ASSERTIONS.items():
        response = client.get(url, follow_redirects=True)
        assert_fn(response)
        print(f"OK  {url}")

This is API synthetic monitoring in miniature: request, assert structure, fail closed. For how API testing in CI differs from production API monitoring, see API testing vs API monitoring.

Multi-step API checks (still Layer A)

Many "user journeys" are really sequences of HTTP calls. You can cover them without a browser:

def assert_httpbin_flow(client: httpx.Client) -> None:
    # 1) Read a baseline payload
    get_resp = client.get("https://httpbin.org/get")
    assert get_resp.status_code == 200
    assert get_resp.json()["url"].endswith("/get")

    # 2) Echo a POST body (stand-in for "create resource")
    post_resp = client.post(
        "https://httpbin.org/post",
        json={"order_id": "synth-1", "amount": 19.0},
    )
    assert post_resp.status_code == 200
    echoed = post_resp.json()["json"]
    assert echoed["order_id"] == "synth-1"
    assert echoed["amount"] == 19.0

Against your own API the pattern is the same: authenticate, create, read back, delete, assert each step. That is end-to-end service monitoring at the protocol layer. Save the browser for steps that only exist in the DOM.

Detect JavaScript-rendered apps from the HTTP response

Many modern apps return HTTP 200 with almost no user-visible content until JavaScript runs. If your only check is status code, that shell looks healthy forever.

# classify_html.py
from __future__ import annotations

import re
from dataclasses import dataclass

import httpx
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "DevHelmSyntheticGuide/1.0 (+https://devhelm.io/blog)",
}


@dataclass
class PageClass:
    url: str
    status: int
    kind: str  # static | spa_shell | api_json | blocked | unknown
    reasons: list[str]


def classify(url: str, client: httpx.Client) -> PageClass:
    response = client.get(url, follow_redirects=True)
    ctype = response.headers.get("content-type", "")
    reasons: list[str] = []

    if response.status_code in {401, 403, 429}:
        return PageClass(url, response.status_code, "blocked", ["non-2xx access control"])

    if "application/json" in ctype:
        return PageClass(url, response.status_code, "api_json", ["content-type json"])

    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "noscript"]):
        tag.decompose()
    visible = re.sub(r"s+", " ", soup.get_text(" ", strip=True))

    root = soup.find(id="root") or soup.find(id="app") or soup.find(id="__next")
    if root is not None and len(visible) < 80:
        reasons.append("mount point with almost no text")
        return PageClass(url, response.status_code, "spa_shell", reasons)

    if len(visible) < 80:
        reasons.append(f"visible text only {len(visible)} chars")
        return PageClass(url, response.status_code, "spa_shell", reasons)

    if "enable javascript" in visible.lower():
        reasons.append("enable-javascript message")
        return PageClass(url, response.status_code, "spa_shell", reasons)

    reasons.append(f"visible text ~{len(visible)} chars")
    return PageClass(url, response.status_code, "static", reasons)


URLS = [
    "https://example.com",
    "https://en.wikipedia.org/wiki/Main_Page",
    "https://httpbin.org/get",
    "https://www.ft.com",
    "https://app.devhelm.io/login",
]

if __name__ == "__main__":
    with httpx.Client(headers=HEADERS, timeout=20.0) as client:
        for url in URLS:
            result = classify(url, client)
            print(
                f"{result.kind:<10} {result.status}  {result.url}  "
                f"({'; '.join(result.reasons)})"
            )

Expected shape of the output:

static     200  https://example.com  (visible text ~…)
static     200  https://en.wikipedia.org/wiki/Main_Page  (visible text ~…)
api_json   200  https://httpbin.org/get  (content-type json)
blocked    403  https://www.ft.com  (non-2xx access control)
spa_shell  200  https://app.devhelm.io/login  (mount point with almost no text)

https://app.devhelm.io/login is the clean teaching case: status 200, an empty #root, and effectively zero visible text until the SPA boots. An HTTP-only uptime check on that URL will stay green while the login UI is broken. That gap is why browser synthetic monitoring exists.

Heuristics are not perfect. SSR frameworks (including Next.js) often ship full HTML in the first response even when they hydrate on the client. Classification tells you whether this URL is safe to trust from Layer A, not whether the product as a whole is an SPA.

When you classify a URL as spa_shell, do not "fix" the HTTP check by asserting on a script filename. Script tags can remain while the UI is blank. Move that URL's critical asserts to Layer C.

Layer B: Scrape blog titles from `https://devhelm.io/blog`

The DevHelm blog listing is server-rendered. Post titles appear as real <h2> elements inside links to /blog/<slug> in the first HTML response. That makes it a good Layer B exercise: prove your parser reads production content, not that the CDN returned a shell.

# scrape_blog.py
from __future__ import annotations

import httpx
from bs4 import BeautifulSoup

BLOG_URL = "https://devhelm.io/blog"
HEADERS = {
    "User-Agent": "DevHelmSyntheticGuide/1.0 (+https://devhelm.io/blog)",
}
MIN_POSTS = 15


def scrape_titles(html: str) -> list[str]:
    soup = BeautifulSoup(html, "html.parser")
    titles: list[str] = []
    for link in soup.select('a[href^="/blog/"]'):
        heading = link.find("h2")
        if heading is None:
            continue
        text = heading.get_text(strip=True)
        href = link.get("href") or ""
        # Skip nav-only /blog links without a slug
        if href.rstrip("/") == "/blog" or not text:
            continue
        titles.append(text)
    # de-dupe while preserving order
    seen: set[str] = set()
    unique: list[str] = []
    for title in titles:
        if title in seen:
            continue
        seen.add(title)
        unique.append(title)
    return unique


def main() -> None:
    with httpx.Client(headers=HEADERS, timeout=20.0) as client:
        response = client.get(BLOG_URL, follow_redirects=True)
    assert response.status_code == 200, response.status_code
    titles = scrape_titles(response.text)
    print(f"Blog posts found: {len(titles)}")
    for title in titles[:5]:
        print(f"  - {title}")
    assert len(titles) >= MIN_POSTS, f"expected >= {MIN_POSTS}, got {len(titles)}"
    print(f"OK: {len(titles)} >= {MIN_POSTS}")


if __name__ == "__main__":
    main()

python scrape_blog.py

You should see a count around 20 (the listing grows over time; the floor assert keeps the check stable) and the first few titles printed. That is content assertion: the page is not merely reachable, it still exposes the listing cards you care about.

Use a floor (>= 15), not an exact count. Exact equality breaks the first time you publish a new post. Floors catch "the listing disappeared" without paging you for growth.

Contrast: HTTP scrape versus browser render

For devhelm.io/blog, HTTP is enough because the titles are in the initial HTML. For app.devhelm.io/login, the same BeautifulSoup pass finds no usable login copy inside #root. The fix is not a smarter regex. The fix is Layer C: load the page in a browser, wait for the client render, then assert.

# render_spa_title.py  (optional; requires Playwright)
from playwright.sync_api import sync_playwright

URL = "https://app.devhelm.io/login"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(URL, wait_until="domcontentloaded")
    page.locator("body").wait_for(state="visible")
    # Prefer a role/label once the Auth0 (or app) UI paints
    page.get_by_label("Email address").wait_for(state="visible", timeout=30_000)
    print("rendered login email field: OK")
    print("title after render:", page.title())
    browser.close()

Keep this snippet short on purpose. Locators, codegen, traces, and a full Python Playwright path live in What Is Playwright? A Practical Guide. Turning an existing journey into a scheduled production check is covered in Playwright monitoring and browser monitoring with Playwright.

If you want a second render target that is not your own login page, https://demo.playwright.dev/todomvc is a public TodoMVC demo. The raw HTML is thin; after a browser load you can assert on the todo input. Same lesson, different site.

What to assert (beyond status codes)

End-to-end service monitoring fails when assertions are lazy. Use a short checklist per layer:

Layer A (HTTP/API)

Status in an allow-list (200, or 401 if you are probing that auth is required)
Content-Type matches the contract
JSON path or HTML substring that proves the payload is the real one
Latency under a budget (a goal you measure, not a hope)

Layer B (rendered listing / SSR page)

Element count floors (>= 15 posts), not exact equality that breaks on every publish
Critical strings (product name, pricing number, "Order confirmed")
Canonical URL after redirects

Layer C (browser)

Visible control the user needs (email field, Pay button)
Post-action URL or heading
Screenshot or trace on failure for forensics

Also decide what not to alert on. A third-party 403 you classified as blocked should not wake the on-call. A 403 on your own checkout host should. Classification turns raw status codes into actionable signals.

Intervals, flaky selectors, and alert routing are a discipline of their own. Read synthetic monitoring best practices before you page humans on every blip.

Layer C: When you need a real browser journey

Escalate to a browser when the failure mode is client-side:

Login and SSO redirects
Checkout and payment iframes
Multi-step wizards that never hit a single JSON endpoint you own
"Button does nothing" bugs behind a green API

That class of work is synthetic transaction monitoring: a multi-step script that mimics a revenue or auth path. It is also where cost and maintenance jump. Prefer one solid checkout check over twenty brittle page opens.

You do not need a browser to know httpbin.org/get returned valid JSON. You do need a browser to know the login form on an empty #root SPA still accepts an email. Match the tool to the risk.

A practical promotion path for most teams:

Write the HTTP asserts for the APIs behind the journey.
Add an SSR/content scrape for any public listing that must stay populated.
Add one browser script for the money or auth path.
Schedule all three; alert on severity, not on every layer equally.

Synthetic monitoring and real-user monitoring answer different questions; run both when you can. The split is explained in synthetic monitoring vs RUM.

Put the layers in one runner

Once the pieces work, a single entrypoint makes CI and cron easier:

# main.py
from __future__ import annotations

import httpx

from classify_html import classify
from scrape_blog import scrape_titles, BLOG_URL, MIN_POSTS, HEADERS

CRITICAL_HTTP = [
    "https://example.com",
    "https://httpbin.org/get",
    "https://devhelm.io/blog",
]


def main() -> None:
    with httpx.Client(headers=HEADERS, timeout=20.0) as client:
        for url in CRITICAL_HTTP:
            response = client.get(url, follow_redirects=True)
            assert response.status_code == 200, f"{url} -> {response.status_code}"
            print(f"HTTP OK  {url}")

        blog = client.get(BLOG_URL, follow_redirects=True)
        titles = scrape_titles(blog.text)
        assert len(titles) >= MIN_POSTS
        print(f"Blog OK  {len(titles)} titles")

        spa = classify("https://app.devhelm.io/login", client)
        assert spa.kind == "spa_shell"
        print(f"Classified login as {spa.kind} (browser layer required)")


if __name__ == "__main__":
    main()

The last assert is intentional: you are documenting that login is not covered by HTTP. The runner stays green only if you still recognize the SPA shell. Separately, schedule the Playwright login check so the UI itself is watched.

Schedule the stack

A script on a laptop is a prototype. End-to-end monitoring means the same asserts run on a schedule, from somewhere outside your VPC, with an alert when they fail.

Minimal path:

Keep the probe and scrape scripts in git.
Run them in CI on a cron (schedule: in GitHub Actions) or a system timer.
Exit non-zero on assert failure so the job is red.
Wire the failure to Slack, PagerDuty, or email.

# .github/workflows/synthetic-http.yml
name: Synthetic HTTP probes
on:
  schedule:
    - cron: "*/15 * * * *"
  workflow_dispatch:
jobs:
  probe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install httpx beautifulsoup4
      - run: python main.py

That gets you Layer A/B from one region every 15 minutes. It does not replace multi-region product monitoring, private locations, or rich failure artifacts. Failures that only happen in one geography (bad DNS, expired edge cert, partial deploy) need probes in more than one place.

When cron is no longer enough, compare platforms in best synthetic monitoring tools and keep definitions in git via monitoring as code.

FAQ

What is end-to-end monitoring?

End-to-end monitoring (e2e monitoring) means verifying a path the way a user or dependent system experiences it: not only that a process is up, but that the request, response, and (when needed) rendered UI still satisfy the contract. In web systems that usually means HTTP asserts plus selective browser journeys.

Is synthetic monitoring the same thing?

Synthetic monitoring is the broader practice of generating scripted traffic on a schedule. End-to-end monitoring is the subset that follows a full path. A single /health ping is synthetic. A login-plus-dashboard check is synthetic and end-to-end.

Do I need Playwright for every check?

No. Use HTTP and API checks for breadth. Use a browser when the page is a JS shell or the journey is interactive. Over-using browsers creates flaky alerts and large bills.

Why did `ft.com` return 403?

Many publishers block datacenter and bot User-Agents. Your synthetic probes will see 403 and 429 on third-party sites. On your product, unexpected 403s are incidents; on someone else's site used as a teaching target, they are classification data.

How is this different from the Playwright guide?

The Playwright guide teaches the browser toolkit: Page API, locators, waits, codegen, login flows, CI. This guide teaches the monitoring stack around that toolkit: when HTTP is enough, how to detect JS shells, how to assert on SSR content, and when to promote a check to a browser. Read them as a pair.

What to do next

Put Layer A asserts on your real APIs and public pages (status, body, latency).
Classify each critical URL as static, API, or SPA shell before you buy browser minutes.
Add one Layer C journey for login or checkout; write it properly with the Playwright guide, then schedule it with Playwright monitoring.
Run multi-region HTTP and API monitors with body assertions at app.devhelm.io. First monitor is live in about a minute, no credit card.