# R4 — Setup and the legal note
**Read-date: 2026-11-10Z.**

## Requirements
`python3`, 3.12 or newer (tested on **3.12.3**). Standard library only — **no pip
install**, no pandas, no requests, no browser. Files live in this folder; nothing
is written next to them (the pack's `selfcheck.sh` sets
`PYTHONDONTWRITEBYTECODE=1` so even `__pycache__` stays out).

## Run it offline (the normal way, and the way that makes no request)
```bash
cd recipes/r4

# a saved HTML page -> CSV on stdout
python3 scrape_listings.py --input fixtures/listings.html --base-url https://giftmart.example

# a browser_collect-style JSON dump -> a CSV file
python3 scrape_listings.py --input fixtures/listings.json --out rows.csv

# a CSV you already have (columns: title,price,url)
python3 scrape_listings.py --input mine.csv --out rows.csv
```
`--base-url` exists because relative links (`/dp/AMK737`) are useless in a
spreadsheet a month later; give the reader the page's address and it joins them.

### Pointing it at your own saved page
The reader finds items by **class name**, so tell it the three names your page
uses. Save the page from your browser (View Source, or DevTools → Copy element),
then:
```bash
python3 scrape_listings.py -i saved.html -f html \
    --item-class product-card --title-class product-title \
    --price-class price--display --item-tag div
```
- `--item-class` marks one listing; a listing nested inside another is folded
  into its parent, never counted twice.
- `--title-class` / `--price-class` name the elements holding those two fields.
  If a class is missing the row is still produced from the anchor text — and if
  no price can be read, the row is **refused and named**, never invented.
- `--format html|json|csv` overrides the guess when a file's name lies about it.
- `--max-rows N` (default 500) caps what gets stored; when it bites you get
  `CAP HIT, more were not stored` on stderr.
- `--default-currency` applies **only** to a price that carries no sign and no
  ISO code, and that assumption is then visible in `price_raw`.

## The live path, and the four things to check before you enable it
```bash
python3 scrape_listings.py --input https://site.example/list \
    --i-have-checked-robots-and-am-authorized
```
Without that flag a `http(s)://` input exits **2** and prints what to check,
**before opening any socket**. Passing it is your assertion that you did this:

1. **robots.txt.** Read `https://site.example/robots.txt`. The script re-reads it
   itself and exits **3** if the path is disallowed for your user-agent — but its
   check is a floor, not a conscience: it covers `User-agent` groups, path
   prefixes, `Allow`/`Disallow` length precedence and `Crawl-delay`. It does not
   know about `Visit-time`, and an absent robots.txt is silence, not permission.
2. **The site's Terms of Service.** This is the clause that actually bites people.
   Many commerce terms forbid "any robot, spider, scraper or other automatic
   device" outright, and some forbid extracting data into another database. That
   is a **contract**, and it applies even where robots.txt says nothing. Read the
   ToS of the site you are actually scraping; if it forbids this, do not pass the
   flag. (In this pack's own research, three well-known platforms' terms forbid
   automated access — which is exactly why the recipes ship file-based.)
3. **Rate.** One request per second minimum, enforced; a longer `Crawl-delay`
   raises the floor. Stay on the site's own peak-hour margins, cache what you
   fetched, and do not loop pagination wider than you need. A slow scraper that
   finishes is cheaper for everyone than a fast one that gets their IP range
   blocked.
4. **Personal data.** A listing page can carry usernames, avatars, review text,
   phone numbers — a shop's "seller info" tile is someone's identity. R4 keeps
   `title / price / url` and nothing else by design, and `--max-rows` limits the
   haul. If you widen it to read reviews or seller names, you are now holding
   personal data, and PIPEDA / GDPR-style duties attach: a lawful basis, a purpose
   you can state, retention you can justify. Ship the CSV, **not** the raw HTML.

Two more things worth saying plainly: a price or a title is data, but the page
around it is someone's copyrighted work — the CSV holds short factual fields and a
hash, not a republication. And a public page is not an invitation: if a site
sells access to its data, buy it.

## Exit codes
| code | meaning |
|---|---|
| 0 | rows written |
| 2 | usage, or a live URL without the flag |
| 3 | robots.txt denied the path |
| 4 | the input could not be read / bad JSON |
| 5 | rows were read but none could be kept (reasons printed) |

## What it does not do
- **No JavaScript.** A page that builds its list in the browser must be saved
  *after* it renders (DevTools → Copy → Copy outerHTML), then read as a file.
  Pretending otherwise would produce an empty CSV and a confident-looking one.
- No pagination loop, no image download, no login, no form filling, no captcha —
  a login wall is a clear "you are not the audience for this", not an obstacle.
- No dedupe and no ranking. That is R5's job, and R5 refuses to rank a row whose
  headline quantity cannot be delivered.

## Verify it yourself
```bash
cd ../..            # the pack root
python3 -m pytest tests/test_r4_scrape.py -q      # 58 passed, all offline
```
The suite monkeypatches `urllib.request.urlopen` to explode, so "offline" is
proven rather than promised.