Web scraping integration: watch a page that has no feed

Web scraping is the fallback. No feed, no API, no dedicated integration, and the source still gets connected. You point it at the page you would otherwise check by hand, it watches for new links, and it builds a card for each article it finds.

It is also the integration that needs the most attention over time, for reasons worth understanding before you set one up.

Point it at the right page

The single decision that determines whether this works is which page you watch.

Open the candidate page and ask one question: if I checked this page every morning, would the new links at the top be the ones I want in my app? If yes, that is the URL.

Article card or long post

The card type setting has two options, and the choice is about rights, not about quality.

Article card creates a card with title, description, image and a link to the original. The reader taps through to the publisher. This is the right choice for anything you do not own.

Long post (beta) pulls the article body into a tchop long post, without the site's navigation and footer. The reader never leaves your app.

That second option reproduces someone else's work inside your product. Use it for your own website, for content you commissioned, or where you have written permission. For third-party publishers, use article cards and link out. Automating the transport does not automate the licence.

What the scraper cleans up for you

Three things happen without you configuring them:

• Title suffixes are stripped. Publishers append their own name for search reasons, so raw titles all end the same way. "New shift plan agreed - DER SPIEGEL" becomes "New shift plan agreed", and a mix of ten articles from one source no longer reads like a stutter.

• Logos and favicons are filtered out. When the only image on an article page is the site logo, the card is created without an image rather than with a logo standing in as a photo.

• Already-imported articles are tracked. An article that sits on the section page for four days is imported once, not four times.

Source name is the one label worth setting by hand. Without it, the publisher's own site name is used, falling back to the domain. Override it when the site's own name is inconsistent or unreadable in a card footer.

Expect to maintain this one

Every other integration reads a structured source. This one reads a web page, and web pages change.

• A redesign can change what the scraper reads. The integration switches to Error, or quietly starts producing worse cards.

• Some sites block automated access. That is their decision and not something we can configure around.

• Some pages cannot be read reliably. Heavily scripted pages and pages that require interaction to reveal their links are the usual cases.

• Card quality follows the source. A publisher with proper preview metadata produces clean cards. A publisher without it produces sparse ones, and no setting changes that.

Errors are silent. Nothing pings you. If you depend on a scraped source, put a glance at the integrations list into a weekly routine, and ask us to set up an email alert for the few sources that genuinely matter.

Use RSS instead when you can

If the site publishes a feed, use RSS. Fewer moving parts, less maintenance, no dependency on page structure. Scraping is for the sites that leave you no choice.

Quick test: add /feed to the domain. If something structured comes back, you do not need this page.

When a scraped page breaks

Error right after setup. Usually the URL. Check it is a listing page and not a single article, and that it opens without a cookie wall or a login.

Cards arrive with no image. Either the articles carry no preview image, or the only image found was a logo and was filtered out on purpose.

Cards arrive with the wrong image. Some sites put a promotional banner in the position a preview image normally occupies. Fix the affected cards by hand.

Titles still carry the site name. Suffix cleaning covers the common patterns. An unusual separator can survive it. Edit the card, and tell us the pattern so we can add it.

The mix filled with unrelated articles. The watched page lists more than you thought. Move to a narrower section or tag page. Mix and backfill are fixed after creation, so this means a new integration.

Long post arrived with the body cut short or scrambled. Long post is beta and depends on the article's structure. Switch that source to article cards.

What people ask about scraping

Can I scrape a page behind a paywall or a login? No. Only pages reachable without authentication.

Can I scrape our own intranet? Only if it is reachable from the public internet. An internal system usually needs the tchop API or a workflow tool instead.

How many articles does it take on the first run? As many as your backfill setting finds on the page, from 0 up to 50. It cannot reach past what the page lists.

Does it follow pagination? It reads the page you give it. Point it at the page whose top holds the newest items.

Can I change the card type later? Yes, card type stays editable. Mix and backfill do not.

Is scraping legal? Reading a public page is not the question. Republishing what you find is. Article cards link out and stay on the safe side of that line. Long posts do not, so treat them as a rights decision.

Check three things first

Check three things: the page lists the articles you want and little else, it opens without a login or cookie wall, and the site has no feed you could use instead.

Then see How integrations work for mix, name, backfill and auto-publish, which behave the same across every source type.