Web Scraping Integration

Watch a page on any website and turn the new articles it lists into cards.

Web scraping is the fallback for sources that have no feed and no dedicated integration. You give it the section page you would check by hand, it watches for new links, and it builds a card for each one.

What it does for you

• Connect sites that offer nothing else. No RSS, no API, no integration. Scraping still works.

• Section-level precision. Point it at a category page rather than a homepage and the mix stays on topic.

• Cleaner cards than a raw link. Title, description, image, author and source name are extracted from the article page, not guessed from the URL.

Settings

Page URL. The section or overview page that lists the articles you want, for example a site's politics section rather than its homepage. Homepages mix everything together and produce a messy feed.

Card type. Two options:

• Article card. Title, description, image and a link to the original. The default and the right choice almost always.

• Long post (beta). Pulls the article body itself into a tchop long post, without the site's navigation and footer. Use only where you have the rights to reproduce the full text.

Source name. Optional override. Without it, the publisher's own site name is used, falling back to the domain. Set it when the site's own name is ugly or inconsistent.

Mix, name, backfill and auto-publish work as described in How integrations work.

What it handles for you

• Titles are cleaned. SEO suffixes such as "- DER SPIEGEL" are stripped, so cards do not all end with the same site name.

• Logos are filtered out. When the only image on a page is the site logo or a favicon, the card is created without an image rather than with a logo as its picture.

• Duplicates are tracked. An article that has already been imported is not imported twice, even if it stays on the page for days.

Limitations

Websites change. A redesign can change the structure the scraper reads. This is the integration most likely to need attention over time.

Not every page is readable. Some sites are built in ways that make reliable extraction hard, and some block automated access outright. When a page cannot be read, the integration reports an error.

Long post is beta, and it is a rights question. Pulling the full body of someone else's article into your app is a copyright decision, not a technical one. For third-party sources, use article cards and link out. Long posts belong to content you own or have written permission for.

Quality varies with the source. Cards are only as good as the metadata the publisher provides. Sites with proper preview metadata produce clean cards, others produce sparse ones.

The usual shared limits apply. Roughly 30 minutes between checks, silent errors, one-time backfill. See the overview page.

Summary

The universal fallback: any page, watched for new articles. Point it at a section page, keep card type on article for anything you do not own, and expect it to need more maintenance than a feed-based source.