agenthub · knowledge.filter
Knowledge Filter
Drop boilerplate before you pay to process it — blog archives, tag pages, pagination, login screens.
Why the node earns its place
Every page that survives this step gets chunked, embedded and stored, and each of those costs something. Filtering on URL shape is the cheapest possible way to remove the pages that are structurally never worth indexing.
Today it is heuristics only. A classifier to catch the noise a regex misses is a later step.
How it works at run time
- 1Test each page's URL against the noise patterns.
- 2Emit the survivors, unchanged.
What you wire
Inputs
- pages *array
Outputs
- pagesarray
Configuration
Nothing to configure — this node has no settings of its own.
What usually goes wrong
Watch for this
It judges URLs, not content. A genuinely useful page living under a blog path is dropped with the archive, so check the surviving list on a first run rather than assuming the crawl was thin.
Behaviour & provenance