agenthub · knowledge.chunk
Knowledge Chunk
Split crawled pages into embedding-sized pieces, each carrying where it came from and a hash of its text.
Why the node earns its place
Embeddings work on passages, not documents. This groups consecutive paragraphs to roughly the target size with no overlap — crude, but bounded and deterministic, which matters more than clever at this stage.
Every chunk carries its source URL and a content hash, and those two are what make re-indexing safe further down the chain.
How it works at run time
- 1Walk each page's paragraphs, filling a chunk until it reaches
target_chars. - 2Attach the source URL and a hash of the text to every chunk.
What you wire
Inputs
- pages *array
Outputs
- chunksarray
Configuration
| Key | Type | Default & options | What it does |
|---|---|---|---|
| target_chars | number | default 1500 | Target chunk size in characters. |
What usually goes wrong
Watch for this
No overlap between chunks means a fact split across a boundary can become hard to retrieve. If recall is poor on long prose, raising target_chars is the lever you have.
Behaviour & provenance