An experimental YaCy fork: better ranking, signed results, peers behind NAT

Why

Measured on the public YaCy network (freeworld) with 16 queries, only 11% of the top 10 results contained all query terms. Looking into the causes:

• Multi-term queries were sent to Solr with minimum match 1 (an OR query), to the own index and to every remote peer.
• Each peer's Solr scores are normalized to its best hit, so a peer that only has one-term matches puts its best one level with other peers' exact matches.
• Japanese and Chinese text is only split at spaces and punctuation, so a whole sentence becomes one "word" and the word index (RWI) cannot find it.
• In a network of new peers, peers younger than 3 days are never asked in a word index search, and small network definitions send remote Solr queries to nobody.
• Results carry no proof of origin. A peer can return any URL with any title for any query, and seeds can be replayed or altered by third parties.
• Peers behind a NAT can only be "junior": they cannot be reached, so what they index is invisible to others.

What changed

Search quality

Trust layer

• Peer identity. Each peer has an Ed25519 key. Its 12-character peer hash is derived from the public key, and the core of its seed (name, ports, key, reachability, declared tags) is signed. Unsigned seeds are rejected by default. A hello challenge proves that the key owner answers at an address.
• Trust lists. Users configure coordinator keys. A coordinator delegates to operators, and operators sign lists of trusted peers with a priority and declared tags such as ads or proxy:<engine>. Lists are versioned instead of expiring and spread from peer to peer.
• Author signatures. The peer that crawls a page signs its URL, title and a Bloom filter of its words. By default a result is shown only if its signature is valid and its author is in the trust set. A peer storing a document for the DHT cannot forge it. It can only withhold it.
• Open mode. Unsigned or untrusted results can be shown, labelled "unverified" and always ranked below verified ones. Results with a forged signature are never shown.

NAT traversal

A sidecar process (Go, go-libp2p) runs next to YaCy with the same key. Behind a NAT it reserves a slot on a circuit relay v2 and announces the circuit address in the signed seed (Reach=relay). Other peers open a local tunnel port to it and use plain HTTP, so YaCy's existing clients work unchanged. By default such peers only answer searches. They do not store DHT data unless they opt in.

See the trust and NAT design for the details.

Results

Two experiments in yacy-lab. Both run closed networks in docker compose, with a deterministic corpus and query set.

Search quality: upstream vs fork, 3 peers each

Each peer crawls one site. Queries go to peer 1 with resource=global, and most relevant pages are on the other peers. The corpus contains two kinds of decoys: pages stuffed with one query term, and thin "tag archive" pages with the whole query in the title. Mean over 11 queries (6 English, 4 Japanese, 1 Chinese), 2 runs with the same result except where noted.

• Keyword-stuffed pages in the top 5 (sum over the 11 queries): upstream 14, fork 0.
• Thin tag archive pages move down but stay in the top 5. Their average rank is 4.5 in the default search and 5.0 with the word index only. With Solr only it goes from 2.0 without the thin page weighting to 3.1 with it. They still rank above relevant pages that lack the query terms in their title. A stronger weighting would move them further down but would also demote legitimate short pages, so the default is kept mild.
• Upstream cannot use other peers' word indexes in a network of new peers, and cannot find Japanese or Chinese words there at all.
• The minimum match default was chosen on this corpus. Whether it is right for the public network still needs to be measured there.

Trust and NAT: 6 fork peers, a relay and a NAT

Three trusted peers, a trusted peer that declares ads, a signed but untrusted peer that crawls spam and plants documents with a borrowed signature, and a peer behind a MASQUERADE router. All 26 checks pass, among them:

• Peer ids derive from the keys. The untrusted peer's spam is not shown by default, not even when a trusted peer holds a copy.
• In open mode the spam appears labelled unverified and below every verified result. The document with a borrowed signature never appears.
• Results of the ads peer carry the tag, and excludeTags=ads removes them.
• The peer behind the NAT cannot be reached directly, but its pages are found through the relay, and it stays connected.
• A new list version spreads by peer exchange alone. Revoking an operator removes its peers' results.

Try it

The demo starts both networks (3 upstream peers, and the fork's trust and NAT setup) on one machine and puts a search page in front of them. You need Docker with about 7 GB of memory.

git clone https://github.com/pad01g/yacy_search_server.git yacy
git clone https://github.com/pad01g/yacy-lab.git
cd yacy
git checkout baseline        && docker build -t yacy-lab/upstream:baseline -f docker/Dockerfile .
git checkout improved-search && docker build -t yacy-lab/fork:latest -f docker/Dockerfile .
docker build -t yacy-lab/sidecar:latest sidecar/
cd ../yacy-lab
docker compose -f compose.demo.yaml -p yacydemo up -d
# open http://localhost:8800 (setup takes about 10 minutes and shows its progress)

The page also lets you change trust: choose which coordinators the searching peer trusts (a second coordinator lists only the spam peer, so trusting it makes the spam "verified"), edit and re-sign the trust list, hand the new version to one peer and watch it spread, and revoke or restore the operator's delegation.

The experiments themselves: docker compose -p yacylab up -d && docker compose -p yacylab run --rm runner (search quality) and docker compose -f compose.trust.yaml -p yacytrust up -d && docker compose -f compose.trust.yaml -p yacytrust run --rm runner (trust and NAT). See the lab README.

For AI agents

An agent can run its own peer and use it as a search tool, without a search API or API key:

• MCP server (io.github.pad01g/yacy-search): docker run -i --rm -e YACY_URL=http://host.docker.internal:8090 -e YACY_ADMIN_PASSWORD=yacy ghcr.io/pad01g/yacy-search-mcp:0.1.0. Tools: search, crawl, index_status, peers, get_ranking_settings, set_ranking_setting, evaluate_ranking, trust_status. It runs next to the agent and talks to the agent's own peer; there is no central service.
• Agent skill: npx skills add pad01g/yacy-lab installs yacy-p2p-search (start a peer, crawl, search, evaluate and tune ranking).
• Trust registry: pad01g/yacy-trust. A merged pull request lists your peer or makes you an operator who vouches for peers.
• Machine-readable summary: llms.txt.

Limits

• Incompatible with the public YaCy network by default (unsigned seeds rejected, CJK word hashes changed). Open mode can search old peers, but only as unverified results.
• Without configured coordinators a peer trusts only its own documents. Closed networks can use trust.signedOnly=true. A public network needs someone to act as coordinator.
• The corpus is synthetic and small (160 pages). It reproduces the weaknesses found on the public network. It does not predict the size of the improvement there.
• The NAT test uses one MASQUERADE router, not real home routers. Hole punching (DCUtR) is enabled but not measured.
• A trusted author can still sign false content (tags and audit are the answer). A storing peer can still withhold results.

Links

• Code: pad01g/yacy_search_server, branch improved-search (changes listed in FORK.md)
• Experiments and demo: pad01g/yacy-lab
• Full design document (Japanese): docs/trust-and-nat.md
• Upstream: yacy/yacy_search_server · YaCy forum