Trust and NAT design

1. Goals and threat model

Out of scope: storage peers withholding results, false content signed by a trusted author (handled by tags and audit), query privacy.

2. Identity

• At start a peer reads DATA/SETTINGS/peer.key (Ed25519, PKCS#8 PEM) and creates it if missing.
• Peer hash (12 characters) = the first 12 characters of the URL-safe base64 of SHA-256(public key). Upstream's choice of ids to fill gaps in the DHT is dropped: the key decides.
• The same key is the libp2p key of the NAT sidecar. The libp2p peer id (12D3KooW…) is computed from the public key and is not stored in the seed.

2.1 Signed seeds

Receivers rewrite parts of a seed (IP, type, flags, last seen) and relay it. Only the fields that the owner alone decides are signed:

• Sig = Ed25519 over "yacy-seed-v1\n" + "Hash=<hash>\n" followed by the other core fields that have a value, as k=v\n in key order (BDate, Name, P2PA, PK, Port, PortSSL, RDS, Reach, SigT, Tags).
• A seed with PK must satisfy Hash == H(PK) and carry a valid Sig, or it is always rejected. Seeds without PK are rejected unless trust.seed.acceptUnsigned=true.
• A peer signs its seed again when the core changes or the last signature is 7 days old. Seeds whose SigT is more than 1 hour in the future are rejected, and for peers not yet known also those older than 30 days. Signed values containing control characters (e.g. a newline that would make one field look like two) are invalid. A known signed seed is never replaced by an older or unsigned one.

2.2 Hello challenge

The IP address is not signed, so hello checks that the key owner really answers at that address:

• The caller sends a random challenge. The responder returns the address it saw the request come from (challengeFor, or p2p through the sidecar) and Sign("yacy-hello-v2|" + challenge + "|" + own hash + "|" + challengeFor).
• The caller verifies the signature with the PK of the returned seed and checks that challengeFor is its own address. Binding the address defeats an attacker who forwards the challenge to the real peer and returns its answer: the real peer signs "a request from the attacker".
• The back-ping (the responder calling the caller back to test reachability) uses the same challenge, so a seed that claims someone else's IP is rejected.
• The caller's own addresses are its interface addresses, a configured static IP, and the addresses that peers with three different keys, reached in at least two different networks (/24, /48 for IPv6, the full address for private ranges), reported within six hours (OwnAddresses, both address families). Never the IP of its own seed: the unsigned yourip of a single answer sets it, so a relaying attacker could make its own address look like ours. An answer is judged once, before its yourip is applied.
• If challengeFor is not one of the caller's own addresses (a peer behind a NAT, just started, or whose address changed), the caller cannot tell a relayed answer from its own. The contact still counts, but proves no address: the peer's known IP is kept, and its answers are not treated as coming from the key owner at that address (their unsigned snippets are dropped). The back-ping counts only answers bound to the caller's address (unsigned seeds, accepted with trust.seed.acceptUnsigned=true, pass as before: they cannot prove anything).
• A known signed peer changes its address only through a hello with back-ping. Seeds attached to search requests (/yacy/search) can be sent by anybody and do not move it.

3. Delegation and trust lists

coordinator key (configured by the user; several allowed, first has priority)
  └─ delegation: "this operator key may publish the list for this network"   (versioned; revocation is a newer version)
       └─ peer list: "trust these peers, with this priority and these tags"  (signed by the operator; versioned)

• One level of delegation. A coordinator may be its own operator.
• No expiry, only versions. If the operators disappear, the last version keeps working.
• Everything travels in one envelope format. The signature covers the payload bytes, so no JSON canonicalization is needed:

{"payload": "<base64url(JSON bytes)>", "signer": "<base64url public key>", "sig": "<base64url signature>"}

{"type": "yacy-delegation-v1", "network": "freeworld", "operator": "<pk>", "version": 3, "revoked": false}
{"type": "yacy-peerlist-v1", "network": "freeworld", "version": 12,
 "peers": [{"pk": "<pk>", "priority": 100, "tags": ["ads"]}]}

• Delegations must be signed by a coordinator. The highest version per (coordinator, operator) wins, and revoked:true invalidates that operator's lists. Lists must be signed by an operator with a valid delegation, and the highest version per operator wins. network must equal network.unit.name or be *.
• The effective trust set layers the lists in coordinator order. A peer keeps the priority and tags of the first list that names it. Matching uses the full public key, not only the 72-bit peer hash.
• priority is 0–100, and result scores are multiplied by 0.5 + 0.5 × priority / 100.
• Without coordinators a peer trusts only its own documents (fail closed). Closed networks where everybody knows everybody can set trust.signedOnly=true to trust every signed peer. A distribution would ship its coordinator key as the default; this fork ships none.

3.1 Declared tags

A small common vocabulary. Coordinators may add their own tags with an x-<name>: prefix.

Tags on the list are authoritative. Tags in the seed are self-declared and only displayed. trust.policy.excludeTags drops documents of authors with those tags, and results carry their author's tags.

4. Distributing lists

• Each peer verifies the envelopes it receives, stores them in DATA/SETTINGS/trust-bundle.json and serves them at /yacy/trust.json. It keeps only envelopes of its configured coordinators and their operators, for its own network; a peer without coordinators keeps and relays nothing.
• The seed carries TV: per coordinator, two sums of versions: of what the coordinator signed itself (delegations and its own list), and of its operators' lists (revoked operators included). Any new envelope raises one of them, and a revocation never lowers them.
• On every peer ping, if a connected peer with a signed seed announces a newer TV (a larger coordinator sum, or an equal one and a larger operator sum), the peer fetches that peer's /yacy/trust.json and merges it, at most 3 peers per round. Comparing the coordinator sum first lets a revocation reach peers that hold inflated operator list versions. TV is unsigned, so peers are asked in random order, and a peer that gave nothing new is not asked again for 10 minutes.
• Versions are accepted up to the current Unix time plus one day (and at most 2⁴⁰), so that nobody can block later versions with a huge one; use Unix time or small counters as versions. The bound applies to newly received statements, not to the peer's own file at start (a clock that went back does not lose lists).
• A peer on the coordinator's own list keeps that entry's priority and tags. Operators' lists under the same coordinator that name the same peer are merged: union of tags, lowest priority (so one operator cannot drop another's ads tag, and no operator can tag a peer the coordinator chose).
• trust.bundle.urls is also fetched at start, every 10 minutes and whenever the setting changes. Learning about new versions over several paths makes it harder to hold a peer on an old version.

5. Document provenance (author signatures)

• A peer signs each document when it indexes a page it crawled. The value is stored in the Solr field provenance_s and carried in DHT transfers and word index results as property prov: 1|<author key>|<signature>|<word Bloom filter>.
• Signed bytes: "yacy-doc-v1\n" + normalized URL + "\n" + title + "\n" + Bloom. The Bloom filter holds the hashes of the document's words, the same words that go into the word index. Its size is the smallest power of two ≥ 12 bits per word (512–65536 bits), with 4 hash functions. For word index results of other authors, a filter vouches for the query words only if absent words match by chance with probability fill^{4 × query words} ≤ 1/16 (a half-full filter for one word, 71% for two); 65536 bits are half full at about 11,000 words.

• Word index results must also have every query word in the Bloom filter, so a storage peer cannot attach a trusted document to unrelated words. A filter of another author's document that is too full for the number of query words (see above) is not used for word index results (the document can still be found through Solr).
• Snippets, text, descriptions, headings and image alt texts are not signed. They are kept only when the answering peer is the author or a trusted peer at a proven address; otherwise the word count, size and freshness date (unsigned ranking inputs) are dropped as well, and a result without a word count is weighted like a page of half the thin-page threshold. Remote results are stored locally (remotesearch.result.store) only when the answering peer is the trusted author.
• DHT storage (transferURL) rejects INVALID and accepts UNSIGNED only in open mode. Storage peers do not need to trust the author.

6. NAT traversal

 behind the NAT                            public side
┌──────────────────────┐          ┌──────────────────┐          ┌──────────────────────┐
│ YaCy ── sidecar ─────┼─ outbound ▶ relay (sidecar)  ◀ outbound ┼── sidecar ── YaCy    │
│  :8090   :8095 /:4001│ reserve  │  :4001           │ circuit  │  :8095     :8090     │
└──────────────────────┘          └──────────────────┘          └──────────────────────┘
          ◀────────────── DCUtR hole punching upgrades to a direct connection (if it works) ──────────────▶

• Sidecar (sidecar/, Go, go-libp2p): a libp2p host with YaCy's key. If the peer is not publicly reachable it reserves a slot on a configured circuit relay v2 and tries DCUtR.
• Tunnels. YaCy asks the control API GET /tunnel/<libp2p peer id>?addrs=… and gets a local port. Plain HTTP to 127.0.0.1:port is carried over a libp2p stream (/yacy/http/1.0.0) to the other sidecar. Because it is an ordinary host:port, YaCy's protocol and Solr clients work unchanged.
• The receiving sidecar forwards only /yacy/* and Solr select to a dedicated YaCy connector (127.0.0.1:8096). It never forwards admin pages, and it strips headers that claim a client address. The connector replaces the client address with an address in 2001:db8::/32 derived from the libp2p peer id, so tunnelled requests get none of the privileges of loopback clients, and rate limits apply per peer. The connector believes the peer id header only together with the sidecar's token.
• The control API requires a loopback Host header and no Origin (against DNS rebinding). /tunnel is authenticated with the token that YaCy writes to DATA/SETTINGS/sidecar.token (owner-only from creation, written atomically), but the token is never sent: YaCy sends a nonce (<unix ms>.<random>, valid for two minutes, once) and HMAC-SHA256(token, "yacy-sidecar-req-v1|" + method + "|" + path + "|" + nonce). A process that took the port while the sidecar was down learns nothing it can use.
• YaCy asks /status?nonce=… on every peer ping and uses the sidecar only if it signs the nonce, its peer id, boot id, reachability and relay addresses with the peer key. /tunnel answers are signed too (nonce, peer, port, boot id), and YaCy uses only ports its own sidecar gave.
• Limits: 256 tunnels, 15 minutes idle; a closed tunnel keeps its port for 10 minutes (longer than YaCy remembers ports), and with 256 such ports new tunnels are refused rather than freeing a port early; 8 concurrent requests per peer and 64 in total; 60 s and 32 MiB per response; request bodies up to 32 MiB only for index transfers, 1 MiB otherwise.
• The sidecar exits when the YaCy connector has been unreachable for 3 minutes (-yacy-watchdog): in Docker it shares YaCy's network namespace and would otherwise stay in the old one after YaCy restarts.
• From a circuit address in a seed only the relay's peer id is taken; the address to dial is rebuilt from the configured relay's own addresses, so a seed cannot make the sidecar connect to internal hosts. Peers reserved on a relay this sidecar does not know are therefore not reachable.
• A relay sidecar must be started with -relay-allow <file> or, deliberately, -relay-open: an open relay carries any libp2p traffic.
• YaCy switches its seed to Reach=relay with the circuit address in P2PA when AutoNAT says "private", or when there is no verdict and three peers in a row report it as junior. It stays there until AutoNAT says "public". p2p.mode=leecher means Reach=none.
• Peers behind a relay only answer searches by default. They are not DHT storage or DHT search targets unless they declare p2p.relay.dhtStorage=true (seed field RDS=1).
• hello through the sidecar must come from the libp2p identity of the seed's key. Both directions are allowed: a NAT peer calling out through the relay, and a public peer calling a NAT peer through its tunnel.

7. Settings

Keys, delegations, lists and bundles are made with the TrustTool command line tool. See FORK.md.

8. Known limits

• Japanese and Chinese word hashes differ from the old network (bigrams), even in open mode.
• Bloom filters of very long documents fill up. Where the chance match exceeds 1/16 (a one-word query on a document above about 11,000 words), word index results of other authors are not used; such documents are found through Solr or with more query words. Earlier fork versions reject the larger filters as INVALID: update fork peers together.
• OwnAddresses needs reports from three keys in two networks. An attacker with addresses in two networks and three keys can make a relayed address look like ours. In networks with fewer peers or networks, set a static IP.
• The TV format (two sums) is not compatible with earlier fork versions; while they are mixed, statements spread only through trust.bundle.urls. CJK splitting applies NFKC to every word: pages with full- or half-width characters keep their old words until they are indexed again.
• The IP of a seed relayed by a third party is unproven until this peer says hello itself.
• Responses themselves are not signed. Unsigned parts (snippets) are kept only from the author or a trusted peer at an address proven by the challenge.
• Through the sidecar, only peers reserved on one of this sidecar's configured relays are reachable.
• Other users of the same host can find the tunnel ports on 127.0.0.1 and send requests under this peer's libp2p identity, and a process that takes a tunnel port while the sidecar is down can answer as that peer until YaCy asks again (up to 5 minutes). Peer ids cost nothing, so many ids with slow requests can fill the 64 request slots. On shared hosts, run YaCy and the sidecar in their own user or container. Upstream's default adminAccountForLocalhost=true also makes every local process an administrator (the fork's Docker image sets it to false).
• A trusted author can sign false content, and a storage peer can withhold results.