Trust and NAT design

1. Goals and threat model

Out of scope: storage peers withholding results, false content signed by a trusted author (handled by tags and audit), query privacy.

2. Identity

• At start a peer reads DATA/SETTINGS/peer.key (Ed25519, PKCS#8 PEM) and creates it if missing.
• Peer hash (12 characters) = the first 12 characters of the URL-safe base64 of SHA-256(public key). Upstream's choice of ids to fill gaps in the DHT is dropped: the key decides.
• The same key is the libp2p key of the NAT sidecar. The libp2p peer id (12D3KooW…) is computed from the public key and is not stored in the seed.

2.1 Signed seeds

Receivers rewrite parts of a seed (IP, type, flags, last seen) and relay it. Only the fields that the owner alone decides are signed:

• Sig = Ed25519 over "yacy-seed-v1\n" + "Hash=<hash>\n" followed by the other core fields that have a value, as k=v\n in key order (BDate, Name, P2PA, PK, Port, PortSSL, RDS, Reach, SigT, Tags).
• A seed with PK must satisfy Hash == H(PK) and carry a valid Sig, or it is always rejected. Seeds without PK are rejected unless trust.seed.acceptUnsigned=true.
• A peer signs its seed again when the core changes or the last signature is 7 days old. Seeds whose SigT is older than 30 days, or more than 1 hour in the future, are rejected for peers not yet known. A known signed seed is never replaced by an older or unsigned one.

2.2 Hello challenge

The IP address is not signed, so hello checks that the key owner really answers at that address:

• The caller sends a random challenge. The responder returns the address it saw the request come from (challengeFor, or p2p through the sidecar) and Sign("yacy-hello-v2|" + challenge + "|" + own hash + "|" + challengeFor).
• The caller verifies the signature with the PK of the returned seed and checks that challengeFor is its own address. Binding the address defeats an attacker who forwards the challenge to the real peer and returns its answer: the real peer signs "a request from the attacker".
• The back-ping (the responder calling the caller back to test reachability) uses the same challenge, so a seed that claims someone else's IP is rejected.
• Peers that do not know their public address (juniors, peers behind a NAT) skip the address check. The forwarding attack remains possible only for them.

3. Delegation and trust lists

coordinator key (configured by the user; several allowed, first has priority)
  └─ delegation: "this operator key may publish the list for this network"   (versioned; revocation is a newer version)
       └─ peer list: "trust these peers, with this priority and these tags"  (signed by the operator; versioned)

• One level of delegation. A coordinator may be its own operator.
• No expiry, only versions. If the operators disappear, the last version keeps working.
• Everything travels in one envelope format. The signature covers the payload bytes, so no JSON canonicalization is needed:

{"payload": "<base64url(JSON bytes)>", "signer": "<base64url public key>", "sig": "<base64url signature>"}

{"type": "yacy-delegation-v1", "network": "freeworld", "operator": "<pk>", "version": 3, "revoked": false}
{"type": "yacy-peerlist-v1", "network": "freeworld", "version": 12,
 "peers": [{"pk": "<pk>", "priority": 100, "tags": ["ads"]}]}

• Delegations must be signed by a coordinator. The highest version per (coordinator, operator) wins, and revoked:true invalidates that operator's lists. Lists must be signed by an operator with a valid delegation, and the highest version per operator wins. network must equal network.unit.name or be *.
• The effective trust set layers the lists in coordinator order. A peer keeps the priority and tags of the first list that names it. Matching uses the full public key, not only the 72-bit peer hash.
• priority is 0–100, and result scores are multiplied by 0.5 + 0.5 × priority / 100.
• Without coordinators a peer trusts only its own documents (fail closed). Closed networks where everybody knows everybody can set trust.signedOnly=true to trust every signed peer. A distribution would ship its coordinator key as the default; this fork ships none.

3.1 Declared tags

A small common vocabulary. Coordinators may add their own tags with an x-<name>: prefix.

Tags on the list are authoritative. Tags in the seed are self-declared and only displayed. trust.policy.excludeTags drops documents of authors with those tags, and results carry their author's tags.

4. Distributing lists

• Each peer verifies the envelopes it receives, stores them in DATA/SETTINGS/trust-bundle.json and serves them at /yacy/trust.json. Anybody may relay them.
• The seed carries TV: per coordinator, the sum of the versions of all envelopes held (delegations plus lists, revoked operators included). Any new envelope raises the sum, and a revocation never lowers it.
• On every peer ping, if a connected peer announces a larger TV, the peer fetches that peer's /yacy/trust.json and merges it. TV is unsigned, so peers are asked in random order, and a peer that gave nothing new is not asked again for 10 minutes.
• trust.bundle.urls is also fetched at start, every 10 minutes and whenever the setting changes. Learning about new versions over several paths makes it harder to hold a peer on an old version.

5. Document provenance (author signatures)

• A peer signs each document when it indexes a page it crawled. The value is stored in the Solr field provenance_s and carried in DHT transfers and word index results as property prov: 1|<author key>|<signature>|<word Bloom filter>.
• Signed bytes: "yacy-doc-v1\n" + normalized URL + "\n" + title + "\n" + Bloom. The Bloom filter holds the hashes of the document's words, the same words that go into the word index. Its size is the smallest power of two ≥ 12 bits per word (512–32768 bits), with 4 hash functions.

• Word index results must also have every query word in the Bloom filter, so a storage peer cannot attach a trusted document to unrelated words.
• Snippets, text, descriptions, headings and image alt texts are not signed. They are kept only when the answering peer is the author or a trusted peer at a proven address. Remote results are stored locally (remotesearch.result.store) only when the answering peer is the trusted author.
• DHT storage (transferURL) rejects INVALID and accepts UNSIGNED only in open mode. Storage peers do not need to trust the author.

6. NAT traversal

 behind the NAT                            public side
┌──────────────────────┐          ┌──────────────────┐          ┌──────────────────────┐
│ YaCy ── sidecar ─────┼─ outbound ▶ relay (sidecar)  ◀ outbound ┼── sidecar ── YaCy    │
│  :8090   :8095 /:4001│ reserve  │  :4001           │ circuit  │  :8095     :8090     │
└──────────────────────┘          └──────────────────┘          └──────────────────────┘
          ◀────────────── DCUtR hole punching upgrades to a direct connection (if it works) ──────────────▶

• Sidecar (sidecar/, Go, go-libp2p): a libp2p host with YaCy's key. If the peer is not publicly reachable it reserves a slot on a configured circuit relay v2 and tries DCUtR.
• Tunnels. YaCy asks the control API GET /tunnel/<libp2p peer id>?addrs=… and gets a local port. Plain HTTP to 127.0.0.1:port is carried over a libp2p stream (/yacy/http/1.0.0) to the other sidecar. Because it is an ordinary host:port, YaCy's protocol and Solr clients work unchanged.
• The receiving sidecar forwards only /yacy/* and Solr select to a dedicated YaCy connector (127.0.0.1:8096). It never forwards admin pages, and it strips headers that claim a client address. The connector replaces the client address with an address in 2001:db8::/32 derived from the libp2p peer id, so tunnelled requests get none of the privileges of loopback clients, and rate limits apply per peer. The connector believes the peer id header only together with the sidecar's token.
• The control API requires a token that YaCy writes to DATA/SETTINGS/sidecar.token, a loopback Host header and no Origin, against other local processes and DNS rebinding. Limits: 256 tunnels, 15 minutes idle, 8 concurrent requests per peer, 60 s and 32 MiB per request.
• YaCy switches its seed to Reach=relay with the circuit address in P2PA when AutoNAT says "private", or when there is no verdict and three peers in a row report it as junior. It stays there until AutoNAT says "public". p2p.mode=leecher means Reach=none.
• Peers behind a relay only answer searches by default. They are not DHT storage or DHT search targets unless they declare p2p.relay.dhtStorage=true (seed field RDS=1).
• hello through the sidecar must come from the libp2p identity of the seed's key. Both directions are allowed: a NAT peer calling out through the relay, and a public peer calling a NAT peer through its tunnel.

7. Settings

Keys, delegations, lists and bundles are made with the TrustTool command line tool. See FORK.md.

8. Known limits

• Japanese and Chinese word hashes differ from the old network (bigrams), even in open mode.
• Bloom filters of very long documents (above about 2700 words) have more false positives, so the word check on word index results is weaker there.
• The IP of a seed relayed by a third party is unproven until this peer says hello itself.
• Responses themselves are not signed. A peer impersonating a trusted peer could replace unsigned parts (snippets). The address-bound challenge prevents that, except for peers that do not know their public address.
• A trusted author can sign false content, and a storage peer can withhold results.