Summary
When a validator syncs from genesis (e.g., after a chain data wipe), it encounters historical blocks where it held AURA slots. The partner_chains_aura_import_queue import logic does not distinguish between "importing old blocks" and "actively authoring," causing the node to send duplicate block requests for already-existing blocks. Peers interpret this as misbehavior and apply INT32_MIN reputation penalty (Substrate sc-network crate), permanently banning the syncing validator.
Impact
- Any validator that needs to sync from genesis (after data wipe, EBS replacement, PVC deletion) will be permanently banned by peers and fork
- Observed on qanet (May 2026): EC2 validators syncing through ~800k blocks hit AURA slots repeatedly, generating 18,000+ ban events in 3 days
- Current workaround is the unban monitor (polls
network_peerReputations, auto-calls network_unbanPeer), but this is a band-aid — the duplicate block requests still occur
Expected Behavior
A syncing validator should not trigger AURA block production logic when importing historical blocks. The node should detect it is in sync mode and skip AURA slot handling for blocks that already exist on the chain.
Reproduction
- Deploy a validator on an established chain (e.g., qanet with ~800k blocks)
- Wipe its chain data (
rm -rf /var/lib/midnight/)
- Restart the node — it syncs from block 0
- Observe peer banning logs on other validators:
Same block request multiple times from <peer_id>
- Syncing validator loses all peers and forks
Relevant Code
partner_chains_aura_import_queue — where AURA slot verification happens during import- Substrate
sc-network crate — where the INT32_MIN reputation penalty is applied
Workaround
Unban monitor running on peers (K8s chart midnight.unbanMonitor.enabled: true). This auto-unbans the syncing validator so it can make progress, but does not prevent the duplicate block requests.
Summary
When a validator syncs from genesis (e.g., after a chain data wipe), it encounters historical blocks where it held AURA slots. The
partner_chains_aura_import_queueimport logic does not distinguish between "importing old blocks" and "actively authoring," causing the node to send duplicate block requests for already-existing blocks. Peers interpret this as misbehavior and applyINT32_MINreputation penalty (Substratesc-networkcrate), permanently banning the syncing validator.Impact
network_peerReputations, auto-callsnetwork_unbanPeer), but this is a band-aid — the duplicate block requests still occurExpected Behavior
A syncing validator should not trigger AURA block production logic when importing historical blocks. The node should detect it is in sync mode and skip AURA slot handling for blocks that already exist on the chain.
Reproduction
rm -rf /var/lib/midnight/)Same block request multiple times from <peer_id>Relevant Code
partner_chains_aura_import_queue— where AURA slot verification happens during importsc-networkcrate — where theINT32_MINreputation penalty is appliedWorkaround
Unban monitor running on peers (K8s chart
midnight.unbanMonitor.enabled: true). This auto-unbans the syncing validator so it can make progress, but does not prevent the duplicate block requests.