Skip to content

bug(backfill): We are seeing some invalid (corrupted) blocks on some production networks and timeouts #2651

@AlfredoG87

Description

@AlfredoG87

Description

On a previous investigation of a BN in Previewnet trying to backfill, there were some connections that timed-out, and also lots of blocks that were received unabled to be parsed, with different parsing issues.

When manually querying the source and the block ranges, I was able to get the blocks, some times after a single retry but most of the time at first. This manual test was done using Postman (not helidon) webClient.

Steps to Reproduce

There is still not a reliable way to reproduce this, the current hypothesis is that HTTP2 WebClient of Helidon is not robust when network is choppy or has fluctuations, resulting in half-dead connections (we have seen this with the publisher as a form of stalled connections) and parsing issues.

The first part of this issue is to be able to find the root cause by introducing network fluctuations and chaos.

One idea is to use Chaos Mesh to introduce some network conditions and see if we can reliably reproduce the error. finally create an isolated test that is able to identify the issue, and if possible to fix it.

Additional context

These are some of the error messages found:

Failed to fetch blocks [151,290->151,309] from node [[lfh02.previewnet.blocknode.hashgraph-devops.com](http://lfh02.previewnet.blocknode.hashgraph-devops.com/)] (attempt 1/3) due to java.lang.RuntimeException: Error fetching blocks - , com.hedera.pbj.runtime.ParseException: java.io.IOException: Cannot understand wire_type of 6.
Failed to fetch blocks [0->19] from node [[lfh02.previewnet.blocknode.hashgraph-devops.com](http://lfh02.previewnet.blocknode.hashgraph-devops.com/)] (attempt 3/3) due to java.lang.RuntimeException: Error fetching blocks - , java.lang.RuntimeException: Timed out waiting for block stream response.
warn rfh01 2026-04-17 20:19:18.250+0000 WARNING [io.helidon.webclient.http2.Http2ClientConnection handle] Unsupported frame type!! UNKNOWN
[org.hiero.block.node.backfill.BackfillFetcher fetchBlocksFromNode] Failed to fetch blocks [0->19] from node [[lfh02.previewnet.blocknode.hashgraph-devops.com](http://lfh02.previewnet.blocknode.hashgraph-devops.com/)] (attempt 1/3) due to java.lang.RuntimeException: Error fetching blocks - , com.hedera.pbj.runtime.ParseException: java.io.IOException: Bad tag [8], field [1] wireType [0].

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No fields configured for Bug.

    Projects

    Status

    🧊 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions