Successful copy resulting in corrupted data

[hgomersall]

The problem I'm encountering is a situation in which bmaptool copy completes the copy successfully but the actual data is corrupted.

The source is a gzipped sparse image. The created bmap file contains a reasonable collection of sparse blocks.

The problem occurs on an ARM A9 writing to an eMMC device, and also on a Intel PC writing to an SD card the latter on a different image. bmaptool reports completion, which presumably is using the checksums (since AFAICT that is the default).

I initially thought that the issue was that the random writes were causing the driver problems. I initially fixed the problem by setting

echo write through > /sys/block/mmcblk1/queue/write_cache

Then, on completion of the write, doing

sync
blockdev --flushbufs /dev/mmcblk1

However, when I changed the partition layout, this fix proved not reliable.

The invalid write can be checked with

dd count=<IMAGE SIZE IN BYTES> iflag=count_bytes if=/dev/mmcblk1 | sha256sum

and comparing that sha256sum to the expected output from the image. The failed write shows a different checksum.

Using dd on the images directly after unzipping them seems to work fine, and I can manually verify the checksum of the device against the source image (though the kernel could be lying to me here as what data is written).

[alchark]

Hi @hgomersall,

Can you check if #58 fixes it for your workflow?

Note also that doing a dd readback from the device may not result in a matching checksum due to several things:

1. bmaptool intentionally skips holes in the sparse image file when writing it to disk, so they will contain whatever previous data that was on disk prior to the write operation. When calculating the checksum on the source image file those regions would return all zeros, while read back from the disk they will contain old data, causing a mismatch (unless you did a full-disk wipe with something like blkdiscard prior to writing the image)
2. dd will likely read full blocks (i.e. a nearest multiple of 512 bytes by default), while the image may end before the block boundary, also causing a mismatch (unless the image is block size aligned)

[hgomersall]

Hi @hgomersall,

Can you check if #58 fixes it for your workflow?

Note also that doing a dd readback from the device may not result in a matching checksum due to several things:

1. `bmaptool` intentionally skips holes in the sparse image file when writing it to disk, so they will contain whatever previous data that was on disk prior to the write operation. When calculating the checksum on the source image file those regions would return all zeros, while read back from the disk they will contain old data, causing a mismatch (unless you did a full-disk wipe with something like `blkdiscard` prior to writing the image)

2. `dd` will likely read full blocks (i.e. a nearest multiple of 512 bytes by default), while the image may end before the block boundary, also causing a mismatch (unless the image is block size aligned)

Thanks for the response. I will try to look into this when I next work on this stuff.

I think I had come to the conclusion there was an issue with sparse writes and existing disk content. My current solution is to just ignore sparsity and write the image, then read it back and do a checksum on it.