GPU prefilter produces incorrect results when masking is enabled

[KimBioInfoStudio]

Bug description

When using GPU-accelerated ungapped prefilter with the default masking setting (--mask 1), some valid hits are missed compared to CPU results. Setting --mask 0 produces results identical to CPU.

Root cause

makepaddedseqdb applies tantan soft-masking to sequences before creating the GPU padded database. Soft-masked residues have their encoding value increased by 32 (e.g., A (0) becomes a (32)). In the GPU kernel, these masked values are passed through ClampToInvalid which maps them to the last row of the PSSM (row 20), where the score is 0.

This means any soft-masked position gets a score of 0 in the GPU prefilter, while the CPU prefilter handles masking differently (applying a score penalty rather than zeroing). This discrepancy causes the GPU to miss hits that the CPU finds, especially for sequences with significant low-complexity regions.

Reproduction

# Create test databases
mmseqs createdb query.fasta queryDB
mmseqs createdb target.fasta targetDB

# Create padded DB with masking (default)
mmseqs makepaddedseqdb targetDB targetDB_pad

# GPU search - may miss hits
mmseqs search queryDB targetDB_pad result_gpu tmp --gpu 1

# CPU search (ungapped) - finds all hits
mmseqs search queryDB targetDB result_cpu tmp --prefilter-mode 1

# Workaround: disable masking
mmseqs makepaddedseqdb targetDB targetDB_pad_nomask --mask 0
mmseqs search queryDB targetDB_pad_nomask result_gpu2 tmp --gpu 1
# result_gpu2 matches result_cpu

Observed behavior

• GPU with --mask 1 (default): missing hits compared to CPU
• GPU with --mask 0: results identical to CPU

This affects both protein and nucleotide GPU searches (including translated search modes blastx/tblastn/tblastx).

Workaround

Pass --mask 0 when creating the padded database or running GPU searches.

Environment

• MMseqs2 commit: dca44bc
• GPU: NVIDIA Blackwell (DGX Spark, compute capability 12.0)
• CUDA: 12.x

[KimBioInfoStudio]

Root cause analysis

After investigating the GPU kernel code, the issue is a divergence between gapless and Smith-Waterman kernels in how they handle soft-masked residues in the padded database.

Encoding in makepaddedseqdb

When masking is enabled, makepaddedseqdb encodes masked residues with +32 offset (makepaddedseqdb.cpp:80):

result.append(1, (seq.numSequence[i] == masker.maskLetterNum) ? charSequence[i] + 32 : charSequence[i]);

This uses the ConvertAA_20_CaseSensitive encoding scheme where masked A (0) becomes a (32), masked C (1) becomes c (33), etc.

Gapless kernel: ClampToInvalid (broken)

The gapless prefilter kernel uses ClampToInvalid at 4 sites in pssmkernels_gapless.cuh (lines 312, 714, 1393, 1797):

asUint = ClampToInvalid{}(asUint);  // min(value, 20)

This clamps any value ≥ 20 to 20 (the "invalid" row). Since masked residues are encoded as 32–51, they all get mapped to PSSM row 20, which is zero-filled → score = 0 for every masked position.

Smith-Waterman kernel: CaseSensitive_to_CaseInsensitive (correct)

The SW kernel in pssmkernels_smithwaterman.cuh (lines 124, 886) does the opposite — ClampToInvalid is commented out and replaced with:

asUint = CaseSensitive_to_CaseInsensitive{}(asUint);  // value % 32

This strips the mask bit and restores the original amino acid encoding, so masked residues score normally.

Why this matters

In the gapless prefilter, every masked position contributes 0 to the alignment score. For sequences with significant low-complexity regions (common in real databases), this can reduce scores enough that valid hits fall below the threshold and are lost. The CPU ungapped prefilter does not have this problem because it builds the PSSM from the query side and uses unmasked target encodings.

Possible fix

Replace ClampToInvalid with CaseSensitive_to_CaseInsensitive in the 4 gapless kernel sites, matching the approach already used in the SW kernel. This is a 4-line change in lib/libmarv/src/pssmkernels_gapless.cuh.

Alternatively, if masking penalties are desired in the prefilter, the PSSM row 20 could be filled with a small negative score (e.g., -1) instead of 0, to apply a controlled penalty rather than complete score zeroing.

[milot-mirdita]

ClampToInvalid is the intended behavior for masking in the prefilter, while the SW alignment should not mask (use the case insensitive function).

The idea here is to suppress false positive matches (i.e. unrelated through evolution) already during the prefiltering, while allowing to accurately score the sequences during alignment without masking.

The issue is that you should be using the same database for both:

# GPU search - may miss hits
mmseqs search queryDB targetDB_pad result_gpu tmp --gpu 1

# CPU search (ungapped) - finds all hits
mmseqs search queryDB targetDB_pad result_cpu tmp --prefilter-mode 1

CPU gapless supports the padded format too and then it would correctly mask in both CPU and GPU and produce consistent scores.

[KimBioInfoStudio]

Thanks for the clarification! That makes sense — using ClampToInvalid in the prefilter to suppress false positives from low-complexity regions, while letting the SW alignment score unmasked.

The discrepancy I observed was indeed from comparing against a non-padded database. Using the same padded database for both CPU and GPU gapless prefilter produces consistent results.

Closing this as working-as-intended.

[KimBioInfoStudio]

Verification on DGX Spark

Confirmed all three translated search modes produce consistent GPU/CPU results with --mask 0:

System:

• GPU: NVIDIA GB10 (DGX Spark, compute capability 12.1)
• Driver: 580.126.09, CUDA 12.x
• OS: Linux aarch64, kernel 6.17.0
• MMseqs2 commit: dca44bc

BLASTX (nucleotide query → protein target): ✅ identical

GPU: synthetic_nucl_query1  P0A6Y8  0.978  180  1.487E-40  126
CPU: synthetic_nucl_query1  P0A6Y8  0.978  180  1.487E-40  126
GPU: synthetic_nucl_query2  P00582  0.995  180  1.993E-41  128
CPU: synthetic_nucl_query2  P00582  0.995  180  1.993E-41  128

TBLASTN (protein query → nucleotide target): ✅ scores identical

GPU: P0A6Y8  target_nucl2  0.968  687  1.746E-156  469
CPU: P0A6Y8  target_nucl1  0.968  687  1.746E-156  469
GPU: P00582  target_nucl2  0.947  528  9.622E-116  352
CPU: P00582  target_nucl2  0.947  528  9.622E-116  352

(Target name/coordinate differences due to DB shuffle and strand representation)

TBLASTX (nucleotide query → nucleotide target): ✅ scores identical

GPU: query1→target_nucl2  0.978  180  1.840E-39  126  (+ 3 more hits)
CPU: query1→target_nucl1  0.978  180  1.840E-39  126  (+ 3 more hits)
GPU: query2→target_nucl2  0.995  180  2.749E-40  128  (+ 2 more hits)
CPU: query2→target_nucl2  0.995  180  2.749E-40  128  (+ 2 more hits)

All e-values and bit scores match exactly across GPU and CPU for all modes.

[milot-mirdita]

This was indeed buggy. Fixed in 9a9e7d9. Thanks for the report

[KimBioInfoStudio]

@milot-mirdita Thx for your fix, could u have a look for the pr #1082 #1081