return_k_alternatives param causing validation error

[mrcchef]

System Info

When I use return_k_alternatives with generation config

generation_config = {
            "max_new_tokens": 64,
            "temperature": 1,
            "do_sample": True,
            "top_p": 0.8,
            "top_k": 10,
            "return_k_alternatives": 5
        }

It cause validation error in the Response Pydantic Model

ValidationError: 36 validation errors for Response
details.tokens.0.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.0.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.1.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.1.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.1.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.2.alternative_tokens.1.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.2.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.2.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.2.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.3.alternative_tokens.1.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.3.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.3.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.3.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.4.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.4.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.4.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.5.alternative_tokens.1.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.5.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.5.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.5.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.6.alternative_tokens.1.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.6.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.6.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.6.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.7.alternative_tokens.1.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.7.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.7.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.7.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.8.alternative_tokens.1.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.8.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.8.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.8.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.9.alternative_tokens.1.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.9.alternative_tokens.2.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.9.alternative_tokens.3.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
details.tokens.9.alternative_tokens.4.logprob
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.6/v/float_type
{'finish_reason': <FinishReason.EndOfSequenceToken: 'eos_token'>,
 'prompt_tokens': 333,
 'generated_tokens': 7,
 'seed': 13699557125188310353,

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

use this generation config

generation_config = {
            "max_new_tokens": 64,
            "temperature": 1,
            "do_sample": True,
            "top_p": 0.8,
            "top_k": 10,
            "return_k_alternatives": 5
        }

and hit the generate API call

Expected behavior

If no alternative tokens are present, then it should not have any.

[mrcchef]

Response for a request
Here logprob values are None

[{'generated_text': 'what skin rash looks like', 'details': {'finish_reason': 'eos_token', 'prompt_tokens': 333, 'generated_tokens': 6, 'skipped_tokens': 0, 'seed': 10266900724815142667, 'prefill': [], 'tokens': [{'id': 12840, 'text': 'what', 'logprob': -1.1347656, 'special': False, 'alternative_tokens': [{'id': 37282, 'text': 'skin', 'logprob': -0.8222656}, {'id': 12840, 'text': 'what', 'logprob': -1.1347656}, {'id': 81496, 'text': 'identify', 'logprob': -1.4316406}, {'id': 0, 'text': '!', 'logprob': None}, {'id': 1, 'text': '"', 'logprob': None}], 'skipped': False}, {'id': 6930, 'text': ' skin', 'logprob': -1.1230469, 'special': False, 'alternative_tokens': [{'id': 1587, 'text': ' does', 'logprob': -0.9663086}, {'id': 6930, 'text': ' skin', 'logprob': -1.1230469}, {'id': 520, 'text': ' at', 'logprob': -1.7792969}, {'id': 459, 'text': ' an', 'logprob': -2.0761719}, {'id': 0, 'text': '!', 'logprob': None}], 'skipped': False}, {'id': 57342, 'text': ' rash', 'logprob': -0.45654297, 'special': False, 'alternative_tokens': [{'id': 57342, 'text': ' rash', 'logprob': -0.45654297}, {'id': 436, 'text': ' r', 'logprob': -1.0039062}, {'id': 0, 'text': '!', 'logprob': None}, {'id': 1, 'text': '"', 'logprob': None}, {'id': 2, 'text': '#', 'logprob': None}], 'skipped': False}, {'id': 5992, 'text': ' looks', 'logprob': 0.0, 'special': False, 'alternative_tokens': [{'id': 5992, 'text': ' looks', 'logprob': 0.0}, {'id': 0, 'text': '!', 'logprob': None}, {'id': 1, 'text': '"', 'logprob': None}, {'id': 2, 'text': '#', 'logprob': None}, {'id': 3, 'text': '$', 'logprob': None}], 'skipped': False}, {'id': 1093, 'text': ' like', 'logprob': 0.0, 'special': False, 'alternative_tokens': [{'id': 1093, 'text': ' like', 'logprob': 0.0}, {'id': 0, 'text': '!', 'logprob': None}, {'id': 1, 'text': '"', 'logprob': None}, {'id': 2, 'text': '#', 'logprob': None}, {'id': 3, 'text': '$', 'logprob': None}], 'skipped': False}, {'id': 128009, 'text': '<|eot_id|>', 'logprob': 0.0, 'special': True, 'alternative_tokens': [{'id': 128009, 'text': '<|eot_id|>', 'logprob': 0.0}, {'id': 0, 'text': '!', 'logprob': None}, {'id': 1, 'text': '"', 'logprob': None}, {'id': 2, 'text': '#', 'logprob': None}, {'id': 3, 'text': '$', 'logprob': None}], 'skipped': False}]}}]