KeyError in User.__init__ and Client.request when Twitter API omits expected fields

[PawiX25]

twikit version: 2.3.3
Python: 3.11

Description

User.__init__ and Client.request crash with KeyError when the Twitter/X API returns JSON responses with missing fields that twikit assumes are always present.

Bug 1: user.py — KeyError in User.__init__

Multiple fields in User.__init__ are accessed via direct dict lookup (legacy['key']). Twitter's API does not always include all legacy user fields in its responses, causing KeyError for fields like entities.description.urls, withheld_in_countries, and potentially any of the ~30 fields parsed in the constructor.

Traceback:

File "twikit/client/client.py", line 1402, in get_user_by_screen_name
    return User(self, user_data)
File "twikit/user.py", line 102, in __init__
    self.description_urls: list = legacy['entities']['description']['urls']
KeyError: 'urls'

After the first crash is hit, subsequent fields also fail:

File "twikit/user.py", line 124, in __init__
    self.withheld_in_countries: list[str] = legacy['withheld_in_countries']
KeyError: 'withheld_in_countries'

Bug 2: client/client.py — KeyError in error handling

Client.request accesses response_data['errors'][0]['code'] directly, but Twitter sometimes returns error objects without a code field.

This causes a KeyError during error handling itself, masking the original error.

Expected behavior

twikit should handle missing fields gracefully using .get() with sensible defaults instead of crashing.

Environment

• Observed during normal polling of user timelines via get_user_tweets() and get_user_by_screen_name()
• Reproducible intermittently — Twitter API responses vary in which fields they include

[PriyanshAroraa]

Can confirm, hitting the exact same issue on 2.3.3 / Python 3.11. Both bugs are real:

Bug 1: get_user_by_screen_name and get_user_tweets both crash on KeyError for urls, withheld_in_countries, and a few others. X's API response is missing these fields intermittently for certain accounts.

Bug 2: Confirmed response_data['errors'][0]['code'] can KeyError too - X sometimes returns error objects without a code field, which masks the actual failure with a confusing KeyError.

Patch that works for me:

user.py - switching all legacy[...] lookups to .get() with sensible defaults (list for list fields, empty string for strings, 0 for counts, False for booleans):

self.location: str = legacy.get('location', '')
self.description: str = legacy.get('description', '')
self.description_urls: list = legacy.get('entities', {}).get('description', {}).get('urls', [])
self.urls: list = legacy.get('entities', {}).get('url', {}).get('urls', [])
self.pinned_tweet_ids: list[str] = legacy.get('pinned_tweet_ids_str', [])
self.is_blue_verified: bool = data.get('is_blue_verified', False)
self.verified: bool = legacy.get('verified', False)
# ... same pattern for all fields
self.withheld_in_countries: list[str] = legacy.get('withheld_in_countries', [])

client/client.py line 165:

error_code = response_data['errors'][0].get('code')

After applying both, get_user_by_screen_name and get_user_tweets work reliably again across 50+ accounts (racing, football, gaming, crypto niches) with zero KeyErrors across thousands of calls.

Happy to open a PR if that helps move it forward.

[PriyanshAroraa]

Followup: opened #421 with the patch.

What I changed:

• user.py — converted every legacy['key'] in User.__init__ to legacy.get('key', <default>). Defaults match the declared type (empty string for strings, 0 for counts, False for booleans, empty list for list fields). Also deep-safed the two nested lookups: legacy.get('entities', {}).get('description', {}).get('urls', []) and the same pattern for url.urls.
• client/client.py — one-liner on L158: response_data['errors'][0]['code'] → response_data['errors'][0].get('code'). The adjacent error_message already used .get(), so this just matches the existing style.

How I'm using it:
Running a polling loop that scans ~50 accounts across different niches (racing, football, gaming, crypto) every 30 min. Calls get_user_by_screen_name then get_user_tweets for each, filters by view count. Before the patch this crashed on maybe 20-30% of the accounts — particularly brand accounts and accounts without a url or empty description.entities blocks seemed to trigger it most.

After the patch: thousands of get_user_by_screen_name + get_user_tweets calls across a ~2 day run, zero KeyErrors. The only failure modes I still see are expected ones (429 rate limits, 226 automated-request flags).

Happy to iterate on the PR if there's a preferred style (e.g. wanting None defaults instead of type-matched ones, or wanting to raise a more descriptive exception upstream rather than silently defaulting).

[PawiX25]

This is already fixed in my PR #418 which I opened the same day as this issue. Please close your duplicate PR #421.