Fix ondemand.s.js URL parsing for new X response format#410
Open
Fix ondemand.s.js URL parsing for new X response format#410
Conversation
X changed the ondemand.s.js format (Mar 2026). The response now uses a numeric index and a separate hash for the filename instead of the previous pattern. Update ON_DEMAND_FILE_REGEX and add ON_DEMAND_HASH_PATTERN to resolve the script URL correctly; adjust INDICES_REGEX and group index for key_byte_indices. Resolves d60#408 d60#409
Reviewer's guide (collapsed on small PRs)Reviewer's GuideAdjusts the ondemand.s.js discovery and URL construction to support X’s new response format that uses numeric indices and separate filename hashes, and updates the byte-index extraction logic accordingly. Sequence diagram for updated ondemand.s.js URL resolution in get_indicessequenceDiagram
participant ClientTransaction
participant HomePageResponse as home_page_response
participant Session as session
participant XFrontend as x_homepage
participant AbsCdn as abs_twitter_cdn
ClientTransaction->>HomePageResponse: validate_response(home_page_response)
HomePageResponse-->>ClientTransaction: response_str
ClientTransaction->>ClientTransaction: ON_DEMAND_FILE_REGEX.search(response_str)
alt ondemand index found
ClientTransaction->>ClientTransaction: Extract on_demand_file_index (group 1)
ClientTransaction->>ClientTransaction: Build hash_regex from ON_DEMAND_HASH_PATTERN and index
ClientTransaction->>ClientTransaction: hash_regex.search(response_str)
alt hash match found
ClientTransaction->>ClientTransaction: Extract filename hash (group 1)
ClientTransaction->>ClientTransaction: Build ondemand.s.js URL with filename hash
ClientTransaction->>AbsCdn: GET ondemand.s.<hash>a.js
AbsCdn-->>ClientTransaction: on_demand_file_response.text
ClientTransaction->>ClientTransaction: INDICES_REGEX.finditer(on_demand_file_response.text)
ClientTransaction->>ClientTransaction: Append each group 1 to key_byte_indices
else hash match not found
ClientTransaction->>ClientTransaction: key_byte_indices remains empty
end
else ondemand index not found
ClientTransaction->>ClientTransaction: key_byte_indices remains empty
end
ClientTransaction->>ClientTransaction: if not key_byte_indices -> raise Exception
ClientTransaction->>ClientTransaction: map key_byte_indices to int and return
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR fixes regex pattern matching in Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip You can disable the changed files summary in the walkthrough.Disable the |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
response_str = str(response)relies on the object's__str__implementation and may mask encoding/format issues; consider using the raw body (e.g.response.textor an explicit attribute) to make the parsing more predictable.- The two-step lookup (index via
ON_DEMAND_FILE_REGEXand hash viaON_DEMAND_HASH_PATTERN) fails silently and then raises a generic"Couldn't get KEY_BYTE indices"error; consider adding more granular error messages or early returns to indicate whether the index lookup, hash lookup, or script fetch failed. hash_regex = re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index))recompiles the pattern on each call; consider either usingre.searchdirectly on the formatted string or caching/parameterizing the pattern to avoid repeated compilation in hot paths.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- `response_str = str(response)` relies on the object's `__str__` implementation and may mask encoding/format issues; consider using the raw body (e.g. `response.text` or an explicit attribute) to make the parsing more predictable.
- The two-step lookup (index via `ON_DEMAND_FILE_REGEX` and hash via `ON_DEMAND_HASH_PATTERN`) fails silently and then raises a generic `"Couldn't get KEY_BYTE indices"` error; consider adding more granular error messages or early returns to indicate whether the index lookup, hash lookup, or script fetch failed.
- `hash_regex = re.compile(ON_DEMAND_HASH_PATTERN.format(on_demand_file_index))` recompiles the pattern on each call; consider either using `re.search` directly on the formatted string or caching/parameterizing the pattern to avoid repeated compilation in hot paths.
## Individual Comments
### Comment 1
<location path="twikit/x_client_transaction/transaction.py" line_range="17" />
<code_context>
from .utils import float_to_hex, is_odd, base64_encode, handle_x_migration
ON_DEMAND_FILE_REGEX = re.compile(
- r"""['|\"]{1}ondemand\.s['|\"]{1}:\s*['|\"]{1}([\w]*)['|\"]{1}""", flags=(re.VERBOSE | re.MULTILINE))
-INDICES_REGEX = re.compile(
- r"""(\(\w{1}\[(\d{1,2})\],\s*16\))+""", flags=(re.VERBOSE | re.MULTILINE))
+ r""",(\d+):["']ondemand\.s["']""", flags=(re.VERBOSE | re.MULTILINE))
+ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"'
+INDICES_REGEX = re.compile(r"\[(\d+)\],\s*16")
</code_context>
<issue_to_address>
**suggestion (bug_risk):** ON_DEMAND_FILE_REGEX supports both quote types, but the hash pattern assumes double quotes only.
Because `ON_DEMAND_HASH_PATTERN` is fixed to `:"..."`, it will fail when the source uses single quotes, even though `ON_DEMAND_FILE_REGEX` would still match. This creates a silent `hash_match is None` path. Consider updating `ON_DEMAND_HASH_PATTERN` to accept both quote types (e.g., `[:]["']([0-9a-fA-F]+)["']`).
```suggestion
ON_DEMAND_HASH_PATTERN = r',{}:["\']([0-9a-fA-F]+)["\']'
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| INDICES_REGEX = re.compile( | ||
| r"""(\(\w{1}\[(\d{1,2})\],\s*16\))+""", flags=(re.VERBOSE | re.MULTILINE)) | ||
| r""",(\d+):["']ondemand\.s["']""", flags=(re.VERBOSE | re.MULTILINE)) | ||
| ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"' |
There was a problem hiding this comment.
suggestion (bug_risk): ON_DEMAND_FILE_REGEX supports both quote types, but the hash pattern assumes double quotes only.
Because ON_DEMAND_HASH_PATTERN is fixed to :"...", it will fail when the source uses single quotes, even though ON_DEMAND_FILE_REGEX would still match. This creates a silent hash_match is None path. Consider updating ON_DEMAND_HASH_PATTERN to accept both quote types (e.g., [:]["']([0-9a-fA-F]+)["']).
Suggested change
| ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"' | |
| ON_DEMAND_HASH_PATTERN = r',{}:["\']([0-9a-fA-F]+)["\']' |
|
Can confirm this works. Applied the patch on my RPi running a Twitter monitoring bot (twikit 2.3.3, Python 3.11) — had all cookie sets failing with Please merge 🙏 |
kerslakechris
pushed a commit
to kerslakechris/test
that referenced
this pull request
Apr 1, 2026
Twitter changed ondemand.s.js structure in March 2026, breaking twikit's ClientTransaction.get_indices. Instead of calling the GraphQL API directly (which requires X-Client-Transaction-Id), we bring back twikit but patch its regex constants and get_indices method at import time to match the new JS structure. Fix sourced from: d60/twikit#410 https://claude.ai/code/session_01TZcmYCvNLAec3dfhBMAzJm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
X changed the ondemand.s.js format. The response now uses a numeric index and a separate hash for the filename instead of the previous pattern. Update
ON_DEMAND_FILE_REGEXand addON_DEMAND_HASH_PATTERNto resolve the script URL correctly; adjustINDICES_REGEXand group index for key_byte_indices.Credits to the contributors of the below issues.
Resolves #408 #409
Summary by Sourcery
Update ondemand.s.js parsing to support X’s new response format and correctly resolve key byte indices.
Bug Fixes:
Summary by CodeRabbit
Bug Fixes
Refactor