Description of the problem
When downloading some external files, i commonly get error 500 due to the use of anti phishing measures (in my case this is Safe Links (microsoft).
the Error message is of the form:
01:12:40 [1229] Download of the external file was canceled because of HTTP error: 500 Internal Server Error using url
https://eur02.safelinks.protection.outlook.com/?url=<VALID_URL>?id=19388&service=Careers%20Service&data=05|02|physics-part2@*******.ac.uk|004643f4da7b4c1b2c2608dce3b9a315|9c9bcd11977a4e9ca9a0bc734090164a|0|0|638635633687562272|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0=|0|||&sdata=0hP/5bYAnaaGIoBpTI20N3Ii8EKHhVpIm+b4BPAc0Xs=&reserved=0
where <VALID_URL> seems to be the true target of the link. There seems to be online sites that "decode" the link but as far as i can tell they just output the url query parameter.
Solution
An option in config to specify "protected" domain names that should be transformed into the correct url.
- cfg:
protected_domains = ("safelinks.protection.outlook.com", ... )
re.search(r"{<---domain--->}\/.*?\?url=(https?:\/\/.*)?", orig_url) ... or similar.
Description of the problem
When downloading some external files, i commonly get error 500 due to the use of anti phishing measures (in my case this is Safe Links (microsoft).
the Error message is of the form:
where
<VALID_URL>seems to be the true target of the link. There seems to be online sites that "decode" the link but as far as i can tell they just output the url query parameter.Solution
An option in config to specify "protected" domain names that should be transformed into the correct url.
protected_domains = ("safelinks.protection.outlook.com", ... )re.search(r"{<---domain--->}\/.*?\?url=(https?:\/\/.*)?", orig_url) ...or similar.