“CAUTION: provisional headers are shown” in Chrome debugger

The resource could be being blocked by an extension (AdBlock in my case). The message is there because the request to retrieve that resource was never made, so the headers being shown are not the real thing. As explained in the issue you referenced, the real headers are updated when the server responds, but there … Read more

How to define the basic HTTP authentication using cURL correctly?

From the documentation page: -u, –user <user:password> Specify the user name and password to use for server authentication. Overrides -n, –netrc and –netrc-optional. If you simply specify the user name, curl will prompt for a password. The user name and passwords are split up on the first colon, which makes it impossible to use a … Read more

HTTPS connections over proxy servers

TLS/SSL (The S in HTTPS) guarantees that there are no eavesdroppers between you and the server you are contacting, i.e. no proxies. Normally, you use CONNECT to open up a TCP connection through the proxy. In this case, the proxy will not be able to cache, read, or modify any requests/responses, and therefore be rather useless. If … Read more

How to define the basic HTTP authentication using cURL correctly?

From the documentation page: -u, –user <user:password> Specify the user name and password to use for server authentication. Overrides -n, –netrc and –netrc-optional. If you simply specify the user name, curl will prompt for a password. The user name and passwords are split up on the first colon, which makes it impossible to use a … Read more

application/x-www-form-urlencoded or multipart/form-data?

TL;DR Summary; if you have binary (non-alphanumeric) data (or a significantly sized payload) to transmit, use multipart/form-data. Otherwise, use application/x-www-form-urlencoded. The MIME types you mention are the two Content-Type headers for HTTP POST requests that user-agents (browsers) must support. The purpose of both of those types of requests is to send a list of name/value … Read more

Problem HTTP error 403 in Python 3 Web Scraping

This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it’s easily detected). Try setting a known browser user agent with: This works for me. By the way, in your code you are missing the () after .read in the urlopen … Read more

application/x-www-form-urlencoded or multipart/form-data?

TL;DR Summary; if you have binary (non-alphanumeric) data (or a significantly sized payload) to transmit, use multipart/form-data. Otherwise, use application/x-www-form-urlencoded. The MIME types you mention are the two Content-Type headers for HTTP POST requests that user-agents (browsers) must support. The purpose of both of those types of requests is to send a list of name/value … Read more