How to download a file over HTTP?
Use urllib.request.urlopen(): This is the most basic way to use the library, minus any error handling. You can also do more complex stuff such as changing headers. On Python 2, the method is in urllib2:
http
Is 418 “I’m a teapot” really an HTTP response code?
Is 418 “I’m a teapot” really an HTTP response code? There are various references to this on the internet, including in lists of response codes, but I can’t figure out whether it’s a weird joke.
application/x-www-form-urlencoded or multipart/form-data?
TL;DR Summary; if you have binary (non-alphanumeric) data (or a significantly sized payload) to transmit, use multipart/form-data. Otherwise, use application/x-www-form-urlencoded. The MIME types you mention are the two Content-Type headers for HTTP POST requests that user-agents (browsers) must support. The purpose of both of those types of requests is to send a list of name/value … Read more
Why Puma listen on ‘tcp://localhost:3000’ instead of ‘http://localhost:3000’
The HTTP protocol works on top of TCP, HTTP adds format to the raw TCP, so it kind of means the same thing. Given that Webrick and Puma are different HTTP servers, they may refer to the same thing in different ways. I hope this will be the worst of your problems during your journey … Read more
Problem HTTP error 403 in Python 3 Web Scraping
This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it’s easily detected). Try setting a known browser user agent with: This works for me. By the way, in your code you are missing the () after .read in the urlopen … Read more
application/x-www-form-urlencoded or multipart/form-data?
TL;DR Summary; if you have binary (non-alphanumeric) data (or a significantly sized payload) to transmit, use multipart/form-data. Otherwise, use application/x-www-form-urlencoded. The MIME types you mention are the two Content-Type headers for HTTP POST requests that user-agents (browsers) must support. The purpose of both of those types of requests is to send a list of name/value … Read more
Problem HTTP error 403 in Python 3 Web Scraping
This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it’s easily detected). Try setting a known browser user agent with: This works for me. By the way, in your code you are missing the () after .read in the urlopen … Read more
How to redirect all HTTP requests to HTTPS
Update: Although this answer has been accepted a few years ago, note that its approach is now recommended against by the Apache documentation. Use a Redirect instead. See this answer.
Do I need Content-Type: application/octet-stream for file download?
No. The content-type should be whatever it is known to be, if you know it. application/octet-stream is defined as “arbitrary binary data” in RFC 2046, and there’s a definite overlap here of it being appropriate for entities whose sole intended purpose is to be saved to disk, and from that point on be outside of … Read more
Use of PUT vs PATCH methods in REST API real life scenarios
NOTE: When I first spent time reading about REST, idempotence was a confusing concept to try to get right. I still didn’t get it quite right in my original answer, as further comments (and Jason Hoetger’s answer) have shown. For a while, I have resisted updating this answer extensively, to avoid effectively plagiarizing Jason, but … Read more