Curl incosistent behaviour – CLI vs Java process

Trying to grab HTML for ttps://www.target.com.au/ with cUrl

This works ok via command-line:

xxx@VirtualBox:~/workspace/$ /usr/bin/curl -L -v -k -i -H "Accept: text/html" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" --max-time 10 https://www.target.com.au/ > target
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 23.0.111.198...
* TCP_NODELAY set
* Connected to www.target.com.au (23.0.111.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
.....

This always fail via Java:

    InputStream is = null;
    try {
        String command = "/usr/bin/curl -L -v -k -i -H \"Accept: text/html\" -H \"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36\" --max-time 10 " + url;
        System.out.println(command);
        Process process = Runtime.getRuntime().exec(command);
        is = process.getInputStream();
        String body = IOUtils.toString(is, StandardCharsets.UTF_8);
        return body;
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(is);
    }

OUTPUT:

/usr/bin/curl -L -v -k -i -H “Accept: text/html” -H “User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36” –max-time 10 https://www.target.com.au/

HTTP/2 403 server: AkamaiGHost mime-version: 1.0 content-type: text/html content-length: 270 expires: Fri, 05 Jul 2019 00:17:29 GMT date: Fri, 05 Jul 2019 00:17:29 GMT set-cookie: akavpau_prod_maintenance_vp=1562286149~id=979d0dae2676e513c633ce4f23c24ce0; Path=/ set-cookie: bm_sz=11D295C99ED7B362AFBB7A506F5BBD77~YAAQXrEHyloISrBrAQAAjzx+vwQpr3VkKRJiHodkXRN0RKsY9mAJuPB0g9bwOPcKkRcYltVyQ/K8f5vygv9S80T59R2NDJoF1Ei/2nfEUUicPAdkhSpnWYdXZiBSv0TTqbXZeEauEzVff4OjwhhvL6sGI43knNUZbliMNZecBDuoXjDUvZD/o/JFAtxWR3lRI+t3; Domain=.target.com.au; Path=/; Expires=Fri, 05 Jul 2019 04:17:29 GMT; Max-Age=14400; HttpOnly set-cookie: _abck=EA0253F20FADD24AE9333A50CD207788~-1~YAAQXrEHylsISrBrAQAAjzx+vwJv1VRMgxWDzHyBa1150prHaQO88ZGzl8kuNGzw3XRjLqOJaOMbT9mm5eWjT1pzMZq5WgyzPZM1+pc3n0UDCkVSZCZzon4/EXAkpbMNMeQfpHaurjCxf17U7javVptDE44op+nti7YNmdKUemKT/wMAL3RbWuPKwMRsduKFp1qyOQvK7tYOemfHd21YEFz/f1dGM+4SNqxRECJD4U+ErNQYJd93q3Mfca6QgOz1sDhvSGUNghKBCovxdjwLjaW77iZfZMI5owWID57L5Q==~-1~-1~-1; Domain=.target.com.au; Path=/; Expires=Sat, 04 Jul 2020 00:17:29 GMT; Max-Age=31536000; Secure

Access Denied

Access Denied

You don’t have permission to access “http://www.target.com.au/” on this server.

Reference #18.5eb107ca.1562285849.864a545

cUrl location is the same in both cases (Ubuntu), same machine.

xxx@VirtualBox:~/workspace/$ curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

Related Posts:

  1. How can I convert MP3 file to a Base64 encoded string? [closed]
  2. How is the AND/OR operator represented as in Regular Expressions?
  3. TCP vs UDP – What is a TCP connection? [duplicate]
  4. did you specify the right host or port? error on Kubernetes
  5. What is archive mode in rsync?
  6. 403 Forbidden vs 401 Unauthorized HTTP responses
  7. Turn off pager for psql’s interactive output
  8. How do I get my C# program to sleep for 50 msec?
  9. How can I sort a dictionary by key?
  10. How to align iframe always in the center
  11. How to turn IDENTITY_INSERT on and off using SQL Server 2008?sql-server-2008
  12. XPath contains(text(),’some string’) doesn’t work when used with node with more than one Text subnode
  13. What does enctype=’multipart/form-data’ mean?
  14. How do I remove version tracking from a project cloned from git?
  15. Error in .External.graphics R
  16. Time Complexity of Prims Algorithm?
  17. Does != have meaning in OCaml?
  18. Assembly’s manifest definition does not match assembly reference
  19. Java – No enclosing instance of type Foo is accessible
  20. How does createOrReplaceTempView work in Spark?
  21. “package XXX is not in GOROOT” when building a Go project
  22. what is the difference between OLE DB and ODBC data sources?
  23. How do I get a list of built-in data sets in R?
  24. How to terminate the script in JavaScript?
  25. Git push won’t do anything (everything up-to-date)
  26. spring @sqlgroup with multiple datasource
  27. Spyder does not autocomplete local variables
  28. What is the definition of a “disparity map”?
  29. Git error on git pull (unable to update local ref)
  30. git status shows modifications, git checkout — doesn’t remove them
  31. Tool for drawing parse trees?
  32. Resource interpreted as stylesheet but transferred with MIME type text/html (seems not related with web server)
  33. how to uninstall MinGW and make cygwin ‘make’ as deafult make program with gcc 3.8.1
  34. Clearing localStorage in javascript?
  35. “docker build” requires exactly 1 argument(s)
  36. ORA-01461: can bind a LONG value only for insert into a LONG column-Occurs when querying
  37. Optional Parameters in Go?
  38. Python FileNotFound
  39. Android java.lang.IllegalStateException: Could not execute method of the activity
  40. SwiftUI – How do I change the background color of a View?
  41. Cassandra “no viable alternative at input”
  42. Android – Package Name convention
  43. Using map in Haskell
  44. R dates “origin” must be supplied
  45. Error “The goal you specified requires a project to execute but there is no POM in this directory” after executing maven command
  46. Error in glm() in R
  47. Excel Filters – show only relevant values in the filter
  48. Swift protocols: method does not override any method from its superclass
  49. cannot load such file — bundler/setup (LoadError)
  50. JavaScript: IIF like statement
  51. How to increase size of DOSBox window?
  52. What is the relationship between PyTorch and Torch?
  53. Implementing a HashMap in C
  54. Mongoose.js: remove collection or DB
  55. slideToggle JQuery right to left
  56. Valgrind Invalid free() / delete / delete[] / realloc() in C
  57. How to set String’s font size, style in Java using the Font class?
  58. How can I run a Java program in Atom editor?
  59. How do you change session timeout in IIS 8.5?
  60. Is it valid to have a html form inside another html form?
  61. While running the script throws cannot find module ‘dotenv’
  62. Devise lockable – How to unlock account using unlock_in
  63. How to bind multiple values to a single WPF TextBlock?
  64. Brackets : how to make autocomplete / autoindent works ?
  65. How to specify the download location with wget?
  66. Error: “expected primary-expression before int”
  67. How do you detect where two line segments intersect?
  68. How to print variable in sml?
  69. Why is the minidlna database not being refreshed?
  70. Get a list of URLs from a site
  71. java.sql.SQLException: Fail to convert to internal representation
  72. In the Fibonacci sequence, is fib(0) 0 or 1 ?
  73. TCP/IP packets and datagrams
  74. How to set initial value and auto increment in MySQL?
  75. Visual Studio 2019 – error MSB8020: The build tools for Visual Studio 2013 cannot be found
  76. mport win32ui in python 3.6
  77. How to print a int64_t type in C
  78. textarea’s rows, and cols attribute in CSS
  79. Contact Form 7 – Execute code AFTER mail send [closed]
  80. wp_localize_script with mce_external_plugins in wordpress
  81. Forbid contributors viewing drafts
  82. Contact Form 7: Redirecting on a condition? [closed]
  83. show taxonomy meta field in template
  84. Multiple og:image for Facebook
  85. Reset counter with jQuery [closed]
  86. show just own posts of custom-post-type to wordpress user
  87. Symlink not updating as expected when using an absolute with relative path
  88. What are the different widely used RAID levels and when should I consider them?
  89. What is a glue record?
  90. Getting “Cannot ioctl TUNSETIFF tun: Operation not permitted” when trying to connect to OpenVPN
  91. Run Oracle SQL script and exit from sqlplus.exe via command prompt
  92. Windows Server restart / shutdown history
  93. Cooling Server Closet – No A/C Is Possible
  94. nmap find all alive hostnames and IPs in LAN
  95. How to filter http traffic in Wireshark?
  96. How to inspect remote SMTP server’s TLS certificate?
  97. Dealing with HTTP w00tw00t attacks
  98. How can I find the path to an executable in OSX
  99. Do SPF Records For Primary Domain apply to subdomains?
  100. Why is the response on localhost so slow?

Leave a Comment