TheLazy.Dev
Portfolio GitHub

Blog

Generate HTTP Request

kracekumar

Kracekumar

Posted on October 11, 2021

Generate HTTP Request

HTTPie is a command-line utility for making HTTP requests with more straightforward syntax(controversial, I agree). The interesting feature is --offline flag which prints HTTP raw request text. The client sends the HTTP request to the server, and the server responds to the request. It's an alternate to curl.

HTTP Syntax

HTTP Flow and syntrax from Wikipedia.

A client sends request messages to the server, which consist of

  • a request line, consisting of the case-sensitive request method, a space, the request target, another space, the protocol version, a carriage return, and a line feed (e.g. GET /images/logo.png HTTP/1.1)
  • zero or more request header fields, each consisting of the case-insensitive field name, a colon, optional leading whitespace, the field value, and optional trailing whitespace (e.g. Accept-Language: en), and ending with a carriage return and a line feed.
  • an empty line, consisting of a carriage return and a line feed;
  • an optional message body.
  • In the HTTP/1.1 protocol, all header fields except Host are optional.
  • A request line containing only the path name is accepted by servers to maintain compatibility with HTTP clients before the HTTP/1.0 specification in RFC 1945.

Throughout the post, I'll use --offline feature to understand how the HTTP request structure looks for educational purposes.

🉑 Accept only JSON response data 🉑 

$http --offline httpbin.org/get Accept:application/json
GET /get HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: httpbin.org
User-Agent: HTTPie/2.5.0
# Sample request sent to the server
$ http httpbin.org/get Accept:application/json
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 310
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:21:40 GMT
Server: gunicorn/19.9.0

{
    "args": {},
    "headers": {
        "Accept": "application/json",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/2.5.0",
        "X-Amzn-Trace-Id": "Root=1-61629484-3b25a3631e2a89bf60f2600e"
    },
    "origin": "xxx.xxx.xxx.xxx",
    "url": "http://httpbin.org/get"
}
Enter fullscreen mode Exit fullscreen mode

  • You can pass more than one value in the Accept header seperated by comma.

  • The response from the server contains Content-Type as JSON. The server can choose to ignore the Accept header.

📚 Request the Tamil language version of duckduckgo 📚 

$http --offline  https://duckduckgo.com Accept-Language:ta
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: ta
Connection: keep-alive
Host: duckduckgo.com
User-Agent: HTTPie/2.5.0

$http  https://duckduckgo.com Accept-Language:ta
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: keep-alive
Content-Encoding: gzip
Content-Security-Policy: default-src 'none' ; connect-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; manifest-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; media-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; script-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' 'unsafe-eval' ; font-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; img-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; style-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' ; object-src 'none' ; worker-src blob: ; child-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; frame-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; form-action  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; frame-ancestors 'self' ; base-uri 'self' ; block-all-mixed-content ;
Content-Type: text/html; charset=UTF-8
Date: Sat, 09 Oct 2021 21:44:33 GMT
ETag: W/"6161a338-16b8"
Expect-CT: max-age=0
Expires: Sat, 09 Oct 2021 21:44:32 GMT
Permissions-Policy: interest-cohort=()
Referrer-Policy: origin
Server: nginx
Strict-Transport-Security: max-age=31536000
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1;mode=block

<!DOCTYPE html>
...
<body id="pg-index" class="page-index body--home">
    <script type="text/javascript" src="/tl5.js"></script>
<script type="text/javascript" src="/lib/l123.js"></script>
<script type="text/javascript" src="/locale/ta_IN/duckduckgo50.js"></script>
<script type="text/javascript" src="/util/u588.js"></script>
<script type="text/javascript" src="/d3012.js"></script>
...
<noscript>
    <div class="tag-home">
        <div class="tag-home__wrapper">
            <div class="tag-home__item">
                Privacy, simplified&period;
                <span class="hide--screen-xs"><a href="/about" class="tag-home__link"> மேலும் கற்க</a></span>
            </div>
        </div>
    </div>
</noscript>
...
</html>
Enter fullscreen mode Exit fullscreen mode
  • The header can carry more than one value separated by a comma.
  • When more than one value is present, an extra parameter q=0.5 represents the weightage among the values. Example: Accept-Language: ta,fr;q=0.50.
  • One of the link description is in Tamil, மேலும் கற்க.

🔑 Post Authorization token as part of login 🔑 

$http --offline POST httpbin.org/auth Authorization:"Bearer my-dear-darkness"
POST /auth HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Bearer my-dear-darkness
Connection: keep-alive
Content-Length: 0
Host: httpbin.org
User-Agent: HTTPie/2.5.0
Enter fullscreen mode Exit fullscreen mode

🍪 Send Cookie as part of a request 🍪 

$http --offline http://pie.dev/cookies Cookie:"name=not-unique;value=23"
GET /cookies HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: name=not-unique;value=23
Host: dev.pie
User-Agent: HTTPie/2.5.0
$http http://pie.dev/cookies Cookie:"name=not-unique;value=23"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1b984c743c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:24:13 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2Zt9wQcKQYStY5dPukoEXFwn7K6pKNaVTiKljZ5h9oL3mcQZj0khYq8Kmzo8PmQinPncStZeHASetQqHzCODe0wbrljPEIJxCWGdRWMbry9rWOG%2FBheDvJs7"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "cookies": {
        "name": "not-unique",
        "value": "23"
    }
}
Enter fullscreen mode Exit fullscreen mode

✉️ Send JSON Request ✉️

  • HTTPie supports sending form-encoded values or JSON values. = sign indiciates JSON key-value pair.
  • = is useful for primitive values like number, string, null, boolean.
  • Example: lang-py 
$http --offline PUT http://pie.dev/put lang=py version=3.10
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 33
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "version": "3.10"
}
$HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1e017fb53c12-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:25:52 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GMW5PQHMwjxpVUeDG7uWo6M%2FiKLMzmQLd1e5BIG3AXljmuQgwCP9nzrFPdaidR2wL14eisfiViDhumDHpepgNB6yIrsVKXRybHa5tRqmH7lUDoQdRqCK1Ijg"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\"}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be1e017fb53c12-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "33",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "version": "3.10"
    },
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/put"
}
Enter fullscreen mode Exit fullscreen mode

✉️ Send non-primitive JSON values ✉️

  • := means the JSON value is non-primitve values like array/list and dictionary.
  • Single quote carries the value.
  • Example: os:='["GNU/Linux", "Mac OSX"]'. 
$http --offline  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 65
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "os": [
        "GNU/Linux",
        "Mac OSX"
    ],
    "version": "3.10"
}
$http  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be224ce97c3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:28:48 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=dgY0R%2FqomBbumKs0IVB7GjPR2chhHJ9wVKAzW33IOnvZ%2Fs4l2LYnvKKeXo6Xhd162AYKGzyIrK4pNrYdH8SEs1NvGmYfJqEDIPmfUOELqpC6HK9iP1zIENa0"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be224ce97c3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "65",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "os": [
            "GNU/Linux",
            "Mac OSX"
        ],
        "version": "3.10"
    },
    "origin": "xx.xxx.xxx.xxx",
    "url": "http://pie.dev/put"
}
Enter fullscreen mode Exit fullscreen mode

📤 Upload files 📤

  • -f flag indicates the data is form-encoded values.
  • @ symbol indicates the value is a file.
  • Example: cv@hello-world.txt. cv form field name. 
$cat hello-world.txt
hello-world
$http --offline -f POST pie.dev/post name='Krace' cv@hello-world.txt
POST /post HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 277
Content-Type: multipart/form-data; boundary=183f70d3da41432d95bcd839e2cc20e2
Host: pie.dev
User-Agent: HTTPie/2.5.0

--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="name"

Krace
--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="cv"; filename="hello-world.txt"
Content-Type: text/plain

hello-world

--183f70d3da41432d95bcd839e2cc20e2--
$ http -f POST pie.dev/post name='Krace' cv@hello-world.txt
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be2564a8fc3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:30:54 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yNaTSZJ7ouYH%2FSJZw6LIR5vNl6KNTFeDKF1u8V60abb3ClKLzdOj0zkchcIAWqTvZDbxXm5MnffDkCLdqviMQuAo7DqFA2GH%2Bm%2FiZ6sH90oOr0HyFGAuS4Gp"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "",
    "files": {
        "cv": "hello-world\n"
    },
    "form": {
        "name": "Krace"
    },
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be2564a8fc3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "277",
        "Content-Type": "multipart/form-data; boundary=252c6fcd1dcc40e09de54958660d672d",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": null,
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/post"
}
Enter fullscreen mode Exit fullscreen mode

Explore more in HTTPie website.

🐈 How to verify the the generated request works? 🐈

Netcat is a utility for sending and receiving data in network connection using TCP or UDP. Netcat take hostname, port, and body as arguments and sends it to the server and displays the response.

$http --offline  PUT http://httpbin.org/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]' | nc httpbin.org 80
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2021 08:27:58 GMT
Content-Type: application/json
Content-Length: 631
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json, */*;q=0.5",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "65",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "HTTPie/2.5.0",
    "X-Amzn-Trace-Id": "Root=1-6162a40e-34b9a83f40868b4a73e8fa09"
  },
  "json": {
    "lang": "py",
    "os": [
      "GNU/Linux",
      "Mac OSX"
    ],
    "version": "3.10"
  },
  "origin": "xxx.xxx.xxx.xxx",
  "url": "http://httpbin.org/put"
}
Enter fullscreen mode Exit fullscreen mode

Netcat doesn't support encrypted network connections.


❓How to understand what HTTP request the client sends to the server?
💡Use HTTPie --offline feature to print the HTTP request.

Details on how to send various HTTPie options in the blog post - https://t.co/mIEG2FpBkP#TIL #Python #CLI pic.twitter.com/kBvfc9QQAt

— kracekumar || கிரேஸ்குமார் (@kracetheking) October 10, 2021

References

💖 💪 🙅 🚩
kracekumar
Kracekumar

Posted on October 11, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

I am purposefully super nice to AI because I imagine they are trained on data where nice people get better help. Doing this, though, has an existential uneasiness to it.
ai I am purposefully super nice to AI because I imagine they are trained on data where nice people get better help. Doing this, though, has an existential uneasiness to it.

November 28, 2024

If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.
undefined If you're a beginner, definitely check this open source guide. I've explained almost everything you need to know.

November 28, 2024

Where GitOps Meets ClickOps
devops Where GitOps Meets ClickOps

November 29, 2024

The best way to get better at writing code is...
development The best way to get better at writing code is...

November 28, 2024

How to Use KitOps with MLflow
beginners How to Use KitOps with MLflow

November 29, 2024