Wget is a non-interactive Unix/Linux command-line tool that retrieves content from web servers. The tool can be very useful for identifying various HTTP server responses or performance-related issues.
For probing an HTTP server and identifying its response, it can be used the
--spider option.
Example:
wget --spider 11131-8.b.cdn12.com/sample.mp4$ wget --spider 11131-8.b.cdn12.com/sample.mp4
Spider mode enabled. Check if remote file exists.
--2020-11-12 18:26:27-- http://11131-8.b.cdn12.com/sample.mp4
Resolving 11131-8.b.cdn12.com (11131-8.b.cdn12.com)... 185.18.187.85
Connecting to 11131-8.b.cdn12.com (11131-8.b.cdn12.com)|185.18.187.85|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1804076 (1.7M) [video/mp4]
Remote file exists.The output displays the HTTP response code and returns a corresponding exit code. When you add the –spider flag, the file will not be downloaded to your local device.
To analyze the HTTP server response header, use -S (–server-response). It prints the headers sent by HTTP servers.
Example:
wget --spider -S 11131-8.b.cdn12.com/sample.mp4$ wget --spider -S 11131-8.b.cdn12.com/sample.mp4
Spider mode enabled. Check if remote file exists.
--2020-11-12 18:33:43-- http://11131-8.b.cdn12.com/sample.mp4
Resolving 11131-8.b.cdn12.com (11131-8.b.cdn12.com)... 185.18.187.85
Connecting to 11131-8.b.cdn12.com (11131-8.b.cdn12.com)|185.18.187.85|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: ucdn
Date: Thu, 12 Nov 2020 16:33:43 GMT
Content-Type: video/mp4
Content-Length: 1804076
Connection: keep-alive
Keep-Alive: timeout=20
Last-Modified: Wed, 15 Apr 2020 17:57:18 GMT
Etag: "60df82646f4bedd777446e75f4305da8"
X-Timestamp: 1586973437.14001
X-Trans-Id: txf26a80bb83fd4b65a41a5-005faa4661
X-Ureq-ID: PYMqMNZBGwPmp6cdWqncDzsChZnI8disIdWORxxlVNiLHHWtRkUmsP1hPw+7jbLt3xmreJWzYS+QTmSGimXc8YzopvLlYreFM8rb7FCXpCtrWZtZn2E=
X-Served-From: l1
Expires: Thu, 22 Apr 2021 15:36:08 GMT
Cache-Control: max-age=13906945
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: HEAD, GET, OPTIONS
Accept-Ranges: bytes
Length: 1804076 (1.7M) [video/mp4]
Remote file exists.
The output is useful to check any headers like Last_Modified, Etag, X-Served-From, Cache-Control, etc.
Wget can be used to override headers otherwise generated automatically. This example instructs Wget to connect to localhost, but to specify “foobar” in the “Host” header:
wget --header="Host: foobar" http://example.com/
The same thing can be accomplished with the cURL command-line tool:
curl -IL --header "Host: foobar" http://example.com/
Wget can show the average speed with which we download a file. In the example below the speed as you can see is “3.79MB/s”.
Example:
wget -O /dev/null 11131-8.b.cdn12.com/sample.mp4$ wget -O /dev/null 11131-8.b.cdn12.com/sample.mp4
–2020-11-13 01:45:43– http://11131-8.b.cdn12.com/sample640.mp4
Resolving 11131-8.b.cdn12.com (11131-8.b.cdn12.com)… 185.18.187.85
Connecting to 11131-8.b.cdn12.com (11131-8.b.cdn12.com)|185.18.187.85|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 1804076 (1.7M) Saving to: ‘/dev/null’
/dev/null 100%[==========================================>] 1.72M 3.79MB/s in 0.5s
2020-11-13 01:45:44 (3.79 MB/s) – ‘/dev/null’ saved [1804076/1804076]