View HTTP Headers With wget

View http headers with wget to see things like Apache server response codes and more.

There are many tools that will allow you to see HTTP headers, such as: curl, fiddler, httpwatch, postman, tcpdump, snoop, and wireshark to name a few. If you don’t have the option to install these, wget can also help you see the HTTP headers and is available in cygwin for windows and most linux installs.

If wget isn’t installed you can use either apt, yum to install it:

Installing Wget on Debian, Ubuntu

Installing Wget on RHEL, CentOS

Installing Wget on Windows

There is a windows binary for wget, but we’ve found that Cygwin works much better and provides other useful tools as well.

Why Check HTTP Headers?

Knowing what your server is returning is useful when you need to see more than what your browser is showing you. Most browsers do not provide specific details, error codes, or other header information such as x-header info.

Here are a few troubleshooting reasons you may want to look at HTTP headers:

• Verify redirects happening as you expect, especially when troubleshooting mod_rewrite
• See if you have a permissions error, server error, or other http response code
• See if you have x-headers providing proper information back to the browser
• XFF headers will tell you if certain proxy types are forwarding/handling your request
• Verify character sets, encoding, compression or other header settings
• Viewing cookie data
• Fixing and Troubleshooting Redirect Loops
• View caching parameters in your HTTP headers, if they exist
• Why the same page or parts of the page are unreachable from different networks

There are also security implications and reconnaissance reasons to view headers:

• Verify redirects location on unknown shortened links
• View current X-XSS-Protection settings related to XSS attacks
• View X-Frame-Options (click jacking)
• View X-Content-Type-Options to harden against MIME-sniffing attacks

As you can see there are at least a few good reasons for troubleshooting and investigation that you might want to look at HTTP headers. Another big advantage of viewing HTTP headers is being able to do this from the client side. Server folks may be in a different timezone and getting a meeting together to have them look at headers and logs on the server could take weeks sometimes. You can do your own client side tests though and then email the server team what parts they should investigate based on your findings.

Wget Syntax

So how do you get on with checking those headers anyway? First look at the basic syntax of wget:

Check HTTP Headers With Wget

Looking at the help file for wget will show several pages of commands. The ones we are most interested in for looking at headers is the –server-reponse (S) and –output-document flags (O). We’ll combine them as -SO and then add the – to make sure it goes to stdout. This allows us to parse it or simply view it. You can also specify the file name if you like for later viewing.

The –timeout (T) and –tries (t) options are optional, but I usually include them by habit. We pipe it to more because it’s going to show the entire request, in addition to headers.

Here is a sample header output:

You can combine this with things like egrep to parse out something you might be looking for. For example if we wanted to see the http status code, we could do this:

HTTP Status Codes

There are various “sub” codes, but the main categories are as follows:

• 1XX (Informational)
• 2XX (Success)
• 3XX (Redirection)
• 4XX (Client Error)
• 5XX (Server Error)

wget flags for troubleshooting

While troubleshooting your http headers, you can use several of the flags to save cookies, set a user, force certain types of SSL and more:

• keep-session-cookies
• save-cookies file.txt
• load-cookies file.txt
• http-user=user
• http-password=password
• no-cache
• user-agent=
• no-check-certificate
• secure-protocol=SSLv3|TLSv1

Wget Error Codes:

• 0 No problems occurred.
• 1 Generic error code.
• 2 Parse error—for instance, when parsing command-line options, the .wgetrc or .netrc…
• 3 File I/O error.
• 4 Network failure.
• 5 SSL verification failure.
• 6 Username/password authentication failure.
• 7 Protocol errors.
• 8 Server issued an error response.

Exit codes that are lower numbered, take precedence over higher-numbered ones when multiple types of errors are encountered.

Specific HTTP Error Messages

Sometimes, a load balancer will make your connection fail. Here are some of the common load balancer related messages we’ve seen:

HTTP 400: BAD_REQUEST

The client sent a bad request. The client either isn’t speaking proper HTTP or is using a type of connect method that the load balancer doesn’t support.

HTTP 405: METHOD_NOT_ALLOWED

In AWS, the load balancer only allows a length of 127 characters in the method. You will get this error if your method has a longer length

HTTP 408: Request Timeout

The client cancelled the request or failed to send a full request. Troubleshoot things that would normally cause network issues such as an overloaded switch or firewall or congested network. It is also possible the client is simply requesting something the server won’t listen to such as a wrong port, but this is usually only the case if the server is configured to tell you it’s the wrong port and that is not standard methodology.

HTTP 502: Bad Gateway

This is likely a parsing issue on the load balancer. Confirm that your application is sending proper HTTP requests.

HTTP 503: Service Unavailable

This most likely related to overload, but could also be misconfiguration. If everything seems to be operating normally without severe load and the issue persists, it’s probably a misconfiguration.

HTTP 504: Gateway Timeout

The load balancer closed the connection because the application took too long to respond. This is most likely related to your keep-alive settings.

In most cases, simply looking at the header while on a network directly connected to the server and comparing it against the headers sent from the load balancer or other intervening device will give you a clue as to what is wrong.

Related RFCs to HTTP Troubleshooting

There is actually a “right way” to handle the HTTP protocol, regardless of what your developer says! The “right way” is documented in various RFCs and it is worth reading through them, especially if you are troubleshooting HTTP header issues:

• RFC7230 – HTTP/1.1: Message Syntax and Routing
• RFC7231 – HTTP/1.1: Semantics and Content
• RFC7232 – HTTP/1.1: Conditional Requests
• RFC7233 – HTTP/1.1: Range Requests
• RFC7234 – HTTP/1.1: Caching
• RFC7235 – HTTP/1.1: Authentication

Want Help Troubleshooting Your HTTP App?

Contact us for help with your network troubleshooting. We’d be happy to help you get things up and running again (or for the first time).