Web Proxies – What They Are, Why They Fail and What To Do

Introduction

We explain what a “Web Proxy” is, what it’s meant to do for us and what happens to our high speed broadband connection when it fails.
Proxy servers

Contents

Objective

By the end of this article, you should be familiar with web proxy servers. You will understand the reasons why they are there, what happens when they fail and some solutions for when they do.

Cutting to the Chase

If you’re not interested in the finer points below, but just want to find a free proxy, search Google to find a free web proxy server.

I can’t give specific server addresses, but there are several that operate in the UK and improve web browsing considerably when compared to some ISP´s mandatorary “server”.

For the full scoop, read on…

Overview

For those of you having problems with your web proxy (for some ISP´s, this is not optional and automatically intercepts all port 80 traffic), the following is a “high-level” explanation of what is causing the problem & suggestions of short-term fixes.

In An Ideal World..

To provide a balanced view of the issue of proxies, this section gives an outline of what should be the advantages of using web proxies.

Proxy servers/caches can be beneficial to browsing experience by decreasing the “fetch time” for pages in distant locations. This asumes, however, that the proxy server is configured correctly, has ample bandwidth and hardware resources. In many instances, ISP´s do have their proxies working correctly and therefore it may be beneficial to use. However, some ISP´s do not have adequate provision for proxy services. It is for these servers that this article is mainly geared.

The Problem & Symptoms

Problems with web “proxy” servers (or caches) can be observed by user-percieved problems such as:

  • Slow web browsing.
  • Can’t connect to web sites (e.g. type in http://www.xyz.com, result is an error message in your browser). For this type of problem, most users are able to connect to the same website using another ISP.
  • Updates to web pages not visible immediately. (e.g. you upload a new web page to your web-server BUT your browser still shows the old version).

The Cause

ISP´s in an effort to control & minimise network traffic sometimes “kindly?” introduce a “transparent” web proxy server for it’s customers.

The “proxy” [cache] intercepts most of the requests you make to the Internet via your web-browser. For example, when you type in http://www.microsoft.com, the request for content is initially serviced by the ISP´s proxy server. If someone has recently requested http://www.microsoft.com, your content request is fulfilled by the ISP´s proxy server based on a page that has been retrieved previously. The idea is that, rather than go half way around the planet for your content, it’s faster to fetch it “locally” from the proxy server.

Proxy Benefits to ISP’s

The “official” line on a proxy server will likely be along the lines of “..improving speed for our customers” etc. Although this can be true in most instances, there are occasions when the reality is somewhat different. The basis for introducing “caching” into a network is primarily to assist ISP´s reduce overall bandwidth requirements (and therefore cost).

Proxy Benefits to Users (Theoretical)

In “theory”, it should be faster to fetch web page content locally from a proxy server. This “theory” hold true when based on 2 fundamental assumptions:

  1. The nature of the page is “Static”. By “Static” we mean that content is non-data driven and does not change on a frequent basis. A good example of such a page is any page ending in “.htm” or “html” extensions.
  2. The proxy server is configured correctly and has sufficient hardware & bandwidth resources to service all requests.

Proxy Benefits to Users (Actual)

Here’s where we get to see why the ISP proxy server causes problems in many instances. Firstly, let’s look at the two points above to see how our (theoretical) proxy server is doing:

1) “Static” Web Pages.

Several years ago the Internet (and in particular the World Wide Web (WWW)) was a “static” place. Most web pages were “qhard coded” with text and graphics and only changed when web-authors uploaded new pages to web servers.  Today’s Internet is vastly different to those static days. We now have web-pages that are generated “qon-the-fly” based on up to the minute information such as share prices, flight information etc.  For such “dynamic” web pages, proxy servers caching ever changing content can be disastrous. For example, what use is a “cached” share price 30 minutes old?

2) Configuration & Resources

Proxies are, like every other server, dependent on hardware resources such as processor power, memory, disk speed etc. As loading increases, so the proxies speed & ability to service requests decreases. At it’s worst, this means that you may experience a “bottle-neck” in your connection to the web if using an overloaded proxy.

Work-Arounds

OK, so lets assume we want to do something about our ISP´s proxy. We have two options which are within user control:

1) Use another (working properly) proxy server.

Given that the ISP´s proxy is intercepting port 80 traffic, if we have our web content delivered on another port, then our ISP proxy server will be excluded from the picture. Here’s how to do just that:

Finding and Configuring a Public Proxy.

To test the speed & reliability from your connection, from MS Windows, do the following:

  1. For Windows XP – Start, Run. Type “Cmd” into the box and hit return.
  2. Type “Ping xx.xx.xx.xx” where the xx’s are the “IP” address (e.g. 192.154.6.100) on the proxy list. Check that you are not receiving any “packet loss” (expressed as a percentage). If there is packet loss, move onto the next IP in the list. From the list you’ve tested (filtered down to remove servers with “packet loss”, choose the server with the lowest average PING and note down the IP address).
  3. You’re nearly done! All that remains is to enter the server information into your browser configuration. Here’s how for Internet Explorer & Firefox:IE
    • With IE running, click “Tools”, “Options”.
    • You’ll see some “tabs”. Click “Connections” and “LAN Settings” at the bottom of the page.
    • In the section that says “Proxy server”, check the box that says “Use a proxy server”. Type in your number as noted above.

    Firefox

    • With Firefox running, click “Tools”, “Options”
    • Click the “General” button
    • Click “Conection Settings” button
    • Use the option for Manual proxy configuration
    • In the “HTTP proxy” box, type the number you noted down above.

You’re done! Crank up your browser and surf (hopefully) with increased reliability.

2.) Fool the ISP proxy into giving you “Fresh” Content.

This work-around requires more ongoing effort. As mentioned above, the proxy server will decide whether you are requesting “new” pages or previously cached content. If the proxy decides you are requesting a fresh page, it simply passes on the request to the “live” server.

Here’s how to fool the proxy into thinking that your request is for “fresh” content:

At the end of the web address you want, (e.g. www.bbc.co.uk) append a “?” immediately followed by a random string of characters (e.g. tgfd23). The complete URL you type in will look something like “http://www.bbc.co.uk?tgfd23” The bit after the question mark can be anything you want. The bit after the “?”q is known as the “QueryString” and is one way of passing information between “dynamic” web pages.

Needless to say, all this can be a bit inconvenient and perhaps is only of use as a “last resort” if you believe a proxy server to be delaying a specific page. A better long term solution is to configure your browser to use another proxy server.

Conclusion

In a nutshell:

  • Some ISP´s provide a web proxy that, unless you specify otherwise, you are forced to use.
  • This proxy server is sometimes NOT there for your benefit. It is installed to control bandwidth & traffic for the ISP.
  • Some ISP´s proxies are not reliable (hence slow web access, DNS failure from time to time).
  • To bypass the ISP proxy, select an alternative (working) proxy on a port other than 80.
  • “Fool” the ISP proxy into giving you “fresh” content by appending a random string to the URL (e.g. http://www.bbc.co.uk?tgfd23).
If you enjoyed this post, please consider subscribing to the RSS feed to have future articles delivered to your email box or feed reader.