This list of frequently asked technical questions about the NZ internet filtering/censorship scheme. It is a partner to the more general FAQ.
It will be updated periodically with new questions and better information (last update 16/03/2010).
Glossary
DIA – Department of Internal Affairs
OIA – Official Information Act
ISP – Internet Service Provider (e.g. Telecom/Xtra)
Internet address – IP address
How does the filtering work?
Setup
The filter server (Netclean Whitebox) is installed and has a list of banned sites loaded on to it.
The ISP and the DIA set up a secure tunnel between the filter server and the ISP’s routers.
The ISP’s routers have the filter server configured as an external BGP neighbour.
Operation
The filter server does a DNS lookup for each website on the list to get the internet address.
The filter server uses BGP over the secure tunnel to advertise to the ISPs a route to each individual filtered internet address (a /32 route). This tells the ISPs to send all traffic for that internet address to the filter server.
When a user requests a web page there are three possible outcomes:
- The requested website is not on one of the filtered internet addresses. The request is made through the internet as usual and never goes through the filter server.
- The requested website is banned and is therefore on one of the filtered internet addresses. The request is diverted to the filter server which sees that the URL is banned and an “Access is refused” page is returned. The internet address of the requesting computer is logged.
- The requested website is not banned but is on the same server as another website that is banned and they share the same internet address. The request is diverted to the filter server which sees that the URL is not banned so it forwards the request to the web server. The internet address of the requesting computer is not logged.
All other non-web traffic on banned hosts is also forwarded to the filter server. This other traffic is not examined and is passed on to the destination.
You can get Netclean’s own description of their system from their Technical Description paper.
What happens if the website switches to a new internet address?
The system periodically refreshes the mappings between website and internet address (i.e. does a new DNS lookup). Netclean recommend doing this daily.
Does it support the next version of IP, v6?
No.
At what level is the filtering applied?
The filters can be applied at three levels:
- Internet address
- Website
- Parts of a website
It is possible to filter down to the level of folders or even individual documents and images on a website. E.g. you could filter http://www.website.com/badcontent but allow http://www.website.com/goodcontent on the same website.
What happens when there are multiple websites on one internet address?
The requests for all of the websites are diverted to the filter server. The filter server receives the requests and looks at the URL to determine whether that site is banned or not. If it is banned a “This website is banned” message is returned, otherwise the filter server forwards the request to the web server.
If a request is made for a non-banned website on an internet address that also has banned websites, does the request still go through the DIA system?
Yes.
Does the filter server get both the request to and the response from the website?
No.
If the filter server passes the request to the web site (i.e. it’s to a non banned website on a filtered internet address), the response from the website goes straight to the user without passing through the filter server.
What if the website uses HTTPS (secure HTTP)?
If the website uses https (e.g. as used for internet banking or online shopping), the filter server can’t examine the request to see what website it is going to on the target internet address.
This means the the filter server must either block all https websites on a filtered internet address (thus breaking some legitimate sites) or allow all requests through. The DIA have stated that the traffic will be allowed to pass through.
How hard is it to implement HTTPS (secure HTTP) for a website?
Implementing HTTPS on a webserver is a matter of generating a certificate (a non-official one is free and easy) and then enabling it for that site.
Will the filtering cause any performance issues?
Requests to blocked websites will, of course, not be available.
Requests to websites that aren’t blocked but are on the same internet addresses as blocked websites might be slower as they will have to pass through the filter server.
Requests to websites that aren’t blocked and are not on the same internet addresses will not be affected.
Does the filtering include all traffic to the internet address or only web traffic?
All traffic for that internet address is forwarded to the filter server. This includes web, email, chat, and P2P file sharing.
The Netclean Whitebox is designed to only filter website traffic. All other traffic will be passed through the filter.
Where will the system be installed?
The system will be installed in Auckland, Wellington and Christchurch. Initially all traffic will be filtered in Auckland with the other two sites only used if the Auckland site fails. The intention is to eventually share the load between the three servers.
What sort of Internet connection does the filter server have?
The filter server will be connected to the Internet using a fibre optic cable at 100Mb/sec, costing $2000 per month.
Comparing this to commercially available connections, it sounds like it will be a connection through the Wellington Citylink network and that price will probably include 5-10Mbps of Internet bandwidth.
What information is logged by the system?
The ISP does not need to log any information relating to the filtering (although they could choose to).
The DIA’s filter server logs the internet address of any computer that tries to access a blocked site. This is kept for up to 30 days and then deleted. The DIA say that the filter logs will not be used for law enforcement.
The filter server does not log the internet address of a computer that tries to access a non-blocked site on the same internet address as a blocked one, even though this request passes through the filter server.
What does an ISP have to do to enable the filtering?
The ISP must do two things:
- Set up a secure connection between their routers and the DIA’s filter server so that the ISP can receive the routes to filter. This tunnel can use either GRE or IPIP.
- Set the filtering server as an external BGP neighbour on their routers. This means that the ISP will comply with the alternate routes advertised by the filter server.
Does this put any load on the ISP’s systems?
The secure connection between the routers and the filter server will not put any significant load on the ISP’s systems.
A typical NZ ISP stores about 290,000 routes on their routers. The filtering adds (at the moment) an additional 7000 routes (one for each banned site). While this does add more load (and possibly disproportionately more due to them being so specific) it would seem that most existing equipment should be able to cope.
However, this is assuming that the ISP will apply the filter to all of its customers. If the ISP wishes to offer separate filtered and unfiltered feeds they will need to put in a significant investment to duplicate/upgrade their core internet routing infrastructure.
Is there any cost to the ISP?
If the ISP will filter all of its connections the ISP does not need to buy any additional equipment. Adding the secure tunnel and setting up the filter server as an external BGP neighbour would take a small amount of engineering and testing time.
However, there would be a significant cost in time and equipment if the ISP wished to offer both filtered and unfiltered internet connections.
Can an ISP offer both filtered and unfiltered connections?
ISPs could choose to offer both filtered and unfiltered connections.
However, while implementing the filter for all connections will be quite easy, it would get a lot more complex if they wished to offer both filtered and unfiltered internet connections. If the ISP wishes to offer this they will need to put in a significant investment to duplicate/upgrade their core internet routing infrastructure.
Does the ISP know which sites are blocked?
The ISP knows which internet addresses will be forwarded to the filter server, but they do not know which of the websites at that internet address are filtered and which are allowed.
It would be possible for an ISP to look at their transparent proxy logs to see which visited sites were blocked by the filter (and therefore on the list).
Is it possible for an internet user to find out what sites are blocked?
The only way for an internet user to find out whether a website is blocked is by trying to access it.
In theory it would be possible to work out which sites are blocked by sending requests to a range of internet addresses and seeing which requests get redirected to the DIA’s filter server.
What software/hardware is being used by the DIA?
Netclean sell their Netclean Whitebox product as an “appliance”. This means that it includes both hardware and software with the aim being that it can just be plugged in and configured.
The operating system is FreeBSD. The Quagga BGP daemon is used.
How well does the Netclean Whitebox scale to heavier loads?
The capacity of the filtering system to handle intercepted traffic depends on the performance of the filter server and the available bandwidth of the filter server’s internet connection (i.e. the DIA’s internet connection).
Netclean claim that their system is highly scalable by adding additional or faster servers. Their marketing includes the following:
“NetClean WhiteBox runs on one of the fastest high speed networks in the world, SUNET in Sweden. This runs at 10Gb/s and supports over 350,000 users. It has no effect on the performance of their Internet access.”
For the year ending 2007, approximately 3.1 million New Zealanders over the age of 10 had access to the internet (Nielsen Broadband Report 2008).
What scale was the DIA’s testing?
They did the testing in three phases. In the final phase they were filtering access for 600,000 Internet users.
“Over the three month period for Phase 3 the system processed on average 40 million general requests at its peak the general requests reached approximately 100 million. … the system filtered access to on average 100,000 requests per month at its peak it was processing 20,000 requests per week for 1 provider.” [sic]
How well did the DIA system scale?
According to the DIA report, the system was at up to 80% capacity during the third phase of testing. The system did experience stability issues and required hardware maintenance twice.
Apparently the live system has significantly more capacity.
What happens if the filter server breaks?
Netclean claim that internet access will continue except that no traffic will be filtered.
Is it possible to circumvent the filtering?
It is relatively easy for a motivated user to circumvent the filtering. This is done by routing the requests to a proxy service in another country that does not filter the required site.
There are also a number of free services that exist to allow people to escape from government monitoring of their internet usage. These services include: Tor, Freenet and WASTE.
This FAQ is in the public domain. As the FAQ is continually being updated as new information comes to light, I suggest including a link back to it if you use any part of it.