Testing the Performance of the Linux Firewall

Over on the Strongarm blog I’ve got an in-depth article about testing the performance of the Linux firewall. We knew it was fast, but how fast? The answer is perhaps surprisingly “significantly faster than the kernel’s own packet handling” – blocking packets to an un-bound UDP port was 2.5* faster than accepting them, and in that case a single-core EC2 server managed to process almost 300kpps. We also tested the performance of blocking DDoS attacks using ipset and/or string matching.

Linux Stateless Firewalling for High Performance

I’m currently doing a fun bit of consulting on high performance Linux with a great company called Strongarm. I’ve written a post on their blog about we went about adapting a standard linux firewall to make it much more efficient and less resilient to DDoS attack. In short, remove the connection tracking modules and easily do it yourself – but watch out for hidden traps especially on the AWS EC2 platform because it uses jumbo frames!

Manually changing ASUS wireless router firmware versions

I recently got a Asus RT-AC3200 wireless router for a project I’m experimenting with. Naturally, the first thing I did was to flash it with some custom firmware, in this case DD WRT which worked fine but there was a feature missing that I needed. I then wanted to change back to the stock firmware, however trying to use the standard upgrade method didn’t work for some reason. Asus offers a recovery tool via their website but only for windows. But, looking in the forums it turns out that it’s pretty simple to do this manually (and on linux).

First, turn the router off. Then turn it on while holding the reset button for a few seconds. It will then boot up into a recovery mode where you can ping it and send files to it but that’s about all. So, plug your computer in, manually configure your (wired) interface to (I don’t know if this is actually required or not) and run:

Then reboot your router and the firmware has been forcibly updated! It’s probably worth to do a ‘factory reset’ after installing any new firmware as the nvram differs between different versions.

Maximizing rsync throughput in 2 easy commands

rsync is an excellent tool for linux to copy files between different systems. However, it doesn’t yet have the ability to run multiple copy processes in parallel which means that if you are limited by the speed you can read filesystem metadata (ie list the files), or you have a very fast network connection and lots of cores on both servers you can significantly speed up copying files by running processes in parallel. For example, one process can copy files at perhaps 50MB/sec, however with a 16-core server, 1gbps network connection and a fast SSD array you can copy data at 1GB/sec (gigabytes). Here’s how:

Firstly, you need to get ssh set up so you can connect between the machines without using a password. Even if you are copying between two remote systems and you use ssh-agent key forwarding (which I highly recommend), this can become a significant bottleneck so it’s best to do the following and generate a new key on the source system:

Hit enter when it prompts for a passphrase so that the key is generated without needing a password to open it. This will create two files, rsync which is your private key and rsync.pub which you want to add to your authorized keys on the remote host using something like:

You should then be able to ssh without needing a password by doing:

Next, we need to go to the remote host and allow lots of ssh sessions to be opened at once; open up /etc/ssh/sshd_config on remote_host and append or change these lines:

Now, on your source host run the following command to ensure that rsync uses the ssh key you just created:

Now for the good stuff – first we need to mirror the directory structure but not the files:

And now we can do the copy in parallel (you might need to install the parallel command using something like apt-get install parallel):

This will copy 30 files in parallel, using compression. Play with the exact number, but 1.5 times the number of cores in your boxes should be enough. You can monitor the disk bandwidth with iostat -mx or the network throughput with a tool like iptraf. One of those, or the CPU usage should now be saturated, and your copy should be going as fast as is physically possible. You can re-run this afterwards to synchronise even quicker than a normal rsync, however you won’t be able to use it to delete files.

Running lots of postgres commands in parallel

Postgres is great, however one limitation is that you can only run one command at a time in the shell. Sometimes however when you are doing administrative functions over multiple tables, for example (re)creating indexes or vacuuming and you have a nice powerful box, you can run many of these commands in parallel for easy speedup. Here’s an easy way to run lots of commands in parallel.

Firstly, create a text file with one command per line. For example

Then, ensure that you have your .pgpass file set up correctly so that you can just run psql [database] [user] without being prompted for a password.

Finally, run the following command:

-P 20 specifies the number of jobs to run in parallel so change this to what your server can cope with.

Finding the amount of space wasted in postgres tables

Because of way that postgres handles transaction isolation (ie using MVCC), when you modify or delete a row in a table it marks it as deleted, and then frees the space at a later point in time using (auto)vacuum. However, unless you use the heavy-weight VACUUM FULL command (which exclusive locks the table and totally rewrites it, causing anything trying to access it to block until the command is finished) the space is never reclaimed by the operating system. Normally this is not a problem – if you have a heavily used table with 20mb of data in it it probably has 5-10mb of overhead with the dead rows, reclaimed free space etc which is acceptable. However there are a few situations where it is useful to know what exactly the overhead is:

  1. Sometimes if your table changes very quickly, is large, and your disks or autovacuum parameters are unable to keep up, it can end up growing massive. For example we had a table that contains 3Gb of data but was taking up 45Gb due to the fact that autovacuum couldn’t keep up with the frequency of changes in the table
  2. If you are using table partitioning to store historic data then to make the most use of space you want to see whether a VACUUM FULL would be advantageous to run or not. For example if you have a table that is recording data collected from each day, some days it may be mostly just inserts so doesn’t need vacuuming; other days it may have a number of changes made and so have quite a lot of free space that can be reclaimed. Additionally, VACUUM FULL optimizes the order of data in the table and the indexes making it more performant.

In the first case, looking at the output of a command like

(sorry I can’t remember where I found this) should show you that there are a very large number of dead tuples waiting to be reclaimed (ie turned in to free space) in the table.

However, if your disks were struggling at one point, but then you tweaked autovacuum so it reclaimed the dead tuples correctly (as in case 1 above), your table could now be 90% free space but there is no easy way to find this out within postgres.

Fortunately, there is an excellent extension called pgstattuple which allows you to find out the amount of free space within a table file that has been reclaimed but not released to the operating system. The following query lists all tables which are over 100Mb in size, and have more than 10Mb of free space and have more than 20% free space (you can tweak these numbers – I just did it for our platform where our typical table size is 1Gb+):

This only uses an approximate count, however even so it can be a bit slow (it just took 10 minutes here) on a system with many tables and heavy IO. You can use this to find the tables that would most benefit from a VACUUM FULL command being run.

A note on the e-ikamet system for foreigners

This year for the first time I got to use the e-ikamet randevu system. It’s quite nicely done, certainly better than the old system! I found two issues with it which might help others who are struggling to get an appointment:

  1. If you mistype your father’s name in the English version of the site, when you go on to the next page everything is broken. This is because the error string is automatically generated on the server-side unfortunately not escaped properly (the apostrophe in “father’s name”). This probably happens in other places too. To fix this, switch to the Turkish version of the site as they probably tested the error messages better, and Turkish doesn’t typically use apostrophes anyway.
  2. If, after completing the process you need to get a randevu (for example, my passport number appears to be incorrectly entered into their system at present), the ‘Get a randevu’ button should not be greyed out even if there are no randevu available. If it is greyed out it means that you got something wrong on your application process, but they don’t have an error message to tell you about it. In my case, I had set the date of my health insurance to expire before the expiration date of the ikamet I was trying to get. Even though there was no error displayed, when I set the insurance date to further in the future, it allowed me to get a randevu.
  3. Another bug was that the maximum size of photo upload seems to be around 500kb – I uploaded a larger passport photo (or perhaps because it was high resolution) and it only showed the top half of my head but didn’t count this as an error. Also, it only showed this when you went back to the page after the upload; not when you first added the photo. If it only shows a partial image just resize the photo and try uploading again.

Full-stack Linux development (AngularJS, Bootstrap, Modern Perl) and Life in Turkey