Category Archives: Shell scripting

Percent signs in crontab

As this little-known ‘feature’ of cron has now bitten me several times I thought I should write a note about it both so I’m more likely to remember in future, but also so that other people can learn about it. I remember a few years ago when I was working for Webfusion we had some cronjobs to maintain the databases and had some error message that kept popping up that we wanted to remove periodically. We set up a command looking something like:

but it was not executing. Following on from that, today I had some code to automatically create snapshots of a certain btrfs filesystem (however I recommend that for serious snapshotting you use the excellent (if a bit hard to use) snapper tool):

But it was not executing… Looking at the syslog output we see that cron is running a truncated version of it:

Looking in the crontab manual we see:

D’oh. Fortunately the fix is simple:

I’m yet to meet anyone who is using this feature to pipe data into a process run from crontab. I’m also yet to meet even very experienced sysadmins who have noticed this behaviour making this a pretty good interview question for a know-it-all sysadmin candidate!

Using ImageMagick to manipulate PNGs stably

This seems to be an issue that has been talked about in a number of places, however I found it very hard to find the correct solution, which is why I have documented it here.

Often as part of the build process for a webapp you’ll want to take original images and shrink them down to be the correct dimensions (either because they require certain dimensions to be accepted, such as icons, or because you want to save space by stripping out unnecessary data). For JPGs you can do this pretty easily like

The -strip removes any EXIF header information both anonymizing the image and saving potentially a few Kb of asset size.

This process is ‘stable’ because if you repeat it (within the same version of ImageMagick), the resulting file’s data will be identical. This means that you won’t get a new version of the built image in your (git) repository each time you run this command.

However recently when trying to do the same for PNGs (because I required transparency) I noticed that each time they were being built, git was committing a new version into the repository. This is bad news because it both grows the size of the repository by storing pointless identical versions of the file, and also makes it a lot harder tracking through history to see what changed because you have loads of PNG images being committed each time you do a build.

Looking at the output of identify -verbose I could see that the part that was changing each time was below:

So it appears that PNG format wants to store the update/create time in the image’s header itself. That was what was changing each time.

Searching on the internet I found a number of suggestions about how to strip these out with the convert command, and I saw that the header changed a bit but I couldn’t find any that were also removing the ‘png:tIME’ element. Finally I managed to come up with the following flags which convert the image stably:

The identify command still outputs the date: property sections but these are now being taken from the create time (ctime) and modify time (mtime) of the file itself rather than from the header and so are not stored in version control.

You might be wondering why I don’t just create a lazy build system that only updates the asset if the mod time of the source asset is greater than that of the built asset – if I was doing this on a bigger project that would be the best way, but as this was just for a small project I wanted to do quickly I thought that doing this would be the easiest way!

Convert emf files to png/jpg format on Linux

For a project recently I was sent some excel files with images embedded in them. Not fun. Then I discovered that these were in some random windows format of emf or wmf (depending on whether I exported as .xlsx or .ods from libreoffice) which I think was just wrapping a jpg/png file into some vector/clipart format. Fortunately there’s a great script called unoconv that uses bindings into libreoffice/openoffice to render pretty much anything, however it doesnt seem possible to change page size/resolution. If you use the PDF output though you can get the image simply embedded in the PDF, then use the pdfimages command to extract the original images out of there. Finally some of these had different white borders so I cropped these and converted to png. Full commands below:

Multi-line commands with comments in bash

As part of the last post I initially used a bash script to generate the commands to output the individual videos. As per usual, when I finally got fed up of the limitations and syntax issues in bash I switched to a proper programming language, perl. However this time I learnt a neat trick to doing multi-line commands in bash with comments embedded using the array feature of bash. A multi-line command typically looks like:

However what if you want to add comments into the command? You can’t.

To solve this create an array:

and then use the following magic to execute it:

Using this you can also conditionally add in extra statements if you’re using a pipeline-type program such as imagemagick (convert) or melt: