I/O stalls on Azure VMs with read-write data disk cache

A few years ago I set up our cloud instances in Azure. There are three caching modes available for premium (SSD) data disks: None, ReadOnly, and ReadWrite. I didn't read up enough on the choice, but ReadWrite seemed safe for journalling filesystems or databases with proper write-ahead logs, so I figured I'd start with it and adjust if there was a problem.

Much later we migrated back to Azure after some time away and I kept the old ReadWrite cache setting. Everything was fine for a couple of months until we introduced a feature that spilled to large temporary files. All of a sudden our apps would stop responding for a minute or more at a time. Our Docker Swarm nodes have relatively small P4 disks (32 GB), with an expected throughput of 25 MiB/sec; we'd be humming along writing temp files at a much higher rate for 20-30 seconds and then all I/O to the disk would stop for 2 or more minutes.

Windows update pipelines with PowerShell

Testing a set of Windows updates before pushing them to production is a fairly typical strategy, but WSUS doesn't provide great tools for it out of the box. With a bit of PowerShell you can generate a file containing a list of new updates, approve them for the test environment, and later approve the same set for prod.

Creating PostgreSQL bulk import files with Python

PostgreSQL supports COPY FROM for bulk loading tables from CSV and binary files on the server's filesystem, and the equivalent \copy to load the file through psql. Turn off indexes, constraints, and WAL, and COPY FROM is pretty fast. I don't know if it's the fastest way to bulk load into postgres but it sure is fast enough for me - we loaded 49 million records totalling over a terabyte in about 8 hours. It's reasonably portable across versions, too, especially if you minimize the use of binary import files.

Generating the CSV and binary formats from Python is easy enough. Here are samples for each.

MSA P2000 G3 10GbE iSCSI disk storage performance stats

I've found it's hard to come by numbers you can use when planning your storage system bandwidth, so I'm publishing stats for a modern 10GbE iSCSI array. I make no claim that the systems were perfectly tuned, and there are lots of reasons why these numbers would not apply to your environment - but I figure some data is better than no data, as long as the context is clear.

Here's the hardware setup:

ALSA automatic sample rate conversion

Recent ALSA releases - not sure how recent, I'm using 1.0.13 - enable software mixing by default. This is fantastic, because it means an infinite number of properly written applications can record and play at the same time.

Fun with defaults

Last week a coworker called me in for some help with an IPsec tunnel to Jamaica. Our side is Openswan, theirs is Checkpoint.

The tunnels were coming up as far as Openswan was concerned, but Checkpoint was rejecting the traffic. Fortunately the guys in Jamaica knew their stuff better than me and were able to point us in the right direction: Checkpoint was configured to use 3DES-MD5, but Openswan had brought up a tunnel with 3DES-SHA1. We told Openswan to use 3DES-MD5 on this tunnel, and the problem went away.

This raises a few points.

1. How did two IKE-compliant products manage to negotiate incompatible tunnels? Well, whatever. Stuff happens.



2. Checkpoint's logging must be pretty good. It told the operators exactly what we needed to know: Openswan was using an unexpected hash method.



3. Most importantly, it's 2007 and there are still people using 3DES and MD5 for tunnels.

In software, AES-256 is at least three times as fast as 3DES. (Run "openssl speed des-ede3 aes-256-cbc".) The effective key length of 3DES is 112 bits, so AES-256 is three times as fast and one billionty times harder to break. Not much of a tradeoff there. If your security gateway is doing 3DES in hardware and is too old to handle AES in hardware, you might as well switch to software -- if you had enough traffic to justify hardware encryption, you'd probably have enough money to buy new hardware.

So 3DES is getting old, but it's still awfully hard to break. Not MD5. MD5 is just dangerous. It's trubs. Even SHA1 is starting to look scared.

I think it's pretty silly that IPsec implementations are still defaulting to 3DES and MD5.

Goog!

I found out about Google's "My Maps" from a blog linked from lolcode. You can overlay maps with placeholders and shapes, and as far as I can tell it even remembers the way you have the map positioned when you send a link around. We were testing some 400 MHz radio modems the other day so I gave it a try. All it's missing is a way to measure line-of-sight distance without using a ruler.