USENIX Update

I suspect that we all have a love/hate relationship with backups. Having backups done is a great feeling, but getting them to the point that you can absolutely count on them can be an exercise in frustration. Not only are they incredibly important, but the more you need them, the harder they are to do right.

In Jacob Farmer’s training, “Eliminating Backup System Bottlenecks”, I’ve been learning more about how to ensure that even large backup jobs complete in time. In time for what? Well, before the next backup job starts would be nice!

As it turns out, there are several possible bottlenecks for backup operations, and the severity of each depends on the size of your network and the sorts of data you are backing up. That’s not to mention the actual methods and devices that you use, each of which can introduce issues to the process. In small networks, bottlenecks are relatively obvious because there are only a few parts to the whole environment. In larger environments, the backup infrastructure gets considerably more complex.

Jacob divides the backup infrastructure into three parts, each of which have their own possibilities for slowing down the process:

The first place to start troubleshooting slow backups is the front end. Front end bottlenecks are caused by large volumes, static files, and slow hosts. These issues can be combated by using intelligent backup solutions rather than copying everything each time. Many backup solutions are available which only copy the daily deltas or only the pieces of large files which have changed.

Central bottlenecks should only be attacked after the front line issues have been resolved. Issues in the central area are primarily related to insufficient processing abilities of the backup servers or network data movers. In a properly designed backup solution, the actual data writers are running at full capacity. This can only happen if the devices and network feeding the data are working optimally. If data movers are being overloaded, adding more can help, as can adding a slave server if necessary.

Back end bottlenecks are complex. Slow backups after improving the front end clients and the centralized data movers are related to the media and drives used to write and store the data. Whether you use disks, tapes, or VTLs, each has its own particular quirks.

When using tape, ensure that you can feed the tape data at the maximum write speed of the drive. Feeding data any slower leads to “shoe shining” where the tape writes at max speed, which causes the tape to move past the end of the data. To write the next bit of information, the tape has to rewind, and then it writes the next chunk, overshoots again, and rewinds…repeat ad nauseam. This dramatically decreases the write speed (and lifetime) of the drive.

Because of this throughput requirement, adding tape drives will only speed up the backup if your current tape drive(s) are running at maximum speed, and if your backup server(s) are capable of increasing the current output rate. Adding slave servers to write to additional tape drives may be a more reliable way of guaranteeing output.

When dealing with backups to disk, the difficulty swings the other direction, and your target may not be fast enough. With multiple backup writers, one array could have its I/O maxed out relatively easily.

The ultimate solution is to test and document a baseline benchmark of all the individual pieces of your backup solution, and critically evaluate each against the other. Using the careful, planned approach rather than an ad hoc purchasing spree will lead to faster backups, quicker recoveries, and greater efficiency of the process.

Richard Elling gave a tutorial on the ZFS filesystem. The tutorial included a complete overview of the filesystem from administrative commands to details of how the data is stored on disk.

The tutorial started off with a brief history of filesytems on Solaris and the design goals behind ZFS. This provided a good basis for the discussion of how ZFS was implemented.

Solaris’ previous filesystem, UFS has been around since the 1980′s. It was designed for a different set of constraints. Systems had limited memory and processing power, so UFS didn’t leverage them very heavily. UFS has also had features bolted on over time. Because of this, UFS has grown unwieldy. ZFS, on the other hand, is being developed in a time where most systems have CPU cycles to spare. Memory is also plentiful with a low acquisition cost. Because ZFS was able to wipe the slate clean and start from scratch, it has been able to avoid some of the common problems associated with other filesystems and volume managers. For example, in other filesystems, the base unit of storage is the disk. There are volume managers that aggregate disks together for the filesystems, but this is a kludge at best. ZFS takes the approach that storage should be treated as a pool. All disks are added to ‘zpools’ and ZFS handles the tasks formerly performed by volume managers. This fundamental difference give ZFS significant power and flexibility.

The tutorial included a significant amount of material on each configuration parameter for zpools and zfs datasets. It also covered the tools used to manage pools and datasets. Richard reviewed how the data protection schemes available in ZFS differ from traditional RAID levels. This review of the administration tools of ZFS would be a great help for the sysadmin with no ZFS experience. After this tutorial, they would be able to create, configure and optimize ZFS.

Upcoming features like encryption and dedupe were touched upon. Both of these features were a little too fresh to be fully covered in this class. Dedupe was completed today. Encryption is scheduled to be integrated sometime in the fourth quarter of this year.

One of the most valuable parts of the tutorial was the section on performance tuning. It covered tuning for some of the most common use cases such as databases. It also covered the game changing concept of a Hybrid Storage Pool. With HSPs, the sysadmin will be able to take advantage of both SSDs and HDDs in a blazing fast combination that won’t break the bank.

The tutorial was complete and thorough. It provided data and analytical tools to allow the sysadmin to make informed decisions about protecting data on ZFS.

I certainly hope this tutorial is available at future LISA conferences. ZFS as a technology is quite young. As ZFS matures and is accepted by a larger audience, there will be a need for sysadmins well versed in its operation. I hope this tutorial will be able to continue to grow and adapt to that requirement.

Hello, thanks for reading the USENIX blog devoted to the LISA ’09 Conference! My name is Matt Simmons, and I’m one of the bloggers who will be writing here with daily updates on events at the conference.

Since I live in New Jersey, I took advantage of the extensive rail network in the North East and arrived in Baltimore Penn Station via Am-Trak this morning. I got out of my cab at the hotel and stared up in appreciation. The conference is being held at the beautiful Marriott Waterfront on the inner harbor, and even the overcast sky couldn’t take away from its impressive façade. I checked in with no issues, delivered my luggage to my room, and virtually ran downstairs to check in at the conference desk.

As luck would have it, I ran right into Pamela Howell, who is the video blogger this year. She showed me around the registration area, introduced me to the USENIX staff working the desks, and helped me get registered. By this time, it was right around noon, so rather than barge into the end of a morning session, I waited it out until the after-lunch sessions began.

I spent the afternoon in Maurita Plouff’s “Management Skills, or, Don’t Panic!” training. This class focused on developing successful workers, influence, and communicating with your management, which teaches that people can be loosely fitted into a series of personality types, and each type responds to different stimuli, and can be motivated in varying ways. Successfully managing the people who report to us, our customers, and our managers requires effectively utilizing these techniques.

In addition to telltales and watchwords of each of the personality types, Maurita wove a blanket of stories from her experiences in IT, discussing everything from messenger boys to party lines to ubiquitous computing environments. I enjoyed the depressingly short period of time I was able to attend this training, and if she offers it again next year, I wholeheartedly recommend signing up. It was a great time, and everyone enjoyed participating with tales of their own management experiences.

Tomorrow I’ll be at the “Working with SELinux” training in the morning and “IPv6: An Introduction” in the afternoon. I can’t wait.

Until next time!

-
By Matt Simmons, author of the Standalone Sysadmin blog

It’s Monday afternoon, and LISA is chugging right along! I’m well into my second day of training, and having a great time, and learning a ton that I’m going to take back to work.

After lunch, I’ve signed up for a training course that sounds great: “Beginning IPv6”. It’s something that I’ve been very vocal about in the past. I’m convinced that by the time it’s obvious that everyone should implement IPv6, it will be too late to do it effectively. The most compelling argument to me is that the emerging markets of the world are going to be coming online using IPv6, because in short order, unassigned IPv4 blocks will all be assigned. That isn’t to say that providers aren’t sitting on blocks, but that it’s going to be harder and more expensive, therefore people who can’t buy them will go with the cheaper alternative that IPv6 will offer. They’re going to be your (and my!) future customers. I want to be ready in a couple of years, so I’m going out of my way to learn now. I didn’t learn IPv4 overnight, and I suspect the case will be made for IPv6 as well.

Rudi van Drunen started his presentation not with technical graphs of dwindling IP addresses. He started it with a slide with headers that said “Numbers run out” and “I need this turtle to dance”, and indeed, there was a CGI turtle. It wasn’t dancing. The turtle is from the KAME project. If you visit http://www.kame.net/ with IPv4, the turtle just sits there, but if you connect with IPv6, you get a dancing turtle. As Rudi said, let the dancing turtle be your incentive to implement IPv6.

As I mentioned, IPv4 addresses are running out. NAT has slowed the expansion, but it won’t be enough to quell the tide. IPv4 addresses are running out, and the internet is still expanding and smart phone use is exploding. We need more and more IP addresses. At this rate, our refrigerators are never going to get on the internet. And neither will the children of tomorrow.

In “IPv6…”, we went over the major changes from the IPv4 that we all know and love. The biggest change is undoubtedly the increase in IP address size. IPv4 uses 32 bit addresses. This provides just over four billion addresses. That sounds like a lot, but the 128 bit address space of IPv6 provides for 3.04×10^38 addresses. THAT is a lot of addresses.

Other changes that will influence networking are maximum transmission units (MTUs). The reason is that the typical MTU is 1500 bytes. With the previous IPv4 header taking up a scant 20 bytes, the larger IPv6 addresses double the header to 40 bytes. At least. Unlike the old version, IPv6 headers have multiple optional header additions. Looking at them from the outside in, each header holds a description of the next header inside of it. This provides the ability of nearly limitless plugins. This also leads to large headers, which can begin to take up an increasingly greedy chunk of the 1500 byte MTU. I forsee jumbo frames becoming more and more important as IPv6 becomes more prevalent and additional layers are implemented.

Speaking of additional layers, IPv6 provides the ability to add features into the protocol itself, through the use of these flexible headers. Where as IPv4 required specific packets to build and tear down IPsec tunnels, IPv6 has an optional header for ESP, encapsulating security packets. In other words, encryption. The header itself specifies that the packet contains encrypted information, rather than relying on encryption-protocol specific packets. This is exciting, and definitely makes me wonder about what new implementations will take effect once the protocol becomes more wide spread.

Overall, I’m still convinced that IPv6 is the future, and we’re going to need to learn it. The conversion isn’t going to happen overnight, and it’s going to take time to absorb the new environment. Why not start now with a lab environment and start learning? It’s not time wasted, it’s time invested in your future.

-
By Matt Simmons, author of the Standalone Sysadmin blog