USENIX Update

Wednesday marked the beginning of the actual conference, as opposed to just the workshops and training sessions of the first few days. The attendance was much larger than earlier in the week, with people coming for the technical sessions and invited paper talks, and vendors have their booths set up to talk about their latest projects. As full as the schedule of the past few days have been, today it’s near the bursting point.

Rather than attend the opening session, I opted to sit in on “Linux Performance Tuning” training session presented by Theodore Ts’o. While the opening featured a keynote by Werner Vogels, I feel like tuning specific bits of the Linux OS is something that has been a gap in my knowledge for too long. This training session afforded me an excellent opportunity to learn from the best. Ted Ts’o is a key developer on the ext4 file system, he has been a kernel developer since the beginning of the Linux kernel, and works for the IBM Linux Technology Center. It’s sort of an understatement to say that he’s qualified to teach this class.

Over the long development of Linux, performance has increased as code became more efficient, but also because behavior of the software working with the hardware has improved. Through time, hardware has increased in speed, but also in the number of configurable settings, which provides the operating systems new and interesting ways of tweaking performance.

Tuning Linux is hard, because potential system bottlenecks are everywhere. What’s worse, certain bottlenecks often present as entirely different issues. Memory under-performance is notorious for this, so learning the tools for proper diagnosis is very important.

Ted started out by working with three tools that nearly everyone uses, but probably not to their fullest extent: free, top, and iostat. The first, ‘free’, displays memory utilization while ‘top’ displays an updating screen with a system summary at top and list of recently active programs at the bottom. The command ‘iostat’, part of the sysstat package, provides output regarding I/O performance of the system and the various storage attached. There are lots of columns out there for increasing the sophistication of your use of the Linux commands: top, free, and iostat. Make sure to read the man pages for each of these as well.

It should be noted that regardless of the tool used to evaluate your system, the impact of running the tool is not zero. There is always some resource utilization by the evaluation tool itself, and depending on the tool, it can be nontrivial. Don’t let this stop you, but only be aware that by watching the output, you are affecting the output, as well.

The most interesting parts of the class were the bits of information that Ted added from his experience working with file systems and storage. These included great pieces of advice, like for the next year or so, at least, don’t buy a laptop with a solid state drive. The drives that manufacturers insert have poor performance and high prices. Instead, get a spindle drive and replace it after market with a good (and somewhat cheaper) solid state drive. Another great tip was that if you are using local storage and it’s not solid state, you can partition your drive to take advantage of the fact that the outside of the spinning disk moves much faster than the inside. Put the file systems that you need fastest read/write access to first to take advantage of this. In fact, Ted suggested that using a fast terabyte drive, and only partitioning the first 300GB or so would lead to performance in the range of solid state media. This class was a wealth of knowledge!

Also covered were network tuning, which can be used to improve performance in extremely fast or extremely slow networks, and file system tuning, which can help performance when dealing with special cases such as large files, small files, quick access, and the like. In addition, Ted devoted a section to NFS tuning, which is in many ways, a combination of disk tuning, network tuning, memory tuning, and the addition of several protocol options which can be tweaked. The good news for NFS is that modern defaults are very good, and should only be changed when clients are atypical, although Ted did suggest using the ‘intr’ to allow you to recover more gracefully, should the NFS server die.

Overall, this was definitely one of, if not the, most valuable training that I took this week. I’m not an expert, but I have a lot more tools and knowledge, and as a mostly-linux administrator, I can tell you that some things are probably going to improve in my infrastructure as a direct result of being in this class. It is a tragedy that it was only a half day.

I’m not blogging to do a sales pitch, but let me just say this. If you have considered coming to LISA, but weren’t sure or couldn’t provide justification, take my word that this class, by itself, was more than worth the entire trip.

-
By Matt Simmons, author of the Standalone Sysadmin blog

Werner Vogels, CTO of Amazon.com, was the keynote speaker at this year’s LISA conference. His keynote was entitled “Ahead in the Clouds”.

One of Werner’s goals in this talk was to level set the audience with regard to the term “Cloud Computing”. He wanted to make sure we understood the reality today of cloud computing versus all the hype.

Werner began with a movie produced by a company called Animoto. This is a company that doesn’t own any server hardware. Their entire server infrastructure is hosted by Amazon. As a matter of fact, when they first started they were just a bunch of guys with laptops sitting in the corner coffee shop. He used this case study as an example of the power of Infrastructure as a Service. Their business model depended upon the rapid provisioning of servers as their customer base grew. They needed their operating expenses to to line up directly with customer use. If their company followed the traditional capital purchasing cycle, two things would have been different. First, they would have been required to seek funding from something like a venture capitalist, diluting their ownership of the company. Second, they could not have scaled to meet customer demand. Since they were using IaaS, they were able to rapidly scale as the customer demand increased.

He wanted to make one thing very clear. Amazon is not in the IaaS business in order to utilize surplus server capacity. They are in the IaaS business because they believe that it could be larger than Amazon.com. I think that this should signal system administrators to watch trends in the Infrastructure as a Service space. If this type of computing infrastructure will scale so large that Amazon’s piece of it is larger than Amazon.com, it will have a tremendous impact on our profession.

As I have discussed in the past there is a bit of uncertainty with the role of system administrators in the cloud. Werner’s view of this issue is that the job of the sysadmin will have to change. They will be required to perform more tasks with scripting and automation rather that interaction at the command line. They will need to be able to affect change whether the change is on one system, one thousand or ten thousand. In particular, the system administrator has new automation not previously available. The provisioning process is now available as an API.

There were several precursors that have allowed us to move in to the era of IaaS: Software as a Service, Distributed Computing, Virtualization, Service Oriented Architecture.

When we talk about SaaS as a precursor to IaaS, we are not talking about SaaS as a business model. Rather we are talking about how we can deliver a software solution as a platform and operate it. Previously, software was something that was authored, packaged, and shipped in a box.

The advances in distributed computing also contribute to the success of IaaS. We must be able to divide complex problems and distribute across many servers. Otherwise our biggest success would be limited by the size of the largest server we could build.

Virtualization is a critical factor in the success of IaaS. Virtualization in this regard is not simply the virtualization of compute resources. It is the virtualization of compute, storage, network, and I/O. We must be able to manage each of these resources discretely in order to meet SLAs and economies of scale needed by IaaS.

The last precursor is Service Oriented Architecture. This is a subtle, but very powerful, tool. Once we commit to SOA, services may be developed separately and loosely coupled. This helps us minimize or eliminate one of the largest expenses in enterprise I/T shops today: integration.

Werner points out that when transitioning from a traditional model to an IaaS model, we must take care to address uncertainty. In the past, developers and operators have hoarded resources such as servers and storage “just in case” there is a need for them in the future. It is a cultural change for them to trust that they may quickly, easily, and reliably provision and release resources when needed.

Amazon has an interesting perspective on organization. Werner noted that the organization matched their service architecture. There are different business units responsible for different internal services.

With a rather dramatic photograph of a data center destroyed by a tornado, Werner emphasised that if you need to provide end to end reliability to your customer, you must have geographic diversity.

I found Werner’s keynote speech engaging and informational. I am encouraged that the CTO of one of the largest players in the IaaS field has a clear direction. I hope that in the future they can continue to provide products higher and higher in the application stack, allowing system administrators to spend less time on infrastructure and more time on providing direct value to our customers.

I suspect that we all have a love/hate relationship with backups. Having backups done is a great feeling, but getting them to the point that you can absolutely count on them can be an exercise in frustration. Not only are they incredibly important, but the more you need them, the harder they are to do right.

In Jacob Farmer’s training, “Eliminating Backup System Bottlenecks”, I’ve been learning more about how to ensure that even large backup jobs complete in time. In time for what? Well, before the next backup job starts would be nice!

As it turns out, there are several possible bottlenecks for backup operations, and the severity of each depends on the size of your network and the sorts of data you are backing up. That’s not to mention the actual methods and devices that you use, each of which can introduce issues to the process. In small networks, bottlenecks are relatively obvious because there are only a few parts to the whole environment. In larger environments, the backup infrastructure gets considerably more complex.

Jacob divides the backup infrastructure into three parts, each of which have their own possibilities for slowing down the process:

The first place to start troubleshooting slow backups is the front end. Front end bottlenecks are caused by large volumes, static files, and slow hosts. These issues can be combated by using intelligent backup solutions rather than copying everything each time. Many backup solutions are available which only copy the daily deltas or only the pieces of large files which have changed.

Central bottlenecks should only be attacked after the front line issues have been resolved. Issues in the central area are primarily related to insufficient processing abilities of the backup servers or network data movers. In a properly designed backup solution, the actual data writers are running at full capacity. This can only happen if the devices and network feeding the data are working optimally. If data movers are being overloaded, adding more can help, as can adding a slave server if necessary.

Back end bottlenecks are complex. Slow backups after improving the front end clients and the centralized data movers are related to the media and drives used to write and store the data. Whether you use disks, tapes, or VTLs, each has its own particular quirks.

When using tape, ensure that you can feed the tape data at the maximum write speed of the drive. Feeding data any slower leads to “shoe shining” where the tape writes at max speed, which causes the tape to move past the end of the data. To write the next bit of information, the tape has to rewind, and then it writes the next chunk, overshoots again, and rewinds…repeat ad nauseam. This dramatically decreases the write speed (and lifetime) of the drive.

Because of this throughput requirement, adding tape drives will only speed up the backup if your current tape drive(s) are running at maximum speed, and if your backup server(s) are capable of increasing the current output rate. Adding slave servers to write to additional tape drives may be a more reliable way of guaranteeing output.

When dealing with backups to disk, the difficulty swings the other direction, and your target may not be fast enough. With multiple backup writers, one array could have its I/O maxed out relatively easily.

The ultimate solution is to test and document a baseline benchmark of all the individual pieces of your backup solution, and critically evaluate each against the other. Using the careful, planned approach rather than an ad hoc purchasing spree will lead to faster backups, quicker recoveries, and greater efficiency of the process.

Richard Elling gave a tutorial on the ZFS filesystem. The tutorial included a complete overview of the filesystem from administrative commands to details of how the data is stored on disk.

The tutorial started off with a brief history of filesytems on Solaris and the design goals behind ZFS. This provided a good basis for the discussion of how ZFS was implemented.

Solaris’ previous filesystem, UFS has been around since the 1980′s. It was designed for a different set of constraints. Systems had limited memory and processing power, so UFS didn’t leverage them very heavily. UFS has also had features bolted on over time. Because of this, UFS has grown unwieldy. ZFS, on the other hand, is being developed in a time where most systems have CPU cycles to spare. Memory is also plentiful with a low acquisition cost. Because ZFS was able to wipe the slate clean and start from scratch, it has been able to avoid some of the common problems associated with other filesystems and volume managers. For example, in other filesystems, the base unit of storage is the disk. There are volume managers that aggregate disks together for the filesystems, but this is a kludge at best. ZFS takes the approach that storage should be treated as a pool. All disks are added to ‘zpools’ and ZFS handles the tasks formerly performed by volume managers. This fundamental difference give ZFS significant power and flexibility.

The tutorial included a significant amount of material on each configuration parameter for zpools and zfs datasets. It also covered the tools used to manage pools and datasets. Richard reviewed how the data protection schemes available in ZFS differ from traditional RAID levels. This review of the administration tools of ZFS would be a great help for the sysadmin with no ZFS experience. After this tutorial, they would be able to create, configure and optimize ZFS.

Upcoming features like encryption and dedupe were touched upon. Both of these features were a little too fresh to be fully covered in this class. Dedupe was completed today. Encryption is scheduled to be integrated sometime in the fourth quarter of this year.

One of the most valuable parts of the tutorial was the section on performance tuning. It covered tuning for some of the most common use cases such as databases. It also covered the game changing concept of a Hybrid Storage Pool. With HSPs, the sysadmin will be able to take advantage of both SSDs and HDDs in a blazing fast combination that won’t break the bank.

The tutorial was complete and thorough. It provided data and analytical tools to allow the sysadmin to make informed decisions about protecting data on ZFS.

I certainly hope this tutorial is available at future LISA conferences. ZFS as a technology is quite young. As ZFS matures and is accepted by a larger audience, there will be a need for sysadmins well versed in its operation. I hope this tutorial will be able to continue to grow and adapt to that requirement.