USENIX Update

The submissions deadline for the 2nd USENIX Workshop on the Theory and Practice of Provenance (TaPP ’10) is approaching.

Please submit all work by December 14, 2009, 11:59 p.m. PST.

The TaPP workshop series builds upon a set of Workshops on Principles of Provenance organized in 2007–2009, which helped raise the profile of this area within diverse research communities, such as databases, security, and programming languages. We hope to attract serious cross-disciplinary, foundational, and highly speculative research and to facilitate needed interaction with the broader systems community and with industry.

More information and submission guidelines can be found here.

TaPP ’10 will take place February 22, 2010 in San Jose, CA, and is co-located with  the 8th USENIX Conference on File and Storage Technologies (FAST ’10), which will take place February 23–26, 2010.

TaPP ’10 is sponsored by USENIX in cooperation with ACM SIGOPS and ACM SIGPLAN

Wednesday marked the beginning of the actual conference, as opposed to just the workshops and training sessions of the first few days. The attendance was much larger than earlier in the week, with people coming for the technical sessions and invited paper talks, and vendors have their booths set up to talk about their latest projects. As full as the schedule of the past few days have been, today it’s near the bursting point.

Rather than attend the opening session, I opted to sit in on “Linux Performance Tuning” training session presented by Theodore Ts’o. While the opening featured a keynote by Werner Vogels, I feel like tuning specific bits of the Linux OS is something that has been a gap in my knowledge for too long. This training session afforded me an excellent opportunity to learn from the best. Ted Ts’o is a key developer on the ext4 file system, he has been a kernel developer since the beginning of the Linux kernel, and works for the IBM Linux Technology Center. It’s sort of an understatement to say that he’s qualified to teach this class.

Over the long development of Linux, performance has increased as code became more efficient, but also because behavior of the software working with the hardware has improved. Through time, hardware has increased in speed, but also in the number of configurable settings, which provides the operating systems new and interesting ways of tweaking performance.

Tuning Linux is hard, because potential system bottlenecks are everywhere. What’s worse, certain bottlenecks often present as entirely different issues. Memory under-performance is notorious for this, so learning the tools for proper diagnosis is very important.

Ted started out by working with three tools that nearly everyone uses, but probably not to their fullest extent: free, top, and iostat. The first, ‘free’, displays memory utilization while ‘top’ displays an updating screen with a system summary at top and list of recently active programs at the bottom. The command ‘iostat’, part of the sysstat package, provides output regarding I/O performance of the system and the various storage attached. There are lots of columns out there for increasing the sophistication of your use of the Linux commands: top, free, and iostat. Make sure to read the man pages for each of these as well.

It should be noted that regardless of the tool used to evaluate your system, the impact of running the tool is not zero. There is always some resource utilization by the evaluation tool itself, and depending on the tool, it can be nontrivial. Don’t let this stop you, but only be aware that by watching the output, you are affecting the output, as well.

The most interesting parts of the class were the bits of information that Ted added from his experience working with file systems and storage. These included great pieces of advice, like for the next year or so, at least, don’t buy a laptop with a solid state drive. The drives that manufacturers insert have poor performance and high prices. Instead, get a spindle drive and replace it after market with a good (and somewhat cheaper) solid state drive. Another great tip was that if you are using local storage and it’s not solid state, you can partition your drive to take advantage of the fact that the outside of the spinning disk moves much faster than the inside. Put the file systems that you need fastest read/write access to first to take advantage of this. In fact, Ted suggested that using a fast terabyte drive, and only partitioning the first 300GB or so would lead to performance in the range of solid state media. This class was a wealth of knowledge!

Also covered were network tuning, which can be used to improve performance in extremely fast or extremely slow networks, and file system tuning, which can help performance when dealing with special cases such as large files, small files, quick access, and the like. In addition, Ted devoted a section to NFS tuning, which is in many ways, a combination of disk tuning, network tuning, memory tuning, and the addition of several protocol options which can be tweaked. The good news for NFS is that modern defaults are very good, and should only be changed when clients are atypical, although Ted did suggest using the ‘intr’ to allow you to recover more gracefully, should the NFS server die.

Overall, this was definitely one of, if not the, most valuable training that I took this week. I’m not an expert, but I have a lot more tools and knowledge, and as a mostly-linux administrator, I can tell you that some things are probably going to improve in my infrastructure as a direct result of being in this class. It is a tragedy that it was only a half day.

I’m not blogging to do a sales pitch, but let me just say this. If you have considered coming to LISA, but weren’t sure or couldn’t provide justification, take my word that this class, by itself, was more than worth the entire trip.

-
By Matt Simmons, author of the Standalone Sysadmin blog

Werner Vogels, CTO of Amazon.com, was the keynote speaker at this year’s LISA conference. His keynote was entitled “Ahead in the Clouds”.

One of Werner’s goals in this talk was to level set the audience with regard to the term “Cloud Computing”. He wanted to make sure we understood the reality today of cloud computing versus all the hype.

Werner began with a movie produced by a company called Animoto. This is a company that doesn’t own any server hardware. Their entire server infrastructure is hosted by Amazon. As a matter of fact, when they first started they were just a bunch of guys with laptops sitting in the corner coffee shop. He used this case study as an example of the power of Infrastructure as a Service. Their business model depended upon the rapid provisioning of servers as their customer base grew. They needed their operating expenses to to line up directly with customer use. If their company followed the traditional capital purchasing cycle, two things would have been different. First, they would have been required to seek funding from something like a venture capitalist, diluting their ownership of the company. Second, they could not have scaled to meet customer demand. Since they were using IaaS, they were able to rapidly scale as the customer demand increased.

He wanted to make one thing very clear. Amazon is not in the IaaS business in order to utilize surplus server capacity. They are in the IaaS business because they believe that it could be larger than Amazon.com. I think that this should signal system administrators to watch trends in the Infrastructure as a Service space. If this type of computing infrastructure will scale so large that Amazon’s piece of it is larger than Amazon.com, it will have a tremendous impact on our profession.

As I have discussed in the past there is a bit of uncertainty with the role of system administrators in the cloud. Werner’s view of this issue is that the job of the sysadmin will have to change. They will be required to perform more tasks with scripting and automation rather that interaction at the command line. They will need to be able to affect change whether the change is on one system, one thousand or ten thousand. In particular, the system administrator has new automation not previously available. The provisioning process is now available as an API.

There were several precursors that have allowed us to move in to the era of IaaS: Software as a Service, Distributed Computing, Virtualization, Service Oriented Architecture.

When we talk about SaaS as a precursor to IaaS, we are not talking about SaaS as a business model. Rather we are talking about how we can deliver a software solution as a platform and operate it. Previously, software was something that was authored, packaged, and shipped in a box.

The advances in distributed computing also contribute to the success of IaaS. We must be able to divide complex problems and distribute across many servers. Otherwise our biggest success would be limited by the size of the largest server we could build.

Virtualization is a critical factor in the success of IaaS. Virtualization in this regard is not simply the virtualization of compute resources. It is the virtualization of compute, storage, network, and I/O. We must be able to manage each of these resources discretely in order to meet SLAs and economies of scale needed by IaaS.

The last precursor is Service Oriented Architecture. This is a subtle, but very powerful, tool. Once we commit to SOA, services may be developed separately and loosely coupled. This helps us minimize or eliminate one of the largest expenses in enterprise I/T shops today: integration.

Werner points out that when transitioning from a traditional model to an IaaS model, we must take care to address uncertainty. In the past, developers and operators have hoarded resources such as servers and storage “just in case” there is a need for them in the future. It is a cultural change for them to trust that they may quickly, easily, and reliably provision and release resources when needed.

Amazon has an interesting perspective on organization. Werner noted that the organization matched their service architecture. There are different business units responsible for different internal services.

With a rather dramatic photograph of a data center destroyed by a tornado, Werner emphasised that if you need to provide end to end reliability to your customer, you must have geographic diversity.

I found Werner’s keynote speech engaging and informational. I am encouraged that the CTO of one of the largest players in the IaaS field has a clear direction. I hope that in the future they can continue to provide products higher and higher in the application stack, allowing system administrators to spend less time on infrastructure and more time on providing direct value to our customers.

Check out the Live Stream of the LISA ’09 Technical Sessions, sponsored by Solaris.

The full program can be found here.

LISA ’09  is sponsored by USENIX and SAGE in cooperation with LOPSA and SNIA