USENIX Update

Two of the Friday sessions I attended were focused on the security. The first was a guru session on SELinux led by Dan Walsh, Red Hat’s lead SELinux developer. SELinux is a labeling and enforcement engine developed by the NSA and released to the public. It has developed a reputation among many sysadmins for being a hassle and is often turned off quickly. In addition to Rik Farrow’s training session earlier in the week (covered in the LISA 10 blog), Dan’s guru session was a chance for admins to learn how to coexist with SELinux.

Dan reviewed the way SELinux labeling works, and how to build SELinux policy by searching for errors in logs and piping the output through the audit2allow program. For example, to find the Booleans that allow the access being denied to an FTP server:

grep ftp /var/log/audit/audit.log | audit2allow

Of course, it’s important to be judicious in adding allows. SELinux is an intrusion prevention system, not an intrusion detection system. As a result, it’s not very good at alerting you that you’ve been compromised. One sign is when a process is looking for access that is unusual.

After taking Rik’s training last year, and attending Dan’s guru session on Friday, I’ve resolved to enable SELinux on my desktop and laptop. We’ll see how it goes.

In the afternoon, Susan Landau gave a talk entitled “Surveillance or Security? The Risks Posed by New Wiretapping Technologies”. Unlike most other sessions, this talk was policy-based instead of technical. In the last decade, especially in the United States, wiretapping has been a widely-discussed issue, making the talk very relevant.

Susan began with a discussion of the challenges imposed on wiretapping mobile phones compared to their wireline counterparts. Wiretapping began almost as soon as the first telegraph lines were strung. General J.E.B. Stuart used wiretapping to learn the movements of opposing armies during the Civil War. Law enforcement use began in earnest during the Prohibition Era.

Relevant laws and judicial rulings from the 20th century were presented as a base to discuss the last 20 years, when wiretapping has been most prevalent. The Communications Assistance for Law Enforcement Act (CALEA) greatly expanded the scope of wiretapping in the U.S., including a requirement that phone switching equipment have wiretap support built in.

As international fiber lines were built in the 1990s, it changed the way many foreign-to-foreign calls were routed, bringing them through the United States. The Foreign Intelligence Surveillance Act (FISA) did not require a warrant for wiretap when one end of the communication was outside the United States, so the NSA lobbied to have this extended to fiber.

Sold initially to aid in kidnapping investigations and later in anti-terrorism efforts, wiretaps have been broadly used (though still, perhaps, not as widely used as during J. Edgar Hoover’s tenure as the FBI Director). Simply understanding the transactional history of phone calls has led to the capture of Khalid Sheikh Mohammed and the London subway bombers. The U.S. Marshals, by locating suspects’ mobile phones, have been able to reduce apprehension times from 42 days to 2 days.

These benefits don’t mean that wiretaps are unequivocally good. Building wiretapping support into infrastructure exposes tapping capability to parties who can compromise the equipment, and reduces the cost of collection such that overcollection becomes a civil liberties concern. Although Susan says the real national security threats are cyberexploitation, she maintains that it’s important for freedom, especially press freedom, remain unrestricted.

The best papers are the ones where after they’re presented you say “I can’t wait to take these back to work!” That’s the case with the papers presented in Wednesday morning’s session. Chris St. Pierre and Matt Hermanson started off with “Staging Package Deployment via Repository Management“. In this paper, they describe how they use a three-level repository scheme to manage the testing and deployment of software packages.

The “upstream” repo is a daily mirror of the distribution’s repositories. This repo gets checked against a package blacklist and copied to the “unstable” repo. Packages which have been in unstable for a week or more without issue get copied into the “stable” repo. This allows testing of packages on a limited number of hosts before being deployed more widely.

The next paper was Philip Guo’s “CDE: Run Any Linux Application On-Demand Without Installation“. His position is that packaging is hard, especially when applications require specific versions of many libraries. For users who lack root privileges, the challenge is especially daunting. Philip’s solution is CDE (not to be confused with the desktop environment), which neatly packages an application’s code, data, and environment.

By using ptrace magic, an unprivileged user can run an application and copy any necessary binaries and libraries into a tree. This package can then be shared with other users who can execute the package unprivileged in a chroot-like environment. CDE packages can also be streamed off of cloud-based services. By making self-contained packages, programs can be run on any other platform with the same architecture and kernel ABI-compatible to the build system. This means, for example, Fedora users can easily share applications with Ubuntu users.

The final paper of the session, entitled “Improving Virtual Appliance Management through Virtual Layered File Systems” was presented by Shaya Potter and co-authored by Jason Nieh. Their work focuses on reducing the effort and storage overhead of maintaining multiple VM images. As it turns out, VM appliances make for easy setup, but can be difficult to manage and lead to sprawl. By stratifying the VM’s filesystem into many layers, even largely heterogeneous machines can be more easily managed.

Each individual application (e.g. Apache httpd) can exist in a separate layer. UnionFS is used to combine the appropriate read-only layers for each machine. A read-write layer is then added on top for local and volatile data. In addition to greatly reducing the disk overhead in large VM deployments, it also makes it much more difficult to compromise individual VMs.

Yesterday’s Women in Tech Panel was moderated by Lois Bennett, who was joined by panelists Carolyn Rowland, Máirín Duffy, and Deb Nicholson. Lois started the discussion with the question, “Is there a problem?”

Carolyn responded, “I didn’t think there was a problem because I’ve adapted, I think.” She said that she tried fitting in by being “like a guy” in how she dressed, talked, and interacted. “I wanted to fit in and be accepted for how smart I was,” she said, adding, “Women sometimes want to feel like women, not like men.” After a while, Carolyn noticed, “It took over who I am.” She asked the question, “Did you down play femininity because you wanted to? Or because you felt you needed to?”

Máirín agreed with Carolyn, saying, “IRC and mailing lists basically made me cry.” She focused on university-based projects, but by her senior year in college, she wanted to make a difference in open source. She got an internship at Red Hat, which then helped her make face-to-face contacts with people.

Deb pointed out that something about the open source community makes it less inviting than the proprietary software community, which has a higher percentage of women participants. An attendee spoke up and shared an experience at her office. Seven men and one woman were interviewed for a position, and ultimately one of the men was hired based on his existing skill set. She pointed out that statistically, the woman wouldn’t get chosen because she might not have as much experience. Deb said that studies show that men consistently overestimate their skill set, whereas women consistently underestimate.

Forking a Child Process

The panel also discussed whether women are penalized for wanting to start a family or for already having children. Carolyn, who has three children, said that when she was pregnant, she went to work every day and had to overcome the fact that she was pregnant and a baby was on the way, whereas her husband didn’t have to carry that with him to the workplace. Two days after delivering each child, Carolyn was back online, making sure her colleagues knew she was still plugged in and not going anywhere. She felt extra pressure to be productive from home when her baby was sleeping, for example, which was pressure her husband didn’t have to deal with at work.

Deb pointed out that other industries offer generous paternity leave and suggested that the IT industry could unionize and commit to a 40-hour work week (which didn’t raise any objections from the audience). Carolyn pointed out that men can’t really understand pregnancy and maternity leave, saying, “If you were my supervisor, I’d be worried about it.”

Máirín said that she and her husband are discussing starting a family, which wasn’t something she was thinking about when she was evaluating potential employers. She wonders whether women self-select out of IT careers because they start families, and she said that one of her friends left the industry until her child was two or three years old. Máirín notes, however, that her employer, Red Hat, has a strong internal community of women who network and support each other.

IRC You

The topic moved to women working with other women, and there is an assumption that if there are two women in an office together, they will like each other and be friends, which isn’t always the case.

Deb said that a lot of men don’t think they are sexist or part of the “problem,” but she pointed out that in an IRC channel, for example, if one loud person is sexist and not one else speaks up about it, a new person in the channel will assume that everyone else agrees with the loud one. She said that being silent while someone louder and more obnoxious represents your community doesn’t help.

Grok

What about recruitment? Máirín talked about the success of the GNOME Project’s Summer of Code. At first, no women applied. The next time around, a GNOME Women’s Summer Outreach program was launched. They took a more neutral approach to promoting the event — instead of focusing on anything competitive or game-related, they talked about the opportunity to learn new skills and earn money. The first year of the new effort, 10 women participated, which encouraged the sponsors to fund more years. She asked, “How are you wording your job postings? Where are you posting them?” Computer labs, for example, have been successful for GNOME outreach promotions.

Culture is slow to change, but Carolyn points out that you can adapt to a culture if you understand it. “We have to adapt because we are the minority,” she says. Still, any culture can be more inviting. She points out that LOPSA could have a mentoring effort to help women adapt without having to hide themselves. “I think that LISA has gotten better and women feel safer here,” she adds.

Action Items

At the end of the panel discussion, Máirín shared the notes she’d been taking. She summarized the talk with the following action items:

1. Do not be quiet. If you see funny business going on, say something.
2. Try to spread awareness that there is a problem.
3. Make sure you have training opportunities in the workplace.
4. If you are a woman in IT, reach out to other women and consider mentoring.
5. Women need to watch their communication (be assertive, don’t say “I think,” sound confident).
6. For women in tech events, invite non-women, too [few men showed up for the panel discussion, and there was discussion about how some men felt unwelcome to attend].
7. Change meeting formats to be more inclusive so everyone has a chance to talk.
8. Make your work more visible (self-promotion).
9. Have a women’s mentorship community (like the one in Red Hat, for example).
10. Review your recruitment strategy (check company website — is it all white men?).
11. The medium is the message (maybe a blog is more comfortable for you than communicating in IRC, for example).

I’ll add this other thought: LISA has a decent turnout of women each year, but we’d love to see more women (and new faces in general) join us. If there’s anything we can do to make LISA — or other USENIX events — more inviting to you, please let us know. And if LISA ’11 was your first time attending a LISA event, we’d love to hear feedback about your experience.

A special thanks to Nicole Forsgren Velasquez for generously sharing her talk notes for this article!

Jesse Robbins started his presentation “GameDay: Creating Resiliency Through Destruction” (slides) with this awesome quote:

“You don’t choose the moment,
the moment chooses you.
You only choose how prepared
you are when it does.”
-Fire Chief Mike Burtch

and even if it was originally targeted to firefighters it relates so much to system administrators. Actually, during all this session, Jesse draws parallels between two of his greatest passions: firefighting and operations.

Gameday is an exercise designed to increase resilience through large-scale fault injection across critical systems where resilience is seen as the ability of a system to adapt to changes, failures, & disturbances. By “system”, he means: people, culture, processes, applications & services, infrastructure, software and hardware.

GameDay increases resilience in 3 ways:
Preparation
- Identification and mitigation of risks and impact from failure
- Reduces frequency of failure (MTBF)
- Reduces duration of recovery (MTTR)
Participation
- Builds confidence & competence responding to failure and under stress.
- Strengthens individual and cultural ability to anticipate, mitigate, respond to, and recover from failures of all types.
Exercises
- Trigger and expose “latent defects”
- Choose discover them, instead of letting that be determined by the next real disaster.

Jesse also had some great practical advice on how people can start doing such a ‘frightening’ exercise: start small. Start with small, controlled failures. Announce them in advance to the team and let them prepare the best they can. Then run the gameday as planned, even if it might be scary to actually turn off a vital part of your infrastructure. Doing this will enable the team to grow and learn from a controlled fire drill. Once you have a good level of trust you can move to a full scale exercise, where you turn off a full datacenter or whatever is identified as the ‘scariest’ thing in your organization, the one thing that everybody is afraid to turn off. This will probably cause a disaster, but this is the only way how your team will learn and build the trust and experience to perform at its best during a real emergency.

Jesse concluded: “there is no substitute for experience… Failure free operations require experience with failure”. Great presentation and great advice. I’m sure many of the attendees were intrigued by the gameday idea and will run such fire drills soon.
I know I will.