I am a soldier in the U.S. Army, currently deployed to Afghanistan. I wanted to be able to share videos with my family from away from home. I wished to maintain my privacy and have better control over my audience. Whether you wish to share videos for educational purposes, share screencasts for documenting software features, or simply entertain, this article will show you how to set up a streaming video website using open source software.

This technique outlines acquiring a video stream from a digital video camera, processing the video stream to the distribution format, and creating a website that will stream the video to users.

Video cameras of the class described in this article have two things in common. First, they store video using the Digital Video (DV) format. Second, they interface with a computer using an IEEE-1394 bus. If a video camera satisfies these two requirements, then it should be compatible with the techniques described here.

In order to facilitate real-time processing and increase the visual quality of recordings, the DV format provides for very limited compression of video data. As a result, files in the DV format tend to be very large. At its rate of approximately 36Mb per second, DV can consume 1GB of disk space in four minutes. Obviously, DV is not a good choice for a distribution format. Before uploading our video content to the website, we will compress its audio using Vorbis and its video using Theora. The audio and video tracks will be encapsulated using the Ogg container format.

Acquiring video

The first step is to acquire a video stream from a digital video camera. Dvgrab is the application that supports this step. Install it on Fedora using the command:

yum install dvgrab

The dvgrab utility interacts with a digital video camera over an IEEE-1394 bus to record the camera’s video stream to a computer’s hard disk. To copy data from a camera, place the camera in play mode and use rewind or fast forward to position the camera’s tape to the beginning of the desired video segment. Connect the camera to the computer using an IEEE-1394 cable. The command dvgrab --format raw --autosplit sample- will begin the transfer. As the transfer begins, the camera will begin to run its tape.

If the camera has an LCD panel, it will display the tape as it plays. The dvgrab utility will record the video as it plays. Because the --autosplit option was used, dvgrab will attempt to identify separate recordings and save them using the filename sample-NUM.dv.

Editing and compressing video

Once a video is captured to disk, it is now ready for editing. There are several up-and-coming free software video editor applications. One such application is Pitivi.

Pitivi is written in Python and uses the GStreamer media framework. To install Pitivi (and some necessary GStreamer plugins), use the command yum install pitivi gstreamer-plugins-good.

Fig 1. Pitivi

Once Pitivi is installed, we will use it to encode our DV recording with Ogg. After starting the Pitivi application, click on the button labeled “Import clips…” Select video files and press the “Add” button to make them available within Pitivi.

Fig 2. Pitivi clips

Once you have selected all of the clips you want, press “Close.” You should now see your videos displayed in the top left corner of the Pitivi application as seen in Figure 2. You may drag and drop the videos into the timeline at the bottom of the application’s window. Once done, the window should look something like Figure 3.

Fig 3. Pitivi timeline

Now that we have assembled our clips, it is time to encode them into the final video. This is done by selecting File->Render. The application will present a new window. Click on the button labeled “Choose File” and enter a name for the file you are about to create. Next, click on “Modify”
to select the target video’s parameters. Figure 4 show the parameters appropriate for our website. We will encode a 320×240 resolution video using Ogg, Vorbis, and Theora. Click “Ok” and then “Record.”

Fig 4. Pitivi encoding parameters

Depending on the length of your video, encoding may take a long time. While you wait for it to encode, download and install the Apache webserver using the command yum install httpd. Once your video processing is complete, copy the resulting file to /var/www/html. I will call this file example.ogg.

Distributing video

Cortado is a Java applet capable of playing streamed video from within a web browser. The applet is open source and is maintained by a company named Fluendo. We will use Cortado to provide a cross-platform way to play the videos on our website. The Cortado applet may be downloaded from Fluendo’s website. The file we will use is cortado-ovt-stripped-0.2.2.jar, which should be copied to /var/www/html. The “ovt” in the filename stands for Ogg, Vorbis, and Theora, the media formats supported by the applet.

Now that we have our video and Java applet installed in /var/www/html, we will write a quick HTML file that references both objects. The following is a simplified index.html for our project that should also be placed in /var/www/html:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-US" lang="en-US">
	<head>
		<title>Test video>/title>
	</head>
	<body>
		<applet code="com.fluendo.player.Cortado.class" archive="cortado-ovt-stripped-0.2.2.jar" width="320" height="240">
			<param name="url" value="http://www.example.com/example.ogg"/>
			<param name="keepAspect" value="true"/>
			<param name="video" value="true"/>
			<param name="audio" value="true"/>
			<param name="bufferSize" value="200"/>
		</applet>
	</body>
</html>

This HTML document references the Cortado Java applet that exists in the same directory. The applet takes several parameters, most notably the “url” and “bufferSize.” The “url” points to the video file and must be a full URL. The fully qualified domain name or IP address referenced must match that of the webserver. The “bufferSize” parameter sets the size of the client-side buffer and should be increased if the video does not play smoothly because of network latency. For a description of Cortado’s parameters, see the Cortado README file, distributed with the project’s source code.

All the tools necessary for building a very simple video website are now at your fingertips. It is time to start the Apache web server and welcome your audience. To start Apache, execute the command:

/sbin/service httpd start

To ensure it starts each time the server reboots, execute:

/sbin/chkconfig httpd on

Loading the URL http://www.example.com/ in a Java-enabled browser will play the video file. When running Fedora, the java-1.7.0-icedtea-plugin is capable of executing the Cortado applet.

You’re now ready to record, encode, and share a video over the web–with complete control over how they are presented. And, better yet: The software used is a completely open source solution for streaming video.

History

When SELinux was first developed, the goal was to confine as many system processes as possible to the least amount of privilege required. Fedora 2 was released with SELinux policy that confined users as well as system processes. We quickly realized that SELinux policy was not mature enough to handle a modern mainstream desktop operating system. After a quick redesign of the policy, we created “targeted” policy, replacing the previously named “strict” policy. The goal of targeted policy was to “target” certain processes in the operating system for confinement and leave the rest of the processes “unconfined.”

Logged-in user’s processes were unconfined in targeted policy. A logged-in user’s process is started by login programs (like login, sshd, or gdm) when the user provides authentication. In Red Hat® Enterprise Linux® 4, there were fifteen targeted applications that were confined, and in Red Hat Enterprise Linux 5 this grew to two hundred.

In Red Hat Enterprise Linux 5, we still did not confine the user. Strict policy was still provided, but few people used the confined user components. Multi Level Security (MLS) policy–which was developed for Red Hat Enteprise Linux 5–attains the highest level of security possible for a main line operating system used in military environments. It provides confinement for the users. MLS, however, only supports servers, so X-Windows applications did not benefit from this confinement.

The MLS development forced us to concentrate more on confining users, and we realized that we could take advantage of this confinement within targeted policy.

Customers/Partner engineers that were looking to work with MLS policy kept asking: How would you write policy for a logged in user who could only talk to a single port? How about a user who could only read a certain directory?

The two types of logged-in users we had developed for strict and MLS policy were user_t and staff_t. This was a problem because these users had basically the same interfaces/transitions and pretty much the same access.

staff_t was able to transition to sysadm_t, which was sort of a poor mans unconfined process. But trying to build a confined user policy out of either of these user types was impossible, since you needed to take lots of privileges away.

I reexamined the policy and decided it was necessary to create a least privileged user, a user that could login to the system with no access other than to read/write his home directory and use /tmp. This user would not be allowed to access the network, or run any setuid applications.

Customers also requested that the user not be able to execute files in his home directory. With this type of user as a base, customers could slowly add privileges to meet their needs.

The guest_t and xguest_t user types

In Fedora 8, we introduced the guest_t and xguest_t user types. The guest_t user has only the privileges necessary to login to a system via a terminal (Login, sshd, rshd, rlogind, and telnet). The xguest_t user has only the privileges required to login to an X-Windows system.
Once we had created these users, we began to find more possible uses for them. For example, at Red Hat, I have ssh access to people.redhat.com and people.fedoraproject.org. We ssh to these accounts and copy files there to set up http web pages, like so:

http://people.redhat.com/dwalsh

Files at this site are actually just files in my home/public_html directory on people.redhat.com. When I ssh to these machines, I really should not use the network to leave them, and I do not need to run any setuid applications while I am on them. So setting up these machines where all users are guest_t would make sense.

xguest_t also comes in handy for use on desktop machines. I have blogged about the use of xguest to confine my wife. [ Ed note.: No wives were harmed in the writing of this article. ] I extended xguest_t to transition to xguest_mozilla_t when running firefox. The xguest_mozilla_t domain is allowed to access http ports but nothing else. The xguest_t account allows me to lock down the system enough so that I know what processes are connecting to the internet. If the user downloads a spam application, it will not be allowed to run in the home directory, nor can it connect to the mail ports.

Other users have suggested using xguest_t for running games and other untrusted software applications. You could also set up an xguest user account to allow others to “borrow” your machine. In Fedora 8 and 9, there is an xguest rpm, which setups a kiosk mode user.
This “xguest” user can log in to your system from the console without a password when SELinux is in enforcing mode. The xguest rpm also sets up pam_namespace to create temporary home directories and /tmp directories, which are destroyed when the user logs out. We think this is an excellent way to run a kiosk machine. The login user can only browse the web (using firefox) and can not leave any apps around to attack other users that log into the system. Everything that user does on the machine is erased when they log out, so no one could grab any password information that might have been left behind.

You can try out xguest by executing:

# yum install xguest

The user_t SELinux user type

For Fedora 9, we combined the strict policy with the targeted policy. We have enhanced the user_t and staff_t SELinux user and now allow you to setup these users in a targeted policy system.

The user_t SELinux user is the standard SELinux user. At Red Hat, we have a distribution of Red Hat Enterprise Linux that is given to all non-engineers by the IT department. Sales people, support, administration, and management staff are given machines installed with a version of Red Hat Enterprise Linux but are not usually given the root password. These are machines for people who do not want to administer their own boxes. The IT department is in charge of updating the software on these boxes and maintaining the security. If users want to add software or modify their machines they have to contact the help desk for an update.

Accounts like these should be set up to use the user_u SELinux user. user_u is a complete login user account, unlike xguest, and it has full networking so that the user can connect to any network port. It does not have the ability to run setuid applications without a transition.

setuid applications are executables that have a special flag set on them. This flag tells the kernel to run the application with the identity of the owner of the application rather then the identity of the person executing the program. Usually setuid apps are owned by root, so running as setuid application as a normal user allows you to gain privileges. Over the years, many setuid applications have had vulnerabilities that allowed root exploits of the system. user_t would not be allowed to run any of these applications.

Since the users of this machine have no reason to ever become root, they do not have the ability to run su, sudo, userhelper, or any other application that requires setuid. Even if you had a setuid shell program on your system, user_t would not be allowed to execute it. Additionally, user_t processes are not allowed to read a lot of system space, so you are somewhat protected from “snooping eyes” looking at how the system is running.

If you have a setuid application that you want the user to be able to run, you can write policy to allow the user_u account to transition to a different domain to execute the code. For example, xlock uses pam to verify the users password. pam executes /sbin/unix_chkpwd, a setuid application. policy allows a transition from user_u:user_r:user_t -> user_u:user_r:chkpwd_t, which can run as root.

Like xguest , the user_u account can be set to disallow execution of programs in the home directory or /tmp.

setsebool -P allow_user_exec_content=0

If you want to set up your system to try out the user account, you can execute the following command as root:

# semanage login -m -s user_u USERNAME

or

# usermod -Z user_u USERNAME

If you want to add a user with user_u, you can execute:

useradd -Z user_u USERNAME

If you want all users on your system to default to user_u you would execute:

# semanage login -m -s user_u __default__

The staff_t SELinux user type

The staff_t user is for users who need to do some system administration, but do not need to be fully unconfined. This is the user that I log in as every day. Staff_t is similar to user_t in that it has full networking and is only allowed to run setuid applications via a transition. staff_t has a transition to sudo so you can write policy to allow the staff_t user to transition to a confined root user via the sudo command. If I run any other setuid application, it will fail, including su.

I actually have a transition defined between the staff_t to unconfined_t. When I become root, I become the unconfined_t user. This allows me to manage my machines any way I want, but requires me to go through sudo to gain the privilege. We have a webadm_t policy available in Fedora 9 that can also be used for a “confined” root user.

My sudoers file has the following line in it:

dwalsh	ALL=(ALL) 	TYPE=unconfined_t ROLE=unconfined_r ALL

To only allow this user to administer apache server I could use:

dwalsh	ALL=(ALL) 	TYPE=webadm_t ROLE=webadm_r ALL

To try out the staff account, execute the following command as root:

# semanage login -a -s staff_u -r s0-s0:c0.c1023 USERNAME

or

# usermod -Z staff_u USERNAME

Configure the staff_u user to allow it webadm_r and/or unconfined_r by executing:

# semanage user -m -R"unconfined_r webadm_r staff_r" staff_u

My next SELinux article will cover using the SELinux GUI to define additional SELinux users or extend the existing users. Also, I will be presenting this information at the Red Hat Summit and hope to be handing out xguest/kiosk liveCDs. If you have any questions or ideas, please come to my talk and let me know.

About the author

Dan Walsh has over 20 years experience in the computer field. He has spent most of his career working on Security Applications and platforms. He spent several years working on the Athena Project while at Digital Equipment Corp. Dan was also involved in designing and developing the AltaVista Firewall and AltaVista Tunnel (VPN) Products. He has worked for Netect developing HackerShield, a Vulnerability Assessment Product. Netect was acquired by BindView, where he continued working on HackerShield and developed a new product, BVControl for UNIX. At Red Hat, Dan has led the SELinux project, concentrating mainly on the application space and policy development. Dan Graduated with a BA in Mathematics from the College of the Holy Cross and a MS in Computer Science from Worcester Polytechnic Institute.

Introduction

Red Hat Network Satellite server allows users to locally host subscribed content from Red Hat Network and custom content in user-managed channels. An example configuration could include a server syncing content updates directly from RHN, while another mission-critical server could be disconnected from the external network, yet still receive updates via manual syncing. In the latter case, these offline servers must be manually updated regularly. Since content updates cannot be synced directly from rhn.redhat.com, RHN Satellite provides two options for our users:

1. Channel dump ISOs hosted on RHN, per Satellite release.

2. RHN-Satellite-Exporter tool running locally on a RHN Satellite server

To Illustrate this, consider the following setup:

Channel dumps

In purely disconnected environments, servers don’t have direct access to RHN to synchronize content. Channel dump media are the primary source for content updates in such cases. As a service to our disconnected RHN Satellite customers, Red Hat provides channel dump media with a base channel dump per channel, supplemented with an incremental dump for every release update. This content can either be sent to customers from Red Hat or downloaded from rhn.redhat.com.

RHN Satellite exporter

In an isolated Satellite environment, there is often a disconnected Satellite as well as a connected Satellite, as shown in the figure above. In this case, the user already has the updated content on their connected Satellite through Satellite-sync across the network. They don’t have to go to RHN to download channel dump ISOs and re-sync their disconnected satellites.

A tool called Satellite exporter extracts the content and generates its own local channel dumps. Exporter queries the connected Satellite database and exports all the content for a given channel from the database and file system on the Satellite server where the content resides. Content type supported includes channels, channel families, package metadata, RPMs, errata, kickstart files and kickstart trees.

For rhn-Satellite-exporter to work as expected, make sure that:

• the Satellite is set up successfully, connected, and has updated content from RHN.
• you have sufficient disk space to store the exported content.

Let’s work with an example to demonstrate how exporter can be utilized to its maximum potential.

I’m a Satellite administrator with two RHN Satellites. I choose one to be a connected server communicating with RHN and syncing content updates directly over the network. The other is a mission-critical server that I prefer to keep isolated—it is disconnected completely from the outside network for security reasons.

Exporting content from connected RHN Satellite

The goal as a Satellite Administrator is to keep the disconnected servers updated. To achieve this, you will make use of the connected Satellite server, which already has content synced from RHN.

First, find out what channel content is available for the connected server to pull down from the connected Satellite. List the channels available with the following command:

 [root@connected ~]#  rhn-satellite-exporter --list-channels --db=rhnsat/****@rhnsat
Channel List:
B = Base Channel
C = Child Channel 

B rhel-i386-server-5
C       rhel-i386-server-vt-5
C       rhn-tools-rhel-i386-server-5 

B rhel-i386-as-4
C       rhn-tools-rhel-4-as-i386

The example above gives all available channels on the connected server that are being synced from RHN. Lets choose to export rhel-i386-server-5 and its child channels, rhel-i386-server-vt-5 and rhn-tools-rhel-i386-server-5.

Now export the base channel content for the specified channels. Each content type is categorized and dumped as shown below. Multiple channels can be exported at the same time with multiple -c or –channel options. This combines all the channel data and dumps it under the directory specified.

If –end-date is not specified, it defaults to the current export date as end-date. The –db option directs the tool to access the source Satellite database to extract the requested content.

1. Create a directory for all the exported content:

   [root@connected ~]#  mkdir /tmp/dumps

2. Export base channel content:

   [root@connected ~]#  rhn-satellite-exporter --db=rhnsat/*****@rhnsat --dir=/tmp/dumps -c rhel-i386-server-5 -c rhel-i386-server-vt-5  -c rhn-tools-rhel-i386-server-5 --debug=5

Exporting incremental channel content (new in Satellite 5.1):

  [root@connected ~]#  rhn-satellite-exporter --db=rhnsat/*****@rhnsat --dir=/tmp/dumps/  --start-date=20071206000000   --end-date=20080206000000 -c rhel-i386-server-5 -c rhel-i386-server-vt-5  -c rhn-tools-rhel-i386-server-5 --debug=5 

So, from the given Satellite server, get all the content for channel rhel-i386-server-5 and its child channels and dumps it under the directory /tmp/dumps/. Content under the directory is dumped into different content types, like so:

[root@connected ~]# ls -l /tmp/dumps/total
drwxr-xr-x    2 root root 4096 Nov 11 02:30 arches
drwxr-xr-x    2 root root 4096 Nov 11 02:30 blacklists
drwxr-xr-x    2 root root 4096 Nov 11 02:30 channel_families
drwxr-xr-x    3 root root 4096 Nov 11 02:30 channels
drwxr-xr-x   12 root root 4096 Nov 11 02:47 errata
drwxr-xr-x    4 root root 4096 Nov 11 02:47 kickstart_files
drwxr-xr-x    2 root root 4096 Nov 11 02:47 kickstart_trees
drwxr-xr-x  102 root root 4096 Nov 11 02:44 packages
drwxr-xr-x  102 root root 4096 Nov 11 02:30 packages_short
drwxr-xr-x  102 root root 4096 Nov 11 02:31 rpms 

3. To exclude specific content types from the content dump, use –no- option. This is available for rpms, packages, errata, and kickstart trees. For example, if you want to exclude rpms from the rhel-i386-server-5 dump:

          [root@connected ~]# rhn-satellite-exporter  -c rhel-i386-server-5  --dir /tmp/dumps --no-rpms 

Packaging exported content (new in Satellite-5.1)

Once we have all the content exported to a directory, we have multiple ways to make this content accessible to our disconnected satellites. The most common method is to burn the exported content onto CD or DVD media and carry it to your remote disconnected Satellite servers. Exporter has options to package the content into ISOs suitable for burning on CD or DVD. This can be achieved using the –make-iso= option.

For CD ISOs:

[root@connected ~]# rhn-satellite-exporter --db=rhnsat/rhnsat@rhnsat --dir=/tmp/  --start-date=20060106000000   -c rhel-i386-server-5   --debug=5 --make-iso=cd

For DVD ISOs:

[root@connected ~]# rhn-satellite-exporter --db=rhnsat/rhnsat@rhnsat --dir=/tmp/  --start-date=20060106000000   -c rhel-i386-server-5  --debug=5 --make-iso=dvd

This should create a Satellite-isos/ directory under the –dir specified, dumping the ISOs into that directory along with MD5SUM manifest for all the ISOs.

[root@connected ~]# ls -ld /tmp/dumps/satellite-isos/*
-rw-r--r--  1 root root         63 Nov 15 16:46 /tmp/dumps/Satellite-isos/MD5SUM
-rw-r--r--  1 root root 656816128 Nov 15 16:43 /tmp/dumps/Satellite-isos/rhn-export-20071115.0-01.iso 

Now that we have the CD ISOs, burn them onto a CD:

[root@connected ~]# cdrecord -v -pad speed=1 dev=0,0,0  /tmp/dumps/satellite-isos/rhn-export-20071115.0-01.iso

This process of generating the channel content essentially remains the same for any channel and any amount of content. Its easy to setup a cron job to generate this content as frequently as it is needed.

Populating disconnected Satellite servers

We can now move the channel content created on the connected Satellite onto the disconnected server.

1. Create a mount point to mount the media:

[root@disconnected ~]# mkdir /mnt/dumps

2. Mount the content onto the disk:

[root@disconnected ~]# mount -t iso9660,udf  /dev/cdrom /mnt/dumps

3. Check to see if the mounted channel dumps have the correct content:

[root@disconnected ~]#  satellite-sync --mount-point=/mnt/dumps –list-channels 

4. Use Satellite-sync to point to the above mount point on disk and sync down the channel content from the dumps:

[root@disconnected ~]#  satellite-sync --mount-point=/mnt/dumps -c rhel-i386-server-5 

Satellite-sync may take some time to complete based on how much content is being synced. Once this step is finished, our disconnected Satellite is all set with the required updated content. Repeat this process as frequently as possible to keep the disconnected servers up-to-date.

Conclusion

This article gives an overview of the multiple ways to populate content on a disconnected RHN Satellite server. It demonstrates exporting content using rhn-Satellite-exporter, packaging content as ISO images, burning them to media, and populating disconnected satellites. Rhn-satellite-exporter is a very powerful tool with a diverse set of options that make content portability between Satellites simple and effective.

About the author

Pradeep Kilambi is a Software Engineer and RHCT/RHCE primarily working on RHN Satellite Client/Server Development. In his spare time, he’s also a dedicated Python hacker and open source advocate.

You may have read our background article about ODF and OOXML and why Red Hat believes OOXML should not be approved as an ISO standard. This time, we focus on how the standardization process has been compromised at ISO.

ISO’s JTC-1 directives were designed to provide a fair, consensus-based way to design standards that are portable, interoperable, and adaptable to all languages and cultures. The OOXML proposal has suffered from two basic problems: (1) voting irregularities, and (2) the use of a fast-track process for a complex, new, large specification that has not received adequate industry review. The resulting specification was driven almost exclusively by one vendor, has not achieved industry consensus, and has had thousands of issues logged against it, largely due to issues involving implementability, portability, and interoperability. Although resolutions have been proposed for many of the issues that have been raised, there is virtually no time to review these resolutions to determine whether they fix the problems. And the voting irregularities have raised serious issues with the fairness of the process.

Stuffing the ballot box

For a standards body to have credibility, the procedures it follows need to be credible. ISO’s JTC-1 directives say that the “objective in the development of International Standards should be the achievement of consensus between those concerned rather than a decision based on counting votes.”¹ Clearly, there has been no achievement of consensus regarding the adoption of OOXML as a standard, and therefore ISO has turned to a voting process.

We believe that the flaws in the ISO voting process for OOXML are so serious that they must be addressed in order to maintain ISO’s credibility as a standards body. For a standards body to review a proposal adequately and achieve consensus, the participants need to be involved in the entire review process, not merely show up to cast a vote.

Unfortunately, the ISO voting process is not restricted to those who have participated in the past.² Thirty-six new countries joined the JTC-1 technical committee–just in time for the OOXML vote³– and 90% of these voted in favor of the OOXML proposal. Only 36% of the original membership voted in favor of approving OOXML as a standard.

Allegations have been made that Microsoft encouraged new countries to join the JTC-1, or to upgrade their status (from O-status to P-status) to influence the vote. Contrary to what has been demonstrated, the JTC-1 directives say that the “objective in the development of International Standards should be the achievement of consensus between those concerned rather than a decision based on counting votes.”⁴

Before an individual country votes in the ISO process, it holds a vote within its own national body. An employee of Microsoft Sweden admitted to offering incentives to business partners to encourage them to vote for OOXML⁵, leading the Swedish Standards Institute (SIS) to declare its vote in favor of OOXML invalid. Critics have speculated that similar practices occurred in Italy, Switzerland, Spain, and other countries. Such allegations have prompted the EU to launch an investigation into Microsoft’s practices during the ISO vote.

Fast track

OOXML was submitted to ISO using the fast-track process, which was intended to make it easier to approve an ISO standard if it has already been approved by an existing standards organization. This was meant to speed up the process for standards whose problems have already been resolved and where consensus has already been achieved. The process generally requires three years or more; fast-track can cut this time to six months.

The first review of the 6,000-page OOXML proposal resulted in a disapproving ballot by national bodies on September 2nd. There were over 3,500 comments. The issues identified with OOXML did not stop there.

After the 3,500 comments during the initial ballot, one delegate wrote, “I and my reviewers found 13 additional errors in the original specification. However, national bodies were not allowed to submit new comments (and rightly so, otherwise there would have been total chaos). Therefore, there was no way to submit and correct them.”⁶

In response, ECMA submitted a proposed Disposition of Comments report that was close to 2,300 pages long. This Disposition of Comments contained proposed changes scheduled to be discussed at a Ballot Resolution Meeting (BRM) in February. This gave only six weeks to review this documentation (that’s a rate of 55 pages per day) before the BRM; it was impossible for all technical issues to be addressed or resolved in that timeframe.

The BRM meeting itself only lasted one week, even though the JTC 1 Directives impose no such time limitation. The normal course for a BRM is to meet, recess with email discussion, and to meet again until consensus is reached on the changes to a proposal. (Remember, the JTC 1 emphasizes consensus in its standardization process.)

Complaints have been lodged with the ISO by some national bodies alleging that the BRM process was inadequate for the number of issues needing resolution. In the final vote at the BRM, only six of the thirty members voted to approve the changes. Four voted to disapprove, and 20 either abstained or refused to register and vote at all.

Obviously, ISO should have referred the proposal to a working committee for further improvements before it was placed on a fast track ballot. But the relevant ISO process was not designed to make this possible. When this is combined with the addition of new members at the last minute in order to influence a vote, the process is fatally flawed.

Start from the goal

The IT industry clearly needs systems so that companies can work well together, and these systems need to work well in all countries. The ISO process for IT standards was designed to promote interoperability, portability, and cultural and linguistic adaptability,1 using a consensus process. We believe strongly in these goals, but the current process is not designed to achieve them. The OOXML proposal has exposed serious flaws in ISO process–especially in the fast-track process–and we believe these flaws need to be fixed.

The credibility of ISO is at stake.

¹ JTC 1 Directives, 5th Edition, 3rd Version, http://www.itscj.ipsj.or.jp/sc34/open/0856rev.pdf
² ISO has two levels of membership, Participating (P level) and Observer (O level). According to ISO rules, a standard needs “Yes” votes from at least two-thirds (67%) of the P level countries, and no more than 25% of all votes from both P level and O level countries can be “No” votes.
³ 11 new P-level members and 25 new O members.
⁴ JTC 1 Directives, 5th Edition, 3rd Version, http://www.itscj.ipsj.or.jp/sc34/open/0856rev.pdf
⁵ http://www.linuxworld.com/news/2007/083007-microsoft-employee-offered-incentives-for.html
⁶ http://elot.ece.ntua.gr/te48/ooxml/brm-clarifications

Come here, I have a secret to tell you.

Python is easy to learn, and more powerful than Bash. I wasn’t supposed to tell you this–it’s supposed to be a secret. Anything more than a few lines of Bash could be done better in Python. Python is often just as portable as Bash too. Off the top of my head, I can’t think of any *NIX operating systems, that don’t include Python. Even IRIX has Python installed.

If you can write a function in Bash, or even piece together a few commands into a script and make it executable, then you can learn Python. What usually throws Bash scripters off is they see something object-oriented like this:

class FancyObjectOriented(object):
    def __init__(self, stuff = "RegularStuff"):
        self.stuff = stuff
    def printStuff(self):
        print "This method prints the %s object" % self.stuff

Object-oriented programming can be a real challenge to get the hang of, but fortunately in Python it is 100% optional. You don’t need to have a Computer Science degree to program in Python–you can get started immediately if you know a few shortcuts. My goal here is to show Average Joe Bash scripter how to write in Python some of the things they would normally write in Bash. Even though it seems unbelievable, you can be a beginning Python programmer, by the end of this article.

Baby steps

The very first thing to understand about Python, is that whitespace is significant. This can be a bit of a stumbling block for newcomers, but it will be old hat very quickly. Also, the shebang line is different than it should be in Bash:

Python Shebang Line:

#!/usr/bin/env python

Bash Shebang Line:

#!/usr/bin/env bash

Knowing these two things, we can easily create the usual ‘Hello World’ program in Python, although whitespace won’t come into play just yet. Open up your favorite text editor and call the python script, hello.py, and the bash script hello.sh.

Python Hello World script:

#!/usr/bin/env python
print "Hello World"

Bash Hello World script:

#!/usr/bin/env bash
echo Hello World

Make sure that you make each file executable by using chmod +x hello.py, and chmod +x hello.sh. Now if you run either script–./hello.py or ./hello.sh–you will get the obligatory “Hello World.”

Toddler: System calls in Python

Now that we got ‘Hello World’ out of the way, lets move on to more useful code. Typically most small Bash scripts are just a bunch of commands either chained together, or run in sequence. Because Python is also a procedural language, we can easily do the same thing. Lets take a look at a simple example.

In order to take our toddler steps it is important to remember two things:

1. Whitespace is significant. Keep this in mind–I promise we will get to it. It is so important that I want to keep reminding you!

2. A module called subprocess needs to be imported to make system calls.

It is very easy to import modules in Python. You just need to put this statement at the top of the script to import the module:

import subprocess

Lets take a look at something really easy with the subprocess module. Lets execute an ls -l of the current directory.

Python ls -l command:

#!/usr/bin/env python
import subprocess
subprocess.call("ls -l", shell=True)

If you run this script it will do the exact same thing as running ls -l in Bash. Obviously writing 2 lines of Python to do one line of Bash isn’t that efficient. But let’s run a few commands in sequence, just like we would do in Bash so you can get comfortable with how a few commands run in sequence might look. In order to do that I will need to introduce two new concepts: one for Python variables and the other for lists (known as ‘arrays’ in Bash). Lets write a very simple script that gets the status of a few important items on your system. Since we can freely mix large blocks of Bash code, we don’t have to completely convert to Python just yet. We can do it in stages. We can do this by assigning Bash commands to a variable.

Note:

If you are cutting and pasting this text, you MUST preserve the whitespace. If you are using vim you can do that by using paste mode :set paste

PYTHON
Python runs a sequence of system commands.

#!/usr/bin/env python
import subprocess

#Note that Python is much more flexible with equal signs.  There can be spaces around equal signs.
MESSAGES = "tail /var/log/messages"
SPACE = "df -h"

#Places variables into a list/array
cmds = [MESSAGES, SPACE]

#Iterates over list, running statements for each item in the list
#Note, that whitespace is absolutely critical and that a consistent indent must be maintained for the code to work properly
count=0
for cmd in cmds:
    count+=1
    print "Running Command Number %s" % count
    subprocess.call(cmd, shell=True)

BASH
Bash runs a sequence of system commands.

#!/usr/bin/env bash

#Create Commands
SPACE=`df -h`
MESSAGES=`tail /var/log/messages`

#Assign to an array(list in Python)
cmds=("$MESSAGES" "$SPACE")

#iteration loop
count=0
for cmd in "${cmds[@]}"; do
    count=$((count + 1))
    printf "Running Command Number %s n" $count
    echo "$cmd"
done

Python is much more forgiving about the way you quote and use variables, and lets you create a much less cluttered piece of code.

Childhood: Reusing code by writing functions

We have seen how Python can implement system calls to run commands in sequence, just like a regular Bash script. Let’s go a little further and organize blocks of code into functions. As I mentioned earlier, Python does not require the use of classes and object-oriented programming techniques, so most of the full power of the language is still at our fingertips—even if we’re only using plain functions.

Let’s write a simple function in Python and Bash and call them both in a script.

Note:

These two scripts will deliver identical output in Bash and Python, although Python handles default keyword parameters automatically in functions. With Bash, setting default parameters is much more work.

PYTHON:

#!/usr/bin/env python
import subprocess

#Create variables out of shell commands
MESSAGES = "tail /var/log/messages"
SPACE = "df -h"

#Places variables into a list/array
cmds = [MESSAGES, SPACE]

#Create a function, that takes a list parameter
#Function uses default keyword parameter of cmds
def runCommands(commands=cmds):
    #Iterates over list, running statements for each item in the list
    count=0
    for cmd in cmds:
        count+=1
        print "Running Command Number %s" % count
        subprocess.call(cmd, shell=True)

#Function is called
runCommands()

BASH:

#!/usr/bin/env bash

#Create variables out of shell commands
SPACE=`df -h`
MESSAGES=`tail /var/log/messages`
LS=`ls -l`
#Assign to an array(list in Python)
cmds=("$MESSAGES" "$SPACE")

function runCommands ()
{
    count=0
    for cmd in "${cmds[@]}"; do
        count=$((count + 1))
        printf "Running Command Number %s n" $count
        echo "$cmd"
    done
}

#Run function
runCommands

Teenager: Making reusable command-line tools

Now that you have the ability to translate simple Bash scripts and functions into Python, let’s get away from the nonsensical scripts and actually write something useful. Python has a massive standard library that can be used by simple importing modules. For this example we are going to create a robust command-line tool with the standard library of Python, by importing the subprocess and optparse modules.

You can later use this example as a template to build your own tools that combine snippits of Bash inside of the more powerful Python. This is a great way to use your current knowledge to slowly migrate to Python.

Embedding Bash to make Python command-line tools^[1]:

#!/usr/bin/env python
import subprocess
import optparse
import re

#Create variables out of shell commands
#Note triple quotes can embed Bash

#You could add another bash command here
#HOLDING_SPOT="""fake_command"""

#Determines Home Directory Usage in Gigs
HOMEDIR_USAGE = """
du -sh $HOME | cut -f1
"""

#Determines IP Address
IPADDR = """
/sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | head -1
"""

#This function takes Bash commands and returns them
def runBash(cmd):
    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    out = p.stdout.read().strip()
    return out  #This is the stdout from the shell command

VERBOSE=False
def report(output,cmdtype="UNIX COMMAND:"):
   #Notice the global statement allows input from outside of function
   if VERBOSE:
       print "%s: %s" % (cmdtype, output)
   else:
       print output

#Function to control option parsing in Python
def controller():
    global VERBOSE
    #Create instance of OptionParser Module, included in Standard Library
    p = optparse.OptionParser(description='A unix toolbox',
                                            prog='py4sa',
                                            version='py4sa 0.1',
                                            usage= '%prog [option]')
    p.add_option('--ip','-i', action="store_true", help='gets current IP Address')
    p.add_option('--usage', '-u', action="store_true", help='gets disk usage of homedir')
    p.add_option('--verbose', '-v',
                action = 'store_true',
                help='prints verbosely',
                default=False)

    #Option Handling passes correct parameter to runBash
    options, arguments = p.parse_args()
    if options.verbose:
        VERBOSE=True
    if options.ip:
        value = runBash(IPADDR)
        report(value,"IPADDR")
    elif options.usage:
        value = runBash(HOMEDIR_USAGE)
        report(value, "HOMEDIR_USAGE")
    else:
        p.print_help()

#Runs all the functions
def main():
    controller()

#This idiom means the below code only runs when executed from command line
if __name__ == '__main__':
    main()

Python’s secret sysadmin weapon: IPython

The skeptics in the Bash crowd are just about to say, “Python is pretty cool, but it isn’t interactive like Bash.” Actually, this is not true. One of the best kept secrets of the Python world is IPython. I asked the creator of IPython, Fernando Perez, how IPython stacks up to classic Unix interactive shells. Rather than trying to replicate what he said, I’ll simply quote directly:

IPython is a replacement for the Python interactive environment that tries to incorporate the most common shell-like usage patterns in a natural way, while keeping 100% syntactic compatibility with the Python language itself. In IPython, commands like ‘cd’ or ‘ls’ do what you’d expect of them, while still allowing you to type normal Python code. And since IPython is highly customizable, it ships with a special mode that activates even more defaults for shell-like behavior. IPython custom modes are called profiles, and the shell profile can be requested via:

ipython -p sh

This will enable all the shell-like features by default. The links below show some basic information about the shell-like usage of IPython, though we still lack a comprehensive guide for all of the features that actually exist under the hood.

http://ipython.scipy.org/moin/Cookbook/IpythonShell
http://ipython.scipy.org/moin/Cookbook/JobControl

IPython also contains a set of extensions for interactively connecting and manipulating tabular data, called ‘ipipe,’ that enables a lot of sophisticated exploration of filesystem objects and environment variables. More information about ipipe can be found here:

http://ipython.scipy.org/moin/UsingIPipe

It is quite possible to use IPython as the only interactive shell for simple systems administration tasks. I recently wrote an article for IBM Developerworks, in which I demonstrated using IPython to perform interactive SNMP queries using Net-SNMP with Python bindings:

Summary

Even if you can barely string together a few statements in Bash, with a little work you can learn Python and be productive very quickly. Your existing Bash skills can be slowly converted to Python skills. And before you know it, you will be a full-fledged Python programmer.

I find Python easier to program in than Bash; you don’t have to deal with hordes of escaping scenarios, for one. Bash has its place–usually when you don’t have the ability to run Python–as Python beats the pants off Bash as a scripting language.

I have included a link to all of the examples, and will have a souped-up version of the Python command-line tool with a few extra tricks sometime soon.

Let me close with saying that if you are interested in replacing Bash with Python, try to start out on the best possible foot and write tests that validate what you think you wrote actually works. This is a huge leap in thinking, but it can propel your code and productivity to the next level. The easiest way to get started with testing in Python is to use doctests, and I have enclosed a link at the bottom of this article. Good luck!

References

• Subversion Repository For Examples
• Checklist Based Testing For SysAdmins
• Doctests
• Online Bash Scripting Guide
• Python Tutorial
• IPython
• Jeff Rush Show Me Do Tutorial
• PEP8
• Net-SNMP and IPython

^[1] This code example has been corrected. Feb 08, 2008, 11AM EST

About the author

Noah Gift is currently co-authoring a book for O’Reilly, “Python For *Nix Systems Administration,” (working title) due sometime in 2008. He works as a software engineer for Racemi, dealing with Bash, Python, SNMP and a slew of *nix operating systems, including AIX, HP-UX, Solaris, Irix, Red Hat, Ubuntu, Free BSD, OS X, and anything else that has a shell. He is giving a talk at PyCon 2008–the annual Python Programming convention being held in Chicago–on writing *nix command line tools in Python. When not sitting in front of a terminal, you might find him on a 20 mile run on a Sunday afternoon.

Have you ever had deja vu? I re-read books on occasion, because I like them, and every once in a while I’ll re-read a book that I think I’m reading for the first time. Then I’ll sit there with this twisted-up look on my face, wondering why all the words seem so familiar. Then I remember when and where I saw them last.

I’ve been reading the new Fedora™ 7 Unleashed book by Andrew and Paul Hudson, and I’ve had that feeling several times. So I’ve made my face and wracked my brain, trying to figure out how I’ve read this before. The answer? I read Fedora Core 6 Unleashed and Fedora Core 5 Unleashed before that.

Its unfortunate. I think these ‘distro tomes,’ so to speak, provide a valuable service to the Linux uninitiated, and can be useful to more seasoned sysadmins interested in the latest technology a new release of a given distribution has to offer. The unfortunate part is that both of these groups have to suffer through repetitive rehashing of methods, processes, and utilities that have been throughly documented in a multitude of locations and media.

To be fair, this book does some things quite well, but as I’ve looked at more and more works in its genre, I’ve come to realize that the good ones are characterized by a bias toward documenting whatever new technology is available, while the less useful ones focus more on maximizing the width of their spine with well-worn stock material. I’m sorry to say that this book leans more toward the latter.

But lets not dwell on the negative (or at least let’s not start there). Theres good in everyone, and there is good in this book. Some of you may remember that I wrote a review of Tammy Fox’s book, Red Hat® Enterprise Linux® 5 Administration Unleashed. This book, like that one, does many of the same things rather well. Specifically, it handles new technologies with aplomb.

Xen, while not new technology per se, is sufficiently in its infancy. The online documentation is still somewhat fragmented, and a concise summary of how xen virtualization operates is a perfect example of what this book excels at. SELinux also gets very fair treatment in this book, as does the Mono programming language. While not strictly bleeding edge, these topics are still sufficiently new that consolidated documentation is a benefit that books like this provide very well.

Most of the sections in this book have their usefulness. Everyone needs to know how the useradd or cp commands work. But I just can’t get past (and yes, I’m back to dwelling on the negative) how re-hashed so many of these chapters are. As part of my review of this book, I went and took a quick look at the table of contents for Fedora Core 6 Unleashed. Written by the same authors, its an amazing example of documentation re-use. Several of the chapters look sufficiently similar as to be identical.

I don’t want to insinuate that the information in these sections isn’t useful, it really is. It contains information everyone needs to know. You can’t administer a system without being able to change passwords, set up remote access, preform backups, so on and so forth. But good gosh! Lets just take one example: Chapter 17, Apache Web Server Administration. Its almost 40 pages of documentation about how to manage, configure, and tune the Apache web server to provide web pages to interested parties. Hmm, I wonder what would happen if I typed ‘Apache howto’ into Google. Dum de dum….hey, 2.550,000 hits! I wonder if Apache has a web site…..hey what do you know? They look like they might have some comprehensive documentation. And Amazon indicates there are 1,285 books on sale at their site exclusively on the subject of the Apache web server.

The same is true of the sections on FTP, database, SSH and network file access. Note to the authors: We’re covered. Your material is useful and factually accurate, but I just can’t get past the fact that it’s not really necessary. You could have written a book that was half as big, was consequently less expensive, and yet still provided all the same useful content.

I have a feeling it’s not the authors fault. You can pick up any number of comprehensive high-level OS documentation books, and see this. There is some material that is just covered again….and again….and again. My inner conspiracy theorist is convinced it’s more than coincidence. I firmly believe that (a) extraterrestrials are out there, and (b) publishers of these books will encourage their authors to add content in an effort to claim more shelf space. I think a better technical bookshelf could be constructed with a series of more targeted topics, but such an approach lowers the revenue generated by this type of books.

So that only leaves the final question: Should you buy this book or not? I’ve been pretty hard on it–in my view, with good reason. Despite that, I still think there is a demographic for whom this book would be fairly useful.

If you have any significant experience with a Linux distribution of any origin, I would expect that you have the skill (both technical and research-oriented) to find the information contained in this book on the web (or elsewhere). Likewise, if you’re a sysadmin, and you have experience (in a Linux or non-Linux environment) I would expect that your technical bookshelf has much greater depth, and much more targeted breadth, specific to your needs. So, clearly, this one’s not for those with experience.

However, if you are new to Linux and are interested in getting your feet wet, you probably don’t want to invest a fortune on reference material, or spend too many hours poking about on mailing lists or web sites putting together a reasonable set of information to get started. If this is you, then yes, this book is a good choice. It offers a comprehensive outline of what you need to know to work with Fedora 7, both as an end user and as an admin. And thats quite likely why these books tend to be a perennial favorite.

The Linux community has a consistent influx of new users and curious onlookers who have the desire to know more. A consumer-oriented, one-stop shop might get them there faster than the community would, with its plethora of websites and arcane knowledge passed down though stories on cryptic mailing lists. So to keep current, these books keep coming. And if you’re new, it could be your first step into a bigger world.

Of course, not all lost or inaccessible data holds clues to life on Mars, and not every shred of information needs to outlive its creator. Many unreadable documents will never be missed, but responsible public policy demands that government documents–contracts, deeds, or court records that remain in force for decades or even centuries–must be archived and accessible. Whatever the case, when data is stored and shared on legacy or defunct proprietary formats, over time it will either become expensive to access or disappear entirely.

When it comes to digitally creating, sharing, and storing documents, the technology to prevent format-based decay already exists and is in wide (and growing) use. It’s called the Open Document Format (ODF) and if you’re not currently using it, someday you will.

The ODF, an XML-based document markup language, was first developed in 1999 by Stardivision and then by Sun Microsystem’s Openoffice.org project. Conceived as an open alternative to the proprietary document handling software, which then dominated the world, the driving force behind the ODF was the need for a vendor-neutral document format independent of any single application, readable and writable for all, without any royalty of licensing “encumbrances.” It was promoted on the basis that business and taxpayers would save money. An open format would create competition in the document application sphere. All documents could be read and shared by everyone. Nothing could be lost to time or changes in proprietary code or licensing requirements. Matters of great public interest - census data, weather data, public health statistics, investigative reports, court records and basic scientific research, all paid for by taxpayers, would no longer be encoded on a single, proprietary closed-standard format, requiring citizens to pay twice for access to their own information. Using ODF, proponents said, would keep public documents public.

The Organization for the Advancement of Structured Information Standards (OASIS) was formed in 2002 to standardize the format, which was recognized by the International Organization for Standardization (ISO) in 2006. The Open Document Format Alliance was formed in March of 2006 to promote the format, making the public, legal and political case for the adoption of open technology standards to governments and public institutions.

“Red Hat was a founding member of the ODF Alliance and Tom Rabon [Vice President of Corporate Affairs] serves on the executive committee,” says Stephanie McGarrah, outgoing Red Hat Public Policy Manager. “Red Hat works with other executive committee members to coordinate efforts to talk with governments around the world about ODF.”

Although the ODF was launched with a great gust of common sense blowing at its back, the momentum of widespread adoption has been hindered by bureaucratic inertia, local politics, persistent misconceptions (reinforced by opponents) about ODF’s viability and the “dangers” of adoption. Most of the fear, uncertainty and doubt has emanated from one source, on whose proprietary formats most of the world’s documents currently reside.

Opponents of the ODF do not concede its inevitable adoption, and actively lobby against it. It’s not that anyone is against the ODF in and of itself, or finds any real reason to question its necessity. The logic behind the ODF and the transparency of its creation is fairly unassailable. Rather, it is the open standards on which the ODF is based that are most attacked. From the detractors’ point of view, things are just fine the way they are now. The “standard” is theirs. They own the document “market,” and think of it as “territory” they “won” fair and square. They can’t foresee a future without them (that’s not in their business plan), and as long as everybody is already using their applications and formats, why change? Opponents of the ODF devote considerable resources to lobbying legislatures and executive branch IT advisory boards in an attempt to convince them that the adoption of the ODF actually limits choice and harms market-driven efficiency by “locking out” vendors like them. They say migration is expensive, and even argue that adoption of the ODF will limit public access by cluttering the environment with too many “incompatible” formats. And who really trusts all this “free stuff,” anyway?

But proponents like the ODF Alliance have arguments of their own, and most of them come from “Actually, the opposite is true…” school of refutation.

The ODF Alliance contends that open standards actually promote choice and vendor competition by leveling the playing field. The standard is open and freely available for anyone to implement. There is no competition over the format, just the application used to handle it. In this universe, the best applications win. The ODF alliance also points out that implementing or migrating to ODF is no more complicated or costly than periodically upgrading from one version of a proprietary application to the next, and by obviating the need for future upgrades, real money is saved over time. As for accessibility, Open Office (and other ODF-compliant applications) are freely down-loadable and ready to use now. There are no actual compatibility issues, they say, only non-cooperation issues.

“I think that some governments either aren’t aware of ODF or don’t have the technical staff in place who understand the value of ODF,” McGarrah explains. “So, it’s the alliance’s job to spread that message to the people in government who make those decisions.”

But the unshakable argument in favor of using ODF for public documents is the fact that it’s a better deal for citizens and taxpayers in the long run. Using closed-standard, proprietary software for public documents is like buying the proverbial $10,000 toilet seat, or prohibiting the federal government from negotiating better drug prices with pharmaceutical companies on behalf of Medicaid and Medicare patients, or trying to feed an army and rebuild a warzone by awarding secret, non-competitive, no-bid contracts. It’s non-competitive in the worst sense.

Despite opposition, adoption of the ODF is making slow but inexorable headway, and as a greater understanding of the issue is reached by policymakers, the ODF will challenge the standing ubiquity of proprietary formats. Moving the issue forward, Japan recently required that all its ministries contract with software vendors whose applications are built around open standards. Brazil, Poland, Malaysia, Italy, Korea, Norway, France, The Netherlands, Denmark, Belgium, The Commonwealth of Massachusetts, and the Dehli State Government in India have all made commitments in principle to adopting the ODF and, perhaps more importantly, recognizing the imperative of using open standards. The ODF Alliance continues to arm and enlighten policy-makers with the information and tools they need to make recommendations and change policy, but no one promoting the ODF think its widespread adoption is imminent. It will take time.

“These decision makers have a lot of other issues to deal with (i.e. Health care, education, transportation, poverty…) so technology decisions aren’t usually at the top of their lists,” says McGarrah. “Progress has been made on the wider adoption of ODF. Several governments have adopted ODF and are working to implement the standard, but there is a lot of work to do.”

More information

• Spread the word, share this ODF artwork by Michael Pittman

Want to read every word of the long and complicated terms and conditions? Have at it. Prefer a brief explanation of its basic talking points? Then you’re at the right article. We happen to know a guy who knows a lot about this stuff–his name is Luis Villa, and he’s hacked on a few Linux projects in his day. Like Evolution PIM, the GNOME 2.0 release (in collaboration with Sun), and the Novell and Ximian desktop projects. And now he’s going to law school. The perfect cross-section for license-explaining.

So why are we here?

On June 29th, after 16 years, the Free Software Foundation issued version three of the GNU General Public License, the sequel to what is arguably the most important copyright license ever. Quite literally everyone who makes software - open, proprietary, or web - needs to understand the v3 and figure out how it impacts them as a potential contributor, consumer, cooperator or competitor.

Can you summarize this Q&A in a haiku?

    new license coming
    creates new uncertainty
    must attempt to grok

(I really, really wanted to use “Snuffleupagus” in that, but sadly that doesn’t leave many syllables for imagery.)

Why are we doing all this again? Isn’t GPL v2 reasonably good?

The license was written in 1991, and the computer industry has changed a lot since then. DRM has changed how users interact with their computers, software patents have become a much bigger problem, and the free software community has grown into a multi-billion dollar industry. So an update was probably not a bad idea.

Given the success of v2, has the license really changed that much?

The core goals, methods, and structures of v2 were successful and have been carried over with very few changes. If you use GPL code and do not redistribute it, you still get to do whatever you want with it. If you modify and redistribute GPL code, or build new applications on top of GPL-licensed libraries, you still have to release modifications and derivatives as GPL-licensed source. And you can can still build proprietary code on top of the new LGPL.

What has changed in the license, then?

The biggest changes:

• internationalization: the new license moves away from language which comes from US copyright law in favor of language which does not exist in any system of copyright law. This should make the license more politically palatable and legally enforceable outside the US.
• increased complexity: This tries to be a more lawyer-friendly document, at the expense of clarity for hackers and executives. That may slow adoption.
• patents: Contributors to a program now grant a patent license to future users; merely distributing without copyrightable contribution does not. This should help create more certainty about the patents owned by major contributors- the folks like Sun, Novell, etc. - but it doesn’t help against those who don’t contribute code, like Microsoft and patent trolls. So the impact is positive but limited.
• patents, the complicated part: the license attempts to prevent future blanket indemnifications like the Microsoft-Novell deal, and to induce Microsoft to grant us patent licenses, but the language is complex and probably has some loopholes. Because of the complexity and limitations of the language, probably only the worst abuses of the old language are prevented. Not bad, but not the end of Microsoft’s FUD by any stretch.
• user control: the new license tries to make it clear that users have the right to control their hardware and software. First, it forbids a legal claim that the code is part of a Digital Rights Management system (though it does not prevent an attempt to construct such a system, as long as the code itself can still be modified.) Second, it requires that distributors provide installation instructions so that consumers can modify and reinstall the GPL’d software on the devices that they own.

What has changed for users?

Most users won’t see any change from the shift from v2 to v3- they’ll be able to keep trucking, since users have all the same rights they used to have, plus a few new ones. There are new requirements for contributors and distributors, but they should be threatening only to the small minority of companies who want to benefit from the GPL while competing on a basis other than quality and service.

Something more to tell us?

Of course! This is a 5,702 word license, so this Q&A is inevitably simplified. Visit my personal (not company endorsed) blog or the FSF’s FAQ for more details.

We bring the advice of experts straight from San Diego to your desktop.

Red Hat Summit 2007 collected hundreds of Linux users all in one place–many of them experienced Red Hat Certified Engineers® (RHCE). And somewhere between all those smart people walking around–and our video crew shooting footage–the idea for some video tips was born.

This tip is from Richard Ray. Look for more in the coming weeks.

The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

Getting started

You will need to have subversion installed. As always, the rpm (Red Hat Package Management) system makes this easy. Let’s check to see if the package is installed.

rpm -q subversion

If the output is something like this:

subversion-1.4.3-0.1.el4.rf

Then you’re all set to start using subversion. If the package is not found, then simply type in:

yum install subversion

Creating a local subversion repository

The best way to understand how subversion works is to jump in with both feet. Let’s create a subversion repository on your machine that you will use to store your version control database. To create a subversion repository, just type in this command:

svnadmin create /usr/local/svn/squidrepo

Go ahead and see what that did. Change into that directory and do an ls:

cd /usr/local/svn/squidrepo
ls -l

You should see a bunch of directories:

conf/ dav/ db/ format hooks/ locks/ README.txt

If this looks scary, weird, and confusing, it should. You don’t want to touch this directory yourself! It is just a directory that SVN uses internally to keep track of changes. You won’t ever need to go in this directory unless you’re doing advanced systems administration on subversion.

Importing a directory into version control

Keeping track of changes to configuration files is a great use for a svn repository. If you have read my articles on squid, you might be thinking, let’s put the squid configuration files into svn. You’re correct; that is a perfect use of subversion. Let’s do an import to get started. In order to use svn, you must first import a working directory full of items you would like to keep track of changes on. To keep track of changes on the squid configuration directory, issue this (one line) command:

svn import -m "initial import of /etc/squid directory" /etc/squid file:///usr/local/svn/squidrepo/

What this does is import your squid configuration directory into the svn version control system. One very tricky detail is that a lot of people think you’re done and that you can now start using version control on the /etc/squid directory. This is wrong! You have only imported the directory. Only subsequent checkouts of the source code from subversion will allow you to maintain versioning.

Let me give you an example. Go ahead and check out the source code into a “sandbox” directory first. Let’s make that sandbox now:

mkdir /tmp/sandbox

Then change into the sandbox:

cd /tmp/sandbox

Then check out the source code we checked in:

svn co file:///usr/local/svn/squidrepo squid

If you notice, you checked out the subdirectory “squid”.

Now type in this command, and note that I am selecting etc/squid and NOT /etc/squid.  This is very important /etc/squid is your working configuration directory!:

svn co file:///usr/local/svn/squidrepo etc/squid

This time you checked out the whole directory tree you originally imported. Subversion will let you decide to if you want to check out a directory or a directory tree.

Implementing version control for Squid configuration files

So now that you know how to import and checkout code, the next step is to actually implement this with your /etc/squid directory. I want to put a word of caution out. If you are on a production server, do not do this step until you have practiced it in a sandbox and feel 100% comfortable. It is never a good idea to move, change, or delete configuration files without a backup, either.

First, stop squid if it is running:

service squid stop

Backup squid configuration directory:

mv /etc/squid /etc/backup_squid

Change into the /etc directory:

cd /etc

Now check the squid file out of version control:

svn co file:///usr/local/svn/squidrepo squid

Now start the squid service again:

service squid on

Using version control to keep track of Squid configuration files: A whirlwind tour

Now that we have a working copy out of svn, and it lives in the correct directory, we can start testing out some of the fun features of subversion. Let’s get some information. If you’re in the /etc/squid directory, issue this command:

svn info

You should see some type of output like this:

[root@cent squid]# svn infornPath: .
URL: file:///usr/local/svn/squidrepo
Repository UUID: cd285af0-e034-0410-ad4e-8dbcf155913f
Revision: 1
Node Kind: directory
Schedule: normal
Last Changed Author: root
Last Changed Rev: 1
Last Changed Date: 2007-07-10 04:19:25 +0000 (Tue, 10 Jul 2007)

This command tells you who checked in this copy, when, and where it is located in the repository, etc.

Next, let’s do something fun. Let’s change our configuration file and add some nonsense to it. Then make svn revert it back to the original condition. Open up /etc/squid/squid.conf and with your favorite text editor, add the following line to the top of the file. I like to use vim, so for me it is:

vim /etc/squid/squid.conf

"This is completely incorrect text. Subversion will help me get rid of it though!"

Now restart squid and see it complain:

service squid restart

You should see an error message like this:

Stopping squid: 2007/07/10 05:01:45| parseConfigFile: line 1 unrecognized: '"This is completely incorrect text. Subversion will help me get rid of it though!"'.
[ OK ]
Starting squid: . [ OK ]

That is annoying. Let’s get rid of the error by using svn to revert the file. Simply type in the following command:

svn revert /etc/squid/squid.conf

You should get back the following message:

Reverted '/etc/squid/squid.conf'

Now restart squid and you will see that the error message went away.

service squid restart
Stopping squid: .. [ OK ]
Starting squid: . [ OK ]

As you can see, this is incredibly powerful, as you can completely screw up your config file and then revert it back in a millisecond. (Of course this all depends on you remembering to commit changes to svn every time you make a modification to your files.) Let’s test this out by adding a simple comment to our squid config file and then committing the changes.

Open /etc/squid/squid.conf and add the following to the top of the squid file:

#First SVN Import

Now commit the changes; remember, you should to be inside /etc/squid:

svn commit -m "added a comment to my /etc/squid/squid.conf file"

You now should get the following feedback:

Sending squid.conf
Transmitting file data .
Committed revision 2.

That feedback tells you that you just committed a change revision number 2. If you ever want to see what is in your svn repository, just type in the command:

svn list

Summary

Our Whistle-stop tour went through creating a local subversion repository, importing a directory tree into version control, implementing version control on a config file directory tree and finally, using svn to revert and make changes to config files. This is probably enough svn to get you started for a while, but I would highly recommend reading the svn book, or at least keeping it around as a reference. Remember, version control can keep your important data and keep it safe. Everyone should be using some sort of version control!  If you use svn to keep track of your configuration files or important documents, I would love to hear how it has saved your day.  This will be great motivation for people who haven’t decided to use version control yet.