We here at Red Hat are pleased to bring you a brand new set of videos aimed at showing off the latest and greatest enhancements in our technologies–featuring the very people who helped create them in the first place. The “SPOTLIGHT ON” series highlights the ways in which collaboration drives innovation by looking at projects that have been improved by community input. In our first installment, we track down Red Hat’s own Karsten Wade and Stephen Smoogen from the University of New Mexico to talk about Extra Packages for Enterprise Linux (EPEL), the Fedora-sourced repository of add-on packages for Red Hat Enterprise Linux. They discuss how EPEL is a tool for user-driven innovation that comes from and benefits enterprise customers with more stable code and lower business costs.

Red Hat has been working on the technologies behind MRG for quite some time–each of the components in MRG has had years of development. For example, Red Hat has been working on realtime technologies in the upstream kernel community for over seven years. Messaging has had a
similarly lengthy development history. Condor, the technology behind our grid scheduler, started development in the 1980’s!

We started work on these technologies because we saw the need for these capabilities, even if we didn’t know when or how we were going to bring
these technologies to market yet. For example, messaging is at the heart of enterprise computing. We had needs for messaging infrastructure at Red Hat–for building out our own capabilities around things like virtualization management. Many of Red Hat’s customers were asking us to provide an open source messaging offering. So, we started working on the AMQP specification and our messaging implementation, even though we didn’t know it was going to end up in something called “Red Hat Enterprise MRG”.

As we started working with customers and the community around the various technologies in MRG, it became apparent to us that the technologies had reached a point of maturity where we could support our most demanding customers with them. Also, we saw significant opportunities for building out fundamentally new capabilities by integrating messaging, realtime, and grid into one platform. And so, MRG was born.

We released MRG v1 at the Red Hat Summit on June 19, 2008. MRG v1 offers support for messaging and realtime, and grid is in Technology Preview. We’ll release a 1.1 update to MRG that will bring grid into full support as well.

JP Morgan Chase, like other investment banks, uses messaging for everything from executing stock trades to providing feeds of market data
to internal data distribution.

Realtime provides deterministic performance. The US Navy is deploying realtime in its DDG 1000 naval destroyers. Realtime is critical in this
environment, because the ships’ computers have to respond precisely without ever pausing, freezing, or getting out of sync with other
events. Otherwise, the results could be disastrous.

One of our large manufacturing customers has been working with Red Hat to build an on-demand grid in Amazon’s EC2 cloud environment for the times it needs access to a grid for calculations. Because this customer isn’t able to utilize fully a dedicated grid, having the option to deploy a grid in the cloud provides them significant cost savings and flexibility.

There isn’t an ideal customer–ultimately, we think that almost any customer will benefit from MRG. MRG provides a new platform and solution for many of the most pressing problems that enterprises face today. We have significant customer interest from many industries.

Having said that, many of our largest customers are MRG early adopters, such as investment banks like JP Morgan Chase, telco companies like
Alcatel Lucent, and multiple agencies in the US Government. We are also working across oil&gas, animation studios, Internet, shipping, stock exchanges, defense, travel, and so on.

MRG takes special advantage of and is highly optimized for Linux to deliver its performance. Additionally, at Red Hat, we have been driving
changes into Linux itself in order to benefit things like messaging performance. So, the fact that we are focusing on just one platform and optimizing both that platform and our implementation on that platform gives us tremendous gains (Note: everything we do is open source and contributed back to the community).

For example, we have written a new high performance journal for durable or persistent messaging that is highly tailored to Linux’s I/O model.
By using this journal, MRG Messaging can achieve throughputs up to about 500,000 durable messages/second/LUN. This rate is about 100 times
faster than other messaging solutions. For more details, you can read Carl Trieloff’s entry in the Red Hat Press blog.

Yes. For example, we support messaging clients across a wide variety of platforms and languages, from Linux to Solaris to Windows, and from C++ to Java/JMS to scription languages like Python. On the grid side, we’ll support scheduling to both Linux and Windows. And, of course, since we integrate with virtualization, this gives us a lot of flexibility in running on other operating systems.

We definitely see a lot of interest in cloud computing from customers. MRG integrates with cloud providers like Amazon EC2 so that you can dynamically provision and add capacity in the cloud from your grid scheduler. This means, for example, that you could have a scenario where you fully utilize your local data center but have additional work you want to compute.

MRG can automatically provision, say, 1000 extra servers for you at EC2, send your work over, get your results back, and tear down the servers when you’re done–all automatically. Some of our other customers are looking at provisioning most or all of their capacity in the cloud because they won’t utilize a data center fully and want to save on capital expenses.

In either case, one of the powerful features of MRG is that it can blend local capacity with cloud capacity. This means you don’t get locked into one cloud provider, and you can grow your infrastructure dynamically in the cloud or in your local data center.

One of the significant things about AMQP is that it is the first protocol standard for business messaging. All other standards, like JMS, aren’t comprehensive enough and don’t specify down to the wire level to provide true interoperability and an open ecosystem. So, I’m not concerned about competing standards–there aren’t really any right now. That’s why there is so much interest in AMQP.

I think that most big businesses will understand and appreciate what AMQP has to offer. Notably, many of the big businesses driving AMQP are not vendors but users. Eventually, if you want to work with these users, you’re going to have to adopt AMQP.

MRG is important to all of our offerings–it’s pretty strategic and central to many of the things that Red Hat is doing. MRG adds realtime capabilities to Red Hat Enterprise Linux and enables you to provide flexible scalability and performance for applications running on Red Hat Enterprise Linux. We’re working with Red Hat Network so that you can provision and manage MRG with our standard management tools. MRG Realtime and a realtime JVM from IBM or Sun can provide deterministic performance for JBoss Java applications. We’re working with the JBoss team to support MRG Messaging as a messaging transport for the JBoss ESB. And, many of our core products and technologies are using MRG technology. IPA and oVirt, for example, are both leveraging our messaging capabilities for distributing data.

More information

• Read more about MRG at the Red Hat Press blog.
• See the official MRG announcement.
• The official MRG pages on redhat.com.

In Red Hat Enterprise Linux 4, The kernel-devel package includes the kernel headers files and you no longer require the kernel source package to build a third party kernel module. To install the kernel-devel package run the following command as root user in a terminal:

#up2date kernel-devel

A full source tree is not required in order to build modules against the current kernel you are using. You can simply point your Makefile to /lib/modules/`uname -r`/build. A more detailed explanation can also be found in the Release Notes.

If you require the kernel source package for reasons other than building a kernel module, you can obtain it in Red Hat Enterprise Linux 4 by typing the following as root user in a terminal:

# up2date redhat-rpm-config rpm-build

# up2date --get-source kernel

# rpm -ivh /var/spool/up2date/kernel*.src.rpm

# cd /usr/src/redhat/SPECS

# rpmbuild -bp --target=i686 kernel-2.6.spec

# cp -a /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9 /usr/src

# ln -s /usr/src/linux-2.6.9 /usr/src/linux

Note: This will build the source tree for a x86 based architecture. For different architectures, (i.e. x86_64) pass the appropriate target variable (i.e. rpmbuild -bp –target=x86_64 kernel-2.6.spec )

Once completed, a symlinked directory pointing to the latest Linux 2.6 kernel source should be available:

# ls -lt /usr/src
total 28
lrwxrwxrwx   1 root root   12 Mar  2 16:36 linux -> linux-2.6.9/
drwxr-xr-x  20 root root 4096 Mar  2 16:21 linux-2.6.9

Note:The steps are also provided in the Red Hat Enterprise Linux 4 Release Notes: http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/release-notes/as-x86/

This information has been provided by Red Hat, but is outside the scope of our posted Service Level Agreements (https://www.redhat.com/support/service/sla/) and support procedures. The information is provided as-is and any configuration settings or installed applications made from the information in this article could make your operating system unsupported by Red Hat Support Services. The intent of this article is to provide you with information to accomplish your system needs. Use the information in this article at your own risk.

Red Hat’s customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

Network handling

Another area that’s shown a lot of improvement since Enterprise Linux 5 is networking, especially for desktop and laptop computers. In Fedora 9, we’ve greatly enhanced NetworkManager, and as a result, have switched to NetworkManager by default for all installs. Some of the features we’ve added to NetworkManager include:

• MobileBroadband support - NetworkManager now supports configuring access via GSM and CDMA cards for even greater connectivity options.
• System configuration support - NetworkManager now reads my system configuration , as configured via anaconda or system-config-network. This allows support for things such as static IPs.
• Multiple device support - NetworkManager will automatically connect to both wireless and wired devices simultaneously. This means that if I disconnect the wired device, I’ll have seamless access through my wireless device, instead of having to wait for it to associate and get an IP address.
• Connection editing - NetworkManager also includes a connection editor. With this, I can easily configure my wireless network, my mobile broadband connection, or even 802.1x for my wired connection.

From the application side, we’re working on getting more and more apps tied into the NetworkManager infrastructure so they will automatically adapt to changing networks. For example, Firefox in Fedora 9 will now automatically go into offline mode if the network goes away, and go back online as soon as it returns.

In the future, we’re looking at extending NetworkManager to support Ipv6, as well as more device types (such as bridging and bonding devices).

Encrypted devices

One of the most requested features since the release of Enterprise Linux 5 is encrypted device support. We support encrypted devices via a technology called LUKS. LUKS, implemented on top of the existing device-mapper cryptography code, standardizes the partition header for the automatic detection of encrypted devices. It also allows for multiple passphrases to decrypt the device. For example, if I insert an encrypted USB stick, the encrypted device is detected via HAL, the GNOME file manager prompts me for the passphrase, and LUKS unlocks the device–which is then mounted and ready to use.

Fedora 9 goes even further: We support encrypting the entire system in the installer if the user desires. Anaconda will prompt for a password for physical devices, and then on boot, you’ll be prompted for that password before proceeding.

Audio handling

The addition of PulseAudio has greatly improved the audio subsystem in Fedora 9. PulseAudio is a networked sound server, used for mixing and playing audio streams on the system. Now, some of you might remember ESD, and wonder why we need another sound server. Simply put, the power and flexibility provided by PulseAudio is far beyond what ESD ever did.

For example, with PulseAudio I can easily adjust the volume of the audio streams individually. If I’m listening to music and I get a SIP call via ekiga, I can mute the music.

PulseAudio abstracts away your hardware devices as well. Say I decide to plug in a USB headset. I can then move any (or all) of my audio streams to that headset, while they’re playing. If I remove the headset, the audio streams are automatically moved back to the remaining device.

We’re working to use PulseAudio natively by all the applications shipped in Fedora and Enterprise Linux, and also use PulseAudio’s network support as a potential means for doing virtualized audio.

User switching and constrained users

Say I’m using my computer. My daughter comes up and says she wants to check her mail. In Fedora, we’ve made it easy for you to switch between users. When I log in, I see my user name on the panel. If I click there, my session is locked and a new login window will launch. From that window, I can select a different user. When this user logs out, I’ll automatically be switched back to my session.

If you notice, there’s a ‘Guest’ entry in the user list. The simple listing belies what’s available–this account is more than just an login named ‘guest’. Through cooperation between the Desktop and the SELinux teams, we’ve introduced a technology called ‘xguest.’

XGuest is a restricted kiosk-type user. I can log in as the guest user with no password. The guest user is specially confined via SELinux and only certain actions are allowed. For example, the only way to get out on the network is via the browser. Furthermore, on logout, any changes, customizations, or data saved by the guest user is thrown away so the next user will start with a clean slate (and guests can do little permanent damage). For example, I can change the background of the desktop. The next time I log in as the guest user, it will be reset to normal.

Virtual file systems

Another feature that was added in Fedora 9 is something called GVFS. GVFS is a userspace-based virtual filesystem. It replaces gnome-vfs in GNOME, adding many new features.

Let’s look at an example. One of the backends for GVFS is an archive mounter. I can enter and examine an archive–such as an ISO image or a tarball of source code– via the file manager by simply right-clicking and choosing ‘Open with Archive mounter.’ Once it’s open, I can navigate and open items with standard GNOME tools. Anything mounted with gvfs will also show up both in the GNOME ‘Places’ menu, and in the file chooser.

GVFS adds a new feature that makes it even more useful. Leveraging the power of FUSE userspace filesystems, anything mounted by GVFS is also exposed to your ’standard’ Linux tools under ~/.gvfs. I can navigate into it with a shell, and read/write files to these locations without actually porting your apps to GVFS.

GVFS has back-ends for archive mounting, Samba/CIFS shares, DAV shares, sftp network mounts, OBEX (for talking to your bluetooth phone), and more. In future releases, we plan on extending the backends available for GVFS, and porting more of the desktop stack to use it natively.

Virtualization

If you’ve been around any sort of technical presentation in the past two years, you’ve certainly heard about virtualization. That’s another area we’re working on improving in Fedora, although a lot of that improvement is done under the covers. For example, if I start virt-manager now, it looks much the same as it does under Red Hat Enterprise Linux 5.

However, the virtualization hypervisor has changed from Xen to KVM. KVM offers many benefits over Xen:

• KVM uses the standard kernel

  Xen runs as a hypervisor that requires a modified kernel underneath. Since it runs a (somewhat) non-standard kernel, it can lead to compatibility problems, especially for anything that has to call into, or use, the BIOS. For example, power management has had problems working under Xen, and console redirection and handling was often an interesting exercise., as anyone who’s used a serial console will tell you. By using the standard kernel, KVM allows full hardware compatibility–whatever works in the standard kernel, works in your virtualized host.

• KVM is in the upstream kernel

  KVM has been accepted into the upstream Linux kernel. This is a great step for virtualization, in that it will now always be available, and won’t need continual porting to newer kernels. As anyone who has followed Fedora knows, attempting to maintain the Xen kernel patchset against the upstream kernel is a large amount of work. Working in the upstream obviates the need for that effort, allowing work to be done on improving the virtualization experience rather than chasing the kernel of the day.

But wait, you say… what about my paravirtualized guests? We’ve got you covered. Fedora 9 introduces new technology called Xenner. Xenner emulates the Xen hypervisor interface as a thin layer on top of KVM.

This shows the power of our virtualization strategy–by abstracting the interfaces away via tools like libvirt and virt-manager, we can change out the virtualization hypervisor, yet still present the same interface to the user and administrator, and run the same guests.

And more…

There’s even more where this comes from–new versions of Firefox, new firewire stacks, and more. If you want to know what’s down the road for Red Hat Enterprise Linux, check out Fedora 9– that’s where the innovation happens.

About the author

Bill Nottingham is an engineer at Red Hat, where he’s worked on Red Hat Linux, Red Hat Enterprise Linux, and Fedora for the past ten years. (yipes!) He currently serves on the Fedora Project Board and the Fedora Engineering Steering Committee, and maintains a variety of packages in Fedora.

One of the most-asked questions in the software world is:

“What’s coming in the next release?”

Red Hat® Enterprise Linux® is no different. You can wait for the beta, and pore over the release notes and the package changelogs. You can corner a product manager at the right moment. But the easiest and best way to get the scoop on what’s coming up in future Red Hat Enterprise Linux releases? Take a look at Fedora.

Fedora, for those that don’t know, is a freely available and distributable Linux-based operating system that showcases the latest in free and open-source software. It’s developed globally within the Fedora project community, and is where Red Hat innovates. Read on as we describe some of the innovations in Fedora that will be headed for future Enterprise Linux releases.

Handling of displays

Have you ever tried to configure a projector on your Linux box? How much swearing was involved?

On Red Hat Enterprise Linux 5, configuring a projector can be complicated. It might involve editing your Xorg configuration file to add archaic configuration directives, and then restarting your session, losing anything you have open.

Fedora improves on this with the introduction of the new XRandr extension. XRandR stands for X Resize and Rotate, and it’s a mechanism for dynamically changing the screen’s resolution and orientation at runtime. What’s more, it includes support for dynamically changing the attached displays at runtime. So all you need to do is attach your projector, open up the Screen Resolution utility in the Hardware Preferences menu, and you’re ready to go. You can configure the displays to be mirror images, or arrange them in any configuration you like.

Package management

If you’ve used Fedora lately, you may have noticed the ‘bug’ icon in the notification area. This is part of PackageKit, a new distribution-neutral package management system that abstracts away the machine’s packaging system (in our case, yum and RPM) to provide a simple, usable, interface for the user.

PackageKit is implemented in a client-server fashion–all the package installations and removals are done in a privileged backend, while the user interface code runs unprivileged, and talks to the backend over d-bus. Fedora (and later Enterprise Linux) uses a yum backend for PackageKit; other backends (for Debian or Ubuntu packages, for example) also exist.

If I click on the applet, I can easily review what updates are available, and which bugs each update will fix. I can then choose the ones I want and apply them.

Not only will PackageKit notify me when updates are available, but it can be configured to automatically install updates. This is especially useful for security updates.

PackageKit also comes with a general software installation tool. With it, I can browse the software on my system, and pick out packages to install. Once I select a package, I can check which files are included, what the package depends on (or what depends on it), and even visit the package’s homepage for more information.

In the future, we’re working on refining and cleaning up the PackageKit UI, and extending its use. New features might include automatically finding available codecs needed for multimedia files, or finding and installing office software like OpenOffice when you’re sent an office document. For more information on PackageKit, see the PackageKit website.

Privileged apps for non-privileged users

In Red Hat Enterprise Linux, you may be familiar with this dialog:

This dialog is part of userhelper, a graphical authentication system. However, it’s a system that has various issues:

• It’s rather inflexible
  • The only configurations available are root or user password
  • It cannot be reconfigured so that only certain users or groups are authorized
• Once they are authorized, any commands run as root
  • This is obviously bad. One of the fundamental tenets of computer security is to limit the amount of code running with elevated privileges. For some of the apps launched by userhelper, the code running as root can include entire graphical toolkits.

In Fedora 9, we set out to solve this with PolicyKit. PolicyKit is a framework for defining system policy to allow unprivileged apps (such as end-user UIs) to talk to privileged code (such as application installers, or configuration systems).

Let’s look at an example. If I right-click on the clock applet, I see the option to ‘Adjust Date and Time.’ If I choose that, and adjust the time, I see a PolicyKit authorization dialog. Once I enter the proper authorization, PolicyKit then invokes the privileged backend with the necessary data, and sets the clock. This was all done without running any part of the clock applet as root.

Now, to the user, the interface is much the same as before. For the administrator, however, PolicyKit opens up a wealth of new possibilities. This can be seen in the ‘Authorizations’ tool in the System Preferences menu. If I choose ‘Setting the system time’, we can see the authorization I just used. I can edit it, or add new authorizations for other users. I can even add implicit authorizations–for example, I could set it so that any user in an active console session can reset the clock.

Of course, to edit these also requires authorization via PolicyKit. And the package update we did earlier? That was authenticated via PolicyKit, too.

In Fedora 9, we have a limited number of tools authorizing via PolicyKit, but in future releases, we intend to expand this across the OS. For more information on PolicyKit, see the PolicyKit reference manual or David Zeuthen’s LCA talk (PDF).

Power management

Another issue we’ve been working on is power management. A recent article in NetworkWorld showed that Red Hat Enterprise Linux 5 used as much as 12% less power than Microsoft® Windows® on identical hardware. But we didn’t stop there–we’ve made it even better in Fedora.

The first improvement is the tickless kernel. One of the best ways to save power is to keep the CPU as idle as possible. In older kernels like the one used by Enterprise Linux, there is a timer that ticks a certain number of times per second. When that timer fires, the kernel looks around to see if there’s anything to do. Generally, this timer runs anywhere from 100 to 1000 times per second. Obviously, the kernel waking up that often isn’t very conducive to staying idle. With the recently-introduced tickless kernel, it only wakes up if prompted by something that needs attention.

Obviously, that won’t help you if you have lots of kernel threads or userspace apps that are keeping the kernel busy. To combat this, we worked with a userspace tool called ‘powertop,’ originally written by Intel. Powertop tracks the number of kernel wake-ups, and associates them with hardware devices, kernel threads, or userspace programs.

Since powertop was introduced in Fedora, we’ve been constantly running it looking for misbehaving applications. Some examples:

• pidgin would ask the X server if it supported the XScreenSaver extension every five seconds… even though the answer would never change
• ntp would wake up more than once per second to do work
• media players would send silence to the sound card after playback was stopped, keeping the sound card active

There have been many more discoveries–like apps that poll much too frequently for changes or kernel drivers that are too busy. We’ve been working on fixing these, and and so far we’vemoved from Enterprise Linux 5 having over 1500 wake-ups per second and consuming over 18W when idle (without wireless) to about 50 wake-ups per second, consuming less than 16W.

Further work to reduce power consumption is being done. For laptops, there are things like backlight control for the LCDs, and X tricks that compress the framebuffer so it takes less juice to refresh the display. But it’s not just laptops–reducing the number of wake-ups and decreasing other unnecessary power usage is just as useful in the datacenter. Less power usage means less heat generated and less cooling needed, lowering energy bills all around.

About the author

Bill Nottingham is an engineer at Red Hat, where he’s worked on Red Hat Linux, Red Hat Enterprise Linux, and Fedora for the past ten years. (yipes!) He currently serves on the Fedora Project Board and the Fedora Engineering Steering Committe, and maintains a variety of packages in Fedora.

I now have that tool in NetworkManager. In this article I will explain what NetworkManager is, what capabilities exist in the tool (in both Fedora and Red Hat Enterprise Linux), and what you can do to extend it to give you more control over your system than before.

What is NetworkManager?

NetworkManager is a software utility that allows a desktop user to manage wired, wireless, modem, WWAN/3G, and VPN network connectivity from a single source. It does not require root access or manual editing of configuration files.

NetworkManager started as a Gnome project and initially appeared in Fedora. It is now supported on multiple desktop environments (Gnome, KDE, Xfce, etc.) and in multiple distributions (Fedora, SuSE, Ubuntu, Gentoo, Debian, etc.). NetworkManager uses dbus and hal to provide network status updates to other desktop applications, allowing them to alter their operation based on this information. For instance, if NetworkManager shows the network is offline, then apps like Evolution and Pidgin will put themselves into offline mode andwait for the network to come online.

How is the NetworkManager software deployed on the system?

NetworkManager is deployed in two parts. The first part is the NetworkManager daemon, which is found in the package NetworkManager. This daemon should be set to start while the system is booting. This can be accomplished by entering the following command as root:

    # /sbin/chkconfig NetworkManager on

You can also start NetworkManager manually by entering the following command as root:

    # /sbin/service NetworkManager start

The second part is the user client, which normally takes the form of an applet. This applet (nm-applet) can be found in the NetworkManager-gnome package, and should be part of the basic Gnome desktop installation. You will not need to add this applet to your desktop. Gnome will add the nm-applet control to the Notification Area applet when the NetworkManager daemon is active.

How does NetworkManager work?

For the user, most everything will be done via the NetworkManager applet. Exactly what needs to be done depends on the type of networking the user needs to activate.

Wired network

If the system the user is logged into is on a wired network (Ethernet), the user does not need to do anything. NetworkManager will look for the link on the network port. When the link is active, it will bring up the interface and then ask for network information via DHCP.

Wireless Network

If the user is trying to connect via wireless, NetworkManager is especially helpful. As long as the wireless device is active, NetworkManager will scan for available networks and will attempt to connect to the last network you connected to that it can see. If the network it is trying to connect to is a secure network (using WEP, WPA, WPA2, or LEAP) it will request the appropriate security information. Once the information is entered, NetworkManager will try to store this information into the GNOME keyring manager.

To connect to a different network than the one that NetworkManager chooses, simply click on the applet and choose a different wireless network.

WWAN network (3G/EVDO/HSDPA/RTTx1/EDGE)

With the release of NetworkManager 0.70, users can now choose WWAN networking. Most of these cards require activation in Windows, but NetworkManager can handle the auto-configuration some cards need for use under Linux. Other cards may still require some minimal account information to activate and use.

If the card is plugged in when NetworkManager starts, it will be autodetected and an attempt to auto-configure the card will be made when you request a connection to the network. If auto-configuration is successful, the user can then just select the card in the applet menu and connect.

VPN connectivity

Once a successful network connection has been made, the user can also use NetworkManager to activate a VPN connection. Currently, there are modules providing support for OpenVPN and Cisco (via vpnc) VPN connectivity.

The VPN connection will be configured, activated, and deactivated via the applet. Username, password, group passwords, and other information can be stored in the GNOME keyring manager, or the user can choose to be prompted to enter some—or all—of the information at each login.

What else can NetworkManager do?

Beside managing your network connectivity, NetworkManager has another key feature. NetworkManager can run scripts when there is a network state change on any interface, using the network interface and the up/down state as variables. In prior releases, this functionality was provided by a separate daemon called NetworkManagerDispatcher. As of NetworkManager 0.70 in Fedora 9, this functionality is now integrated into NetworkManager itself.

In Bash scripts written for NetworkManager, the variable $1 equals the interface whose state has changed and triggered the script. Variable $2 equals the state of the interface (up or down). No other variables are needed.

Let’s take a look at one of the scripts that is included with Fedora 9:

# cat /etc/NetworkManager/dispatcher.d/05-netfs

#!/bin/sh

export LC_ALL=C

if [ "$2" = "down" ]; then
        /sbin/ip route ls | grep -q ^default || {
                [ -f /var/lock/subsys/netfs ] && /etc/rc.d/init.d/netfs stop
        }
fi

if [ "$2" = "up" ]; then
        /sbin/ip -o route show dev "$1" | grep -q '^default' && {
                /sbin/chkconfig netfs && /etc/rc.d/init.d/netfs start
        }
fi

When an interface comes up and adds itself as the default route, the script starts the netfs service. This script also stops the netfs service when an interface goes down and no default route remains. Effectively, this will mount your NFS and CIFS shares when you have access to the network, and will unmount those same shares when the network goes down. Using this script as an example, you can easily write your own scripts to run various commands as the network state changes.

How can I best use NetworkManager in the field?

Now that you have a good idea of how NetworkManager works and what it can do, let’s talk about how to best use NetworkManager in the field. Now that you have NetworkManager managing your network connectivity, make sure your network interfaces are not trying to start on boot. Nothing is more annoying than having your laptop tell you that your wired network is not available when you are sitting on a plane. If you are using NetworkManager 0.70 (currently in Fedora 9), you should also disable the network service itself, as it may conflict with NetworkManager.

You can go further, writing NetworkManager scripts to activate various services only when they are needed. Many of the init scripts in Linux make the assumption that your system is a server or a workstation with continuous access to the network. Things like ntp, cups, sshd, even rhnsd do not need to be running while you have no network connectivity. These services can be disabled, set to run only when NetworkManager starts them via a custom script on a network state change.

Using the previously posted script as a guide, a script to manage sshd might look like this:

# cat /etc/NetworkManager/dispatcher.d/10-sshd

#!/bin/sh
#
# Start and stop sshd based on network availability using NetworkManager
#

export LC_ALL=C

if [ "$2" = "down" ]; then
        /sbin/ip route ls | grep -q ^default || {
                [ -f /var/lock/subsys/sshd ] && /etc/rc.d/init.d/sshd stop
        }
fi

if [ "$2" = "up" ]; then
        /sbin/ip -o route show dev "$1" | grep -q '^default' && {
                /sbin/chkconfig sshd && /etc/rc.d/init.d/sshd start
        }
fi

You could substitute “rhnsd” or “cups” for “sshd”, and the script should work equally well for those tasks..

If you are a administrator tasked with managing Red Hat or Fedora systems of remote employees, the scripting functionality can be even more handy. You can write a script that looks for the activation of the VPN interface then sends an email letting you know the system is online. You could have the system check in with a Satellite server located within your firewall, installing updates you previously scheduled for it. The possible uses here are many.

The student is now the master

No longer do I envy my Windows-based peers and their easy mobile connectivity. NetworkManager is constantly impressing me, adding functionality and allowing me to be more efficient on the road. This Swiss Army knife of Linux networking gives me the control I need over my connectivity whether at home, coffee house, or airport. Now that you know what NetworkManager is, how it works, and how best to use it, try it out of your own system. I trust you will find NetworkManager works as well for you as it did for me.

More information

1. NetworkManager main project page
2. NetworkManager in Fedora
3. dbus and hal
4. KNetworkManager

The first talk I had a part in was Func, which I co-presented with Adrian Likins. Func, as mentioned before in Red Hat Magazine, is an API for controlling lots of nodes for arbitrary systems management tasks. It is ideally suited for replacing legacy SSH infrastructure as well as building new network applications that require secure network communications infrastructure.

We gave an overview for folks that hadn’t heard of Func before, and showed off several examples of things you can do with the Python API. Interest in Func is growing, and lots of folks are using it in ways we hadn’t originally intended (which is, of course, the idea).

One such application that surprised us was Open Symbolic, which aims to be a user-friendly systems management application written on Func. Additionally, some of our own IT folks have written a very lightweight “cloud” type management application using Func, which you should hear more about shortly. At the conclusion of that talk there were some very good questions about how things work, and also some interest in future expansions (with hopefully some patches coming down the pipe).

My second talk was about Cobbler, our next-generation installation server. I was very pleased to see Cobbler’s talk was standing-room only, showing that there is a lot of need for good tools to help streamline Linux installations and low-level datacenter setup tasks. Again, there were a tremendously good set of questions asked, and the community is already getting a lot larger with people trying out Cobbler after hearing about it at Summit. I’m happy to see that.

The current idea behind Cobbler is to build a common installation service that various applications can use to deploy Linux (more on this below)rather than having to keep creating these components. I’ll also be extending it to deal with image-based deployments in addition to traditional kickstart deployments.

As always, we’ll continue to gather a large community of sysadmins to work together and build common tooling we can all share. The basic premise is that by working together, we can build tools that are more powerful than the tools we could build alone. By following an open development model with shared tools, the need to reinvent the same wheels to do the same tasks goes away.

We’re also building a large set of community documentation based on deployment best practices and tricks, and I hope to roll this into a nice openly licensed manual and deployment guide later this year. Open community, open code, open docs — neat!

And while all of this is nice, the most exciting part of Summit was the announcement about Spacewalk. Spacewalk is the project name for the open-sourcing of Red Hat Satellite Server. This is not a “community edition”, it’s everything. The upstream for all code in the project is going to the website, and we’re already seeing a lot of interest from existing Satellite customers and prospective users. We had a lot of great discussion at FudCON about places the project might go, and there have already been several non-Red Hat patches made against the codebase.

We also announced that Satellite will be using Cobbler to provide advanced deployment support next year, so we can already see the advantages of getting these tools to work together. Planned work includes adding open source database alternatives and improving our support for Fedora and derivative distributions.

In the future, Func may have a place in Spacewalk as well. We can also look at incorporating other proven open source management applications, tools, and libraries. Everyone is encouraged to join the Spacewalk mailing lists or IRC channels (#spacewalk on irc.freenode.net) if they are interested in learning more or have ideas/questions.

These three applications, in general, constitute a new way of thinking. Namely, how can we apply the Fedora Development model towards enterprise management applications and sysadmin-level tools and processes?

Rather than buy tooling from a vendor or inventing internal frameworks that disappear when one changes jobs or departments, what can we share and openly build together and re-use? While we use the Fedora model to produce a community-driven distribution, we’re now also using it to produce better ways to manage Enterprise Linux. This development model applies just as equally to Enterprise Linux as it does Fedora. It just works.

As Paul Frields (the Fedora Project Leader) indicated at FudCON, the future of Fedora (and in turn, Fedora Hosted Projects and Extras Packages for Enterprise Linux (EPEL)) is about lowering barriers to entry and encouraging collaboration–we are already doing well on technical grounds. How do we encourage more collaboration around management software? For starters, we work at making it easier to install. And we must make sure the communities remain active and integral, and continue to grow.

A lot of management software is hard to find and install because it’s not yet part of the distribution. I would encourage all ISVs out there that write open source software to look at getting their content into EPEL, where it can be easily found via search and is yum-installable. I’d also like them to take a look at how we are building strong communities around our projects using Fedora Hosted Infrastructure–and to take those examples to their own projects and communities.

Regardless of who we work for, open communication and collaboration are how we get ahead–and Fedora (and EPEL) are providing an outstanding breeding ground for open source software now–even in what was previously thought of as an “Enterprise” type cathedral environment. I should also mention that the Fedora community is full of packaging experts and smart folks who can help you. Join #fedora-devel on irc.freenode.net and say hi.

If you’re not interested in development, but adminster systems and want to know how you can help out, take a look at Mike McGrath’s new CSI venture.

Good ideas come from where we least expect them. The Fedora development model and infrastructure are really powerful ways to create software, documentation, and tools that solve problems for people who need to manage their Linux infrastructure. Whether you are interested in Cobbler, Func, or Spacewalk–or even if you’re interested in Linux systems management in general–let’s get together, find better ways to collaborate, and invest in the operating systems that we all love.

So, that’s my summary of the Red Hat Summit and FUDCon. Boston’s a beautiful city and I will give it a pass for not having sweet tea at all available restaurants. I look forward to Summit and FUDCon next year–we’re growing fast in Fedora and Red Hat Enterprise Linux land, and there’s still many more places to go.

With new hardware releases, customers are faced with situations in which they want to take advantage of increased speeds but are forced to stay on older hardware because their operating environments are not supported on the newer hardware. Virtualizing their operating environment helps them get past this issue. Virtualization also helps them:

• Consolidate hardware to
• improve utilization
• reduce floor space requirements
• reduce power consumption
• Take advantage of hardware speed up without having to upgrade the software environment
• Reduce downtime for upgrades
• Create development and test environments

RHEL 5 virtualization lets customers virtualize their existing systems and take advantage of the benefits mentioned above.

There are two modes to virtualize servers:

• Para-virtualized servers. In this mode, the virtualized server has direct access to the hardware/hypervisor and delivers performance close to bare metal. Systems running Red Hat Enterprise Linux 4.5 and newer can be deployed as para-virtualized guests.
• Fully-virtualized servers. In this mode, the virtualized server interacts with the hypervisor through a hardware abstraction layer. The hypervisor presents its hardware as generic hardware through the abstraction layer so most operating systems can be run in a virtual mode on it. However, all the I/O, network, and memory requests from the guest have to be translated by the hypervisor. This translation results in very poor performance by systems deployed as fully virtualized guests.

The I/O and network performance on fully virtualized guests can be improved by implementing para-virtualized drivers within these guests.

But memory access can still be an issue, particularly with process-based applications where processes need to modify virtual memory. Specifically, the guest operating system (guest OS) in the virtual machine creates page tables that translate virtual memory addresses to guest pseudo-physical addresses, which means that older CPUs cannot directly use them. To deal with that limitation, the hypervisor has to create so-called shadow page tables, which translate the same virtual memory addresses to real physical addresses. Maintaining these shadow page tables can cause performance and scalability problems with certain workloads. One particular cause of performance issues is the fact that the hypervisor needs to intercept each page table write by the guest OS in order to keep the guest page tables and the shadow page tables in sync.

The AMD Barcelona processor has a feature called Rapid Virtualization Indexing (RVI) which allows the processor to directly use the virtual to pseudo-physical address page tables created by the guest OS. This works because the CPU translates these pseudo-physical addresses to real machine physical addresses using a second set of page tables, which define this translation for each virtual machine. Because the CPU can do both of the memory address translations, no shadow page tables are required and the guest OS can alter its page tables directly, without trapping to the hypervisor. Context switches in the guest OS also benefit from RVI technology because the shadow page table gets flushed with most context switches on the guest and there is a cost incurred to re-populate it.

All systems running Red Hat Enterprise Linux 4.4 or older have to be run as fully virtualized guests, as do all other operating systems (e.g. Windows, Solaris, etc.).

A series of tests on RVI were carried out on an HP DL585 system using an OLTP workload in an Oracle database. Oracle was chosen as the database for the testing because it is a process-based workload, and it is also a widely used database. The testing was carried out with 8 and 16 CPUs to understand the effects of vertical scaling on fully virtualized guests and the benefits of RVI.

Hardware used for the testing

System: HP DL 585
Memory: 32 G
CPUs: 16 AMD Barcelona @ 2.3 GHz

Test results from 8 CPU testing

For the purpose of comparison, the transactions per minute generated with 20 users was baselined at 100, and all other numbers are relative to that number. The table and the graph above show the remarkable advantage that PV drivers and RVI provide to a fully virtualized guest. Line 2 in the chart, which represents the FV guests without RVI or PV drivers, shows a huge drop-off in the transactions per minute relative to line 1, which shows the baseline of transactions per minute on Dom0. By turning on the RVI feature, some of the performance is regained. But without the PV drivers, the performance remains way below the mark, as shown on line 3. Line 4 shows that by adding PV drivers without RVI, the performance remains way off. Finally line 5 shows that by using RVI and PV drivers, the performance of the workload gets within 80% of Dom0.

Test Results from 16 CPU testing

The results from the 16 CPU testing shows similar trends to the 8 CPU testing, but the delta gets wider as the user count is increased when RVI and PV drivers are not used.

Figure 3 below shows the comparison between the 8 and 16 CPU runs. All the numbers are relative to the 8 CPU numbers without PV drivers and RVI, which has been baselined at 1. The figure shows that as the guest size increases, the performance of the FV guest without RVI and PV drivers decreases. With RVI and PV drivers, performance inside the guest comes close to 80% of the performance on Dom0.

Comparison between 8 CPU and 16 CPU FV guests

Conclusion

From the results it is clear that when a process-based workload is run in a fully virtualized guest, performance takes a big hit due to the way the TLB information is maintained and the way I/O works inside the guest. But by adding PV drivers on an AMD Barcelona based system which uses RVI, performance aligns with Dom0 performance. With this feature, AMD lets customers take advantage of hardware speed-up without having to upgrade their software environments by letting them run their systems as fully virtualized guests on the Barcelona-based systems.

Intro

Responsible for 1,000 systems? One hundred systems? Ten? If so, you likely have processes in place for maintaining these systems, if only to preserve your sanity! Perhaps you have custom ssh scripts to command the systems remotely, or maybe you have your own yum repositories to maintain software patches critical to your systems. If the burden of maintaining these systems causes you a headache or your needs go beyond the methods you use today, Red Hat has tools available to make your life as a system administrator easier.

Satellite

Red Hat Network Satellite is a systems management platform that will make the deployment of your Linux systems easier, faster, and more scalable. Satellite automates many of the day-to-day operations that would otherwise require manual script writing, remote ssh execution, and a considerable amount of human effort. Satellite can centralize the storage and deployment of Red Hat products alongside your own custom software content in one server for easy and controlled distribution to as many systems as you desire.
Users deploy Satellite in environments ranging from tens to tens of thousands of Linux systems.

If any of the features below sound useful to you, give Red Hat Network Satellite a try:

• One-click software updates in an easy-to-use web interface
• Role-based administration
• Flexible delivery architectures - Satellite, Proxy, and Hosted
• Virtual Machine Management - Provision, control, and configure virtual machines
• System grouping for easier administration
• Automation of previously manual tasks
• Life-cycle management of your entire Linux infrastructure
• Performance tracking for your Linux systems

Latest features in Satellite 5.1.0

With the latest release of Red Hat Network Satellite, 5.1.0, we introduced some excellent new features to help Satellite better integrate with your environments. The most notable feature we introduced in the release is the Multiple Organizations feature, sometimes referred to as ‘Multi-Org.’ This feature allows you to partition your servers, users, software, and configuration into separate ‘Organizations’ within the Satellite. Each organization can manage their own sets of users and systems allowing you to control access to your data without having to install and maintain separate Satellite servers.

An example usage of this feature would be a user that wants a Satellite for managing systems in a multi-departmental organization. Each department would be given its own separate Organization within the Satellite: one organization for Finance, one for IT, one for Engineering, and so on. Each organization would then be granted a set amount of licenses (’entitlements’) to Red Hat Enterprise Linux as determined by a central Satellite administrator. Likewise, the Satellite Administrator would create administrators for each Department within the Satellite. Each department would then have rights to register and administer their
systems without fear of users in other departments seeing or manipulating their systems.

Other exciting features in 5.1.0 include:

• API Call enhancements: Vastly expanded set of XML-RPC APIs to help you better automate your experience with RHN Satellite.
• Satellite support for x86 64-bit (Intel/AMD) and z390/x (IBM) platforms. You can run your Satellite on more hardware than ever before!
• Provisioning support for Power PC (PPC) platform: Kickstart your PPC boxes from Satellite.
• Apache 2.0 support for Satellite and Proxy on RHEL4
• Exporter tool for moving RHN configuration information
• Web User Interface (UI) enhancements: Expanded CSV export support and performance enhancements.

Last but not least

Big news in the world of RHN Satellite: We are going open source! Our development, source code, and communications will all be done in the open with GPLv2 licensing. Much more information at our website:

http://spacewalk.redhat.com/

Attend one of our Satellite lab sessions to talk one-on-one with developers on our team and get the latest information.

More information

Red Hat press release webcast

Red Hat press blog announcement

Developer Blogs: Jesus Rodriguez, Devan Goodwin

Red Hat Delivers on Linux Automation with Identity Management and Open Source Systems Management Solutions

Red Hat Network to Be Open-sourced

“You want to know the run-levels for Red Hat® Enterprise Linux® or Fedora®?” You pause, thinking. “Well, I use Linux every day, and I know single user mode is level 1…”. You stammer a bit, and say you’re drawing a blank for the rest. The recruiter thanks you in that “sorry” tone-of-voice, and hangs up the phone.

Let’s cross that question off the recruiter’s list forever. (Sorry, recruiters.) In this article, we cover how to create, use, modify, and ultimately master run-levels. Bookmark this page with your favorite bookmarking service, and rest easy about ever missing that interview question again.

Of course, there are more reasons to know about run-levels than just to pass an interview. Interacting with run-levels is quite useful once you get used to it. In this article we are going to cover the basics, and then go beyond that to create our own run-level that we write a script against.

What’s a run-level?

A run-level is a system state that is defined by the services listed in /etc/rc.d. Typically, advanced administration of a machine is done by switching run-levels (or state) to perform tasks such as minimal resource usage, run-level 3, shutdown run-level 0, or maintenance. Maintenance is run-level 1–or single user mode, as it is commonly known.

Unix/Linux run-level comparison chart

If you happen to work with AIX, Solaris, HP-UX, Ubuntu, Fedora, Free BSD and/or Red Hat Enterprise Linux, then it can get a bit confusing figuring out which run-level does what. Please refer to the Wikipedia reference at the bottom of this article to get more information on cross platform run-level comparisons.

Red Hat run-level chart

Here is a chart of Red Hat-specific run-levels:

What run-level am I?

Just like life, with an operating system you need to know where you are now, in order to get to where you want to go next. If you are planning on modifying your run-level, you need to first know what your current run-level is. In order to do this, you can use one of two commands, like so:

[root@localhost ~]# who -r
         run-level 3  2008-04-29 08:17                   last=5
[root@localhost ~]# runlevel
5 3

If we look at the output of who -r, we can tell that we are currently running at run-level 3–which is multi-user, but console only.We can also tell that we were previously running at run-level 5, which is multi-user with console and X11 login.

Changing run-levels

Once you know what run-level you are at, it is very simple to change to a different one. All you need to do is type: “init” followed by number of the runlevel you would like to switch to. Here is an example of switching to single user mode, or runlevel 1:

init 1

This command will change your system to single user mode, and it will ask you for the root password. When you arrive in single user mode, there are no services running, as this level it is most often used for maintenance, backup, or recovery. Once you are in single user mode it is quite common to enable, for example, network and NFS to backup your operating system, like so:

service network start; service nfs start

When you are done with your work, type in the run-level you would like to go to–perhaps run-level 5 which brings up the X11 login window:

init 5

Later in this article, we will write our own run-level and then use it to script a maintenance operation.

Permanently changing the default run-level

While changing the runlevel manually is most common, sometimes it is useful to change the default run-level from level 5 to level 3 permanently. This can help conserve resources inside of, for example, a virtual machine. You may also choose to define your own custom run-level, and wish to make that the default.

You will need to edit /etc/inittab and change this line with your favorite text editor:

id:5:initdefault:

Change ‘5′ to the run-level you wish your machine to be at when it boots. To change the run-level so that it never loads the GUI on boot would look like this:

id:3:initdefault:

Note:

A word of caution on editing /etc/inittab. It is very important to keep /etc/inittab in version control, and/or keep a backup of it when you are editing the file. If you make a change incorrectly you can render your operating system unbootable.

Tip:

If you happen to get yourself in this pickle, there is a way out. You can interrupt the Grub boot loader and press “A”, and then append the word “emergency” to the end of the kernel arguments. This will boot the operating system without using init. Then, you can fix what you altered by copying back the original version of /etc/inittab.

Creating your own run-level HACK

First, a word of caution. Do not do this on a production machine, period! This section is a VERY dirty hack that you should only use on a virtual machine you can experiment with, or a machine you don’t mind rebuilding.. It is always a good idea to do testing inside of a virtual machine before doing something that could potentially render a box unbootable. This is a very quick and dirty way to alter a run-level for the purposes of learning, but perhaps you can get some ideas from it that can be used in a more production-oriented way. Ideally, some of the readers of this article will post some production quality hacks to creating custom run levels.

1. cd to /etc/rc.d/rc4.d/
2. do a sanity check to make sure you are running Red Hat: cat /etc/redhat-release
3. backup existing run-level directory:

   mkdir /tmp/rc4.d.original/
   cp /etc/rc.d/rc4.d/* /tmp/rc4.d.original/
   

4. rm -f /etc/rc.d/rc4.d/*

At this point

 cp /etc/rc.d/rc1.d/* /etc/rc.d/rc4.d/

We have now copied the run-level scripts for single user mode into our own custom run-level 4. We can hijack the the S99single script and tell it to do something different. In this example, we are going to write a custom Python script that gets forked to the background and backs up the machine over rsync. Let’s edit that file we copied:

vim /etc/rc.d/rc4.d/S99single

Change the last part of it to look like this:

# Now go to the single user level.
echo $"Telling INIT to go to single user mode."
echo "This is a custom code. Forking custom script"
/custom.py &
exec init -t1 S

We’ve inserted two lines. One echoes that we are forking off a custom script. The second line forks a python script, shown below, that backs up the machine via rsync. Note that this assumes you have set up ssh keys on the remote backup server.

custom.py script:

#!/usr/bin/env python
import time
import subprocess

rsync = "rsync -av / 10.0.1.3:/Volumes/Backup/server_backup/"
network = "service network start"
init = "init 3"

cmds = [rsync, network]

def single_user_backup():
    """Starts network service, creates backup and returns to init 3"""
    try:
        subprocess.call(network, shell=True)
        subprocess.call(rsync, shell=True)
    finally:
        subprocess.call(init, shell=True)

def main():
    """Runs program"""
    print "sleeping for 60 seconds"
    #time.sleep(60)  #Gives machine time to quiesce
    single_user_backup()

main()

The main function runs a sleep command for 60 seconds, just to give the single user mode scripts time to quiesce the box. Remember, this script is forked to the background. Next, function single_user_backup attempts to start network services and run rsync to remotely back up the whole / volume to another server. This is obviously crude and there will be lots of errors trying to back up /proc, for example, but it give you an idea of how an automated backup could work with a custom run-level. Finally, the machine gets called back to init 3, which is console only multi-user mode.

Tip:

Again, this is just an idea for a backup script, but not one I would actually run in production in my wildest dreams. One problem with this technique is that because of symbolic links in run level 1, we actually, changed run level 1 and our run level 4. This is not acceptable, obviously, for any sane user, but it is acceptable as a way to have fun with a disposable virtual machine!

If you can think of a more realistic backup script that would work from a custom run-level, I would love to see it. Create a how to on your blog, and then post a response to this article. Also, it would interesting to see other things such as database backups and migration done with custom run-levels as well. Leave a comment and let me know what you’d do.

Running your own run-level

To run the newly created run-level, you only need to type:

init 4

You will then see the custom print statements we inserted. The machine
will sleep for 60 seconds, and then run the rsync backup.

Summary

This article covered quite a bit of ground in a short while. We went over what a run-level was, how to tell what run-level you are at, how to change run-levels, and, finally, how to make your own run-level with custom, frankenstein quality, code. Hopefully, this showed you some new tricks and spurs some ideas for further innovation with run-levels.

References

• Section 19.1. Runlevels (Red Hat Enterprise Linux 4 manual)
• Wikipedia: Runlevel
• Wikipedia: Init
• Python for bash scripters: A well-kept secret (RHM, Feb 2008)

About the author

Noah Gift is the co-author of Python For Unix and Linux by O’Reilly Publishers. He is an author, speaker, consultant, and community leader, writing for publications such as IBM Developerworks, Red Hat Magazine, O’Reilly, and MacTech. His has both a consulting company and a personal website. Noah is also the current organizer for the www.pyatl.org”>Python User Group for Atlanta, GA. He has given presentations at PyCon and PyAtl. In his free time, he enjoys spending time with his wife Leah, and their son Liam, playing the piano, and exercising religiously.