We here at Red Hat are pleased to bring you a brand new set of videos aimed at showing off the latest and greatest enhancements in our technologies–featuring the very people who helped create them in the first place. The “SPOTLIGHT ON” series highlights the ways in which collaboration drives innovation by looking at projects that have been improved by community input. In our first installment, we track down Red Hat’s own Karsten Wade and Stephen Smoogen from the University of New Mexico to talk about Extra Packages for Enterprise Linux (EPEL), the Fedora-sourced repository of add-on packages for Red Hat Enterprise Linux. They discuss how EPEL is a tool for user-driven innovation that comes from and benefits enterprise customers with more stable code and lower business costs.

Background

The dd command is one of those ancient UNIX tools that is extremely powerful, yet at the same time, the syntax can make it feel slightly archaic. A lot of seasoned sysadmins and developers still remember the first time they saw the dd command used by a bearded wizard. He might have used it to test the disk I/O, capture a disk image, or restore it.

In some ways, dd can seem like Old Spice–only the guys over 60 use it. But the younger generation should know that dd still has some tricks up its sleeve. In this article, we’re going to put a new twist on this old favorite and show how grandpappy really does know best sometimes. The new twist is to mix dd with Python and the Google Chart API to make a UNIX 2.0 mashup tool. (”UNIX 2.0″ is a play on words for what happens when you change the original behavior of a tool like dd to make it do something a bit different.)

Setup

For this article, we assume you’re running Fedora Core 8. We’re actually just renting some time from Amazon in all of these examples. To do that we allocated a 1 GB Elastic Block Storage volume from Amazon and attached it as the device /dev/sdd to an Amazon Machine Instance (AMI) running Fedora Core 8. Learn more about using Amazon Cloud Computing with Red Hat.

Using dd for disk benchmarking with Google Charts API and Python

We benchmarked the throughput of the disk by running the dd command with various block sizes from 128 KB to 1 MB. (Note: If you want to run the script on your own machine, make sure that the volume you use doesn’t contain any valuable data, because the data will be erased by the dd command. Remember, data loss makes grandpappy mad!)

For the benchmark, we wrote a Python script that uses the commands module to run and capture the output of the dd command. The script also uses the csv module to generate a comma-separated values file so that we can graph the results later. For this example, we chose to graph the results using the Google Chart API.

#!/usr/bin/env python
import commands
import re
import csv

Next we define the main function, which takes a device name and a block size as parameters, and returns the throughput measured with dd and the unit of measure (e.g. MB/s). We use the regular expression module (re) to isolate the throughput value and unit of measure from the output of the dd command.

Editor’s note: In the code below, unit = "" has been added since the article was posted.

def get_disk_throughput(device, blocksize):

    blocksize = str(blocksize) + 'k'

    cmd = "dd if=/dev/zero of=%s bs=%s" % (device, blocksize)

    output = commands.getoutput(cmd)

    throughput = 0

    unit = ""

    for line in output.split('n'):

        s = re.search(' copied,.*, (S+) (S+)$', line)

        if s:

            throughput = s.group(1)

            unit = s.group(2)

            break

    return (throughput, unit)

Here is the portion of the script that is executed when it’s run from the command line. We open a csv file and associate it with a csv writer. We then use the writerow method of the writer to append the header and each data row. We iterate over the list of block sizes and call the get_disk_throughput function for each block size.

We also compose the Google Chart URL by filling in the exact data values, represented by the throughput numbers that we obtain from the get_disk_throughput function. Then we print the URL to stdout. If you check the URL, you’ll see the chart generated with our data.

For details on the Google Chart API and what each parameter to the URL represents, see the Developer’s Guide.

Editor’s note: This is an updated version of the code that originally appeared with this article. It has command-line argument processing, and it composes the Google Chart URL in a better, more self-explanatory fashion. See the end of the article for the original.

f = open('disk_throughput.csv', 'w')
        writer = csv.writer(f)
        writer.writerow( ('Block size (KB)', 'Throughput') )
        blocksizes = [128, 256, 512, 1024]
        gchart_url = "http://chart.apis.google.com/chart?"
        gchart_type = "cht=bvs"
        gchart_title = "&chtt=Disk%20throughput"
        gchart_size = "&chs=400x250"
        gchart_axis_labels = "&chxt=x,y"
        gchart_data = "&chd=t:"
        gchart_labels = "&chl="
        max_t = 0.0
        for blocksize in blocksizes:
            (t, u) = get_disk_throughput(device, blocksize)
            if float(t) > max_t:
                max_t = float(t)
            writer.writerow( (blocksize, t) )
            print 'Block Size: %sk Throughput: %s %s' % (blocksize, t, u)
            gchart_data += t + ","
            gchart_labels += str(blocksize) + "k" + "|"
        gchart_data = gchart_data.rstrip(',')
        gchart_labels = gchart_labels.rstrip('|')
        gchart_axis_range = "&chxr=1,0," + str(max_t+10.0)
        gchart_scaling = "&chds=0," + str(max_t+10.0)
        gchart_url += gchart_type + gchart_title + gchart_size + gchart_data + gchart_labels
        gchart_url += gchart_axis_labels + gchart_axis_range + gchart_scaling
        print "Google Chart URL (just paste in a browser):", gchart_url
    finally:
        f.close()

Here is the output of the script in one of our runs:

Block Size: 128 Throughput: 62.8 MB/s

Block Size: 256 Throughput: 61.8 MB/s

Block Size: 512 Throughput: 57.1 MB/s

Block Size: 1024 Throughput: 56.5 MB/s

Now here is the actual image that gets created:

Full script

#!/usr/bin/env python

import sys
import commands
import re
import csv
from optparse import OptionParser

def get_disk_throughput(device, blocksize):
    blocksize = str(blocksize) + 'k'
    cmd = "dd if=/dev/zero of=%s bs=%s" % (device, blocksize)
    output = commands.getoutput(cmd)
    throughput = 0
    unit = ""
    for line in output.split('n'):
        s = re.search(' copied,.*, (S+) (S+)$', line)
        if s:
            throughput = s.group(1)
            unit = s.group(2)
            break
    return (throughput, unit)

if __name__ == "__main__":

    usage = "usage: %prog options"
    parser = OptionParser(usage=usage)
    parser.add_option("-d", "--device", dest="device",
            help="Disk device to operate on (NOTE: any data on that device will be lost)")
    (options, args) = parser.parse_args()
    device = options.device
    if not device:
        parser.print_help()
        sys.exit(1)

    try:
        f = open('disk_throughput.csv', 'w')
        writer = csv.writer(f)
        writer.writerow( ('Block size (KB)', 'Throughput') )
        blocksizes = [128, 256, 512, 1024]
        gchart_url = "http://chart.apis.google.com/chart?"
        gchart_type = "cht=bvs"
        gchart_title = "&chtt=Disk%20throughput"
        gchart_size = "&chs=400x250"
        gchart_axis_labels = "&chxt=x,y"
        gchart_data = "&chd=t:"
        gchart_labels = "&chl="
        max_t = 0.0
        for blocksize in blocksizes:
            (t, u) = get_disk_throughput(device, blocksize)
            if float(t) > max_t:
                max_t = float(t)
            writer.writerow( (blocksize, t) )
            print 'Block Size: %sk Throughput: %s %s' % (blocksize, t, u)
            gchart_data += t + ","
            gchart_labels += str(blocksize) + "k" + "|"
        gchart_data = gchart_data.rstrip(',')
        gchart_labels = gchart_labels.rstrip('|')
        gchart_axis_range = "&chxr=1,0," + str(max_t+10.0)
        gchart_scaling = "&chds=0," + str(max_t+10.0)
        gchart_url += gchart_type + gchart_title + gchart_size + gchart_data + gchart_labels
        gchart_url += gchart_axis_labels + gchart_axis_range + gchart_scaling
        print "Google Chart URL (just paste in a browser):", gchart_url
    finally:
        f.close()

Summary

In this article we shattered the myth that you must be 60, have a massive grey beard, and have worked at Bell Labs to use the dd command. Even for a newer generation, dd can be used in some inventive ways. We combined Python, the Google Chart API, and Red Hat on Amazon’s cloud computing infrastructure to create a novel way to measure and chart disk I/O and performance. Go celebrate by buying yourself a bottle of Old Spice.

References

Python: http://www.python.org/
dd example scripts: http://tldp.org/LDP/abs/html/extmisc.html
Google Chart API: http://code.google.com/apis/chart/

Original code

if __name__ == "__main__":

    try:

        f = open('disk_throughput.csv', 'w')

        writer = csv.writer(f)

        writer.writerow( ('Block size (KB)', 'Throughput') )

        device = '/dev/sdd'

        blocksizes = [128, 256, 512, 1024]

        google_chart_url = "http://chart.apis.google.com/chart?cht=bvs&chd=t:"

        google_chart_data = ""

        google_chart_labels = ""

        max_t = 0.0

        for blocksize in blocksizes:

            (t, u) = get_disk_throughput(device, blocksize)

            if float(t) > max_t:

                max_t = float(t)

            writer.writerow( (blocksize, t) )

            print 'Block Size: %s Throughput: %s %s' % (blocksize, t, u)

            google_chart_data += t + ","

            google_chart_labels += str(blocksize) + "k" + "|"

        google_chart_data = google_chart_data.rstrip(',')

        google_chart_labels = google_chart_labels.rstrip('|')

        google_chart_url += google_chart_data +"&chl=" + google_chart_labels

        google_chart_url += "&chtt=Disk%20throughput" +"&chs=400x250&chxt=x,y"

        google_chart_url += "&chxr=1,0,%s&chds=0,%s" % (str(max_t+10.0), str(max_t+10.0))

        print google_chart_url

    finally:

        f.close()

Authors

Noah Gift is the co-author of Python For Unix and Linux by O’Reilly, and Google App Engine in Action by Manning. He is an author, speaker, consultant, and community leader, writing for publications such as IBM Developerworks, Red Hat Magazine, O’Reilly, Manning, and MacTech. He has a master’s degree in CIS from Cal State Los Angeles, a B.S. in nutritional science from Cal Poly San Luis Obispo, and is an Apple and LPI certified sys admin. He’s worked at companies that include Caltech, Disney Feature Animation, Sony Imageworks, Turner Studios, and most recently, WetaDigital.

Grig Gheorghiu is the director of technology for RIS Technology, a web hosting company based in Los Angeles. Grig has 15 years industry experience, during which time he has worked as a programmer, research lab manager, system/network/security architect, IT consultant, and lead test engineer.

Grig is an active member of the Python and agile testing communities. He maintains a blog dedicated to agile testing, Python programming, and automated testing tools and techniques. Grig is the founder of the Southern California Python Interest Group, aka “the SoCal Piggies”. He lives in Los Angeles with his wife and two children.

If using a proxy server, Satellite will initiate connections to port 5222 on each proxy server connected. This is used for communications relating to the “OSA” service (also known as “Push to Client“). If using the OSA service, access to this port should not be restricted.

A Satellite service will also connect to individual client systems on port 4545 if the monitoring service be enabled on these specific systems. This port needs to be unrestricted if the RHN Monitoring service is being used.

For Proxy:

The RHN proxy server will connect to ports 80 and 443 on the “upstream” system, either RHN Hosted or an internal Satellite server.

If the Monitoring Scout is enabled on a proxy server, then the proxy will initiate connections to port 4545 on Monitoring-enabled client systems.

For Client Systems:

Client systems will initiate connections to ports 80 and 443 of their upstream RHN server (Satellite, Proxy or RHN Hosted).

If OSA is being used, then the client systems will also connect to the Jabber service running on the Satellite or Proxy server, on port 5269.

Red Hat’s customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

Everybody wants to be heard. Some of us have a rhythm to go along with it.

I was introduced to ccmixter a couple months ago and have been hooked ever since. It is a utopia of sound for music makers and mixers alike.

From acapellas and samples to remixes, ccmixter wants you to download, sample, cut-up, and share music of all types. Some artists post entire albums for the community to remix.

Best of all, it’s all licensed under Creative Commons so there’s no worries.

Check out my page of remixes.

If you create a remix and want to get in touch with Napoleon–or just have a question–feel free to leave a comment (or contact information) on this post.

# usermod -L <username>

2.) Change the password expiration date to 0 to ensure the user changes the password during the next login:

# chage -d 0 <username>

3.) To unlock the account after the change do the following:

# usermod -U <username>

Red Hat’s customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

The open source movement has pioneered the Collaborative Innovation trend, and it is no surprise that the rapid growth of the Internet and the equally rapid growth of the open source community have mirrored each other. The Linux® operating system and Wikipedia website are both good examples of open source projects that embody the ideals of Collaborative Innovation. And those in the technology industry aren’t the only ones to take notice. Policy makers and corporate leaders in all markets are exploring how this powerful trend can be harnessed for social and economic development.

Let us take Linux as an example. In September 1991, Linus Torvalds released 10,000 lines of source code under the General Public License (GPL). The GPL gives users four freedoms:

1. The freedom to run the program, for any purpose
2. The freedom to study how the program works, and adapt it to your needs
3. The freedom to redistribute copies so you can help your neighbor
4. The freedom to improve the program, and release your improvements to the public, so that the whole community benefits

Over the years, thousands of volunteers contributed to the code released by Torvalds. It is estimated that Linux now has around 100 million lines of source code and that the commercial value of this source code is approximately eight billion dollars. ^[1] This represents an enormous wealth of knowledge that is freely available.

The innovation that is possible through the efforts of thousands of people collaborating on the Linux source code is a powerful (and constantly growing) advantage for open source software. In the next few years, we may see the pace of innovation in open source outstrip anything that proprietary vendors and their closed group of paid programmers can produce.

Explaining this phenomenon, Tim O’Reilly says that, “Sustained innovation is no longer just about who has the most gifted scientists or the best equipped labs. It’s about who has the most compelling ‘architecture of participation.’”

Henry Chesborough, author of the book Open Innovation explains the contrast between the open and closed innovation models with this chart:

The term “Collaborative Innovation” may be new, but the concept and the practice have been part of Red Hat’s corporate philosophy since the company’s inception. For 15 years, Red Hat has applied this framework to successfully compete with proprietary software vendors who have built multi-billion dollar empires using the closed innovation model. The Fedora® Project is a prime example of Red Hat’s Collaborative Innovation strategy. Red Hat engineers work with the open source community to develop cutting-edge technologies for Fedora. When these innovative technologies mature, they are incorporated into Red Hat® Enterprise Linux.

By working with smart people inside and outside Red Hat, the company is able to create a transparent, cost-efficient model of technology development. Despite having just 2,600 employees, Red Hat has been able to build and provide world-class solutions that are deployed in demanding environments like the New York Stock Exchange and the Federal Aviation Administration as well as large government and private deployments across the world. While Red Hat does not generate all of the code that makes up Red Hat Enterprise Linux, the company is able to create value by providing services, training, and support around open source software. This is no mean feat considering that the primary operating system competitor is a deeply entrenched company that employs more than 50,000 people.

Eager to take full advantage of its possibilities, thought leaders across the world are applying the collaborative innovation model in areas like content (Wikipedia), medicine (Open Source Drug Discovery), scientific publishing (Public Library of Science), flexible copyrights (Creative Commons), and many other areas. Red Hat’s greatest contribution to the Collaborative Innovation movement–so far–has been its success in building a business model around open source software that can be replicated in other fields. With the Internet becoming an integral part of our lives, Collaborative Innovation is set to become one of the most important aspects of our future.

Related links

• Design Thinking, part 1 and part 2

About the author

Venkatesh Hariharan is Corporate Affairs Director at Red Hat and works on open source, open standards, and other policy issues. He is interested in the impact of technology on society. He co-founded IndLinux.org in 1999 and pioneered localization of Linux to Indian languages. He blogs at www.osindia.blogspot.com and his photos are at www.flickr.com/photos/venky7.

The raw devices interface has been deprecated in Red Hat® Enterprise Linux® 5. The rawdevices service and /etc/sysconfig/rawdevices file no longer exist and raw devices are now configured via udev rules. However the preferred method for performing raw I/O (ie. bypassing filesystem caching) is to open EXT3/EXT2 files with the O_DIRECT flag.

This is an excerpt from the raw command’s man page:

Although  Linux  includes  support  for rawio, it is now a deprecated interface. If your application performs device access using this
interface, Red Hat encourages you to modify your application to open the block device with the O_DIRECT flag. The rawio interface will
exist for the life of Red Hat Enterprise Linux 5, but is a candidate for removal from future releases.

• Creating the raw devices:
  Nevertheless, to create raw devices, add entries to /etc/udev/rules.d/60-raw.rules in the following formats:

  For device names:

  ACTION=="add", KERNEL=="<device name>“, RUN+=”raw /dev/raw/rawX %N”
  

  For major / minor numbers:

  ACTION=="add", ENV{MAJOR}="A", ENV{MINOR}="B", RUN+="raw /dev/raw/rawX %M %m"
  

  Replace <device name> with the name of the device needed to bind (such as /dev/sda1). “A” and “B” are the major / minor numbers of the device needed for binding, an “X” is the raw device number that the system wants to use.

  If there is a large, pre-existing /etc/sysconfig/rawdevices file, convert it with the following script:

  #!/bin/sh
  grep -v "^ *#" /etc/sysconfig/rawdevices | grep -v "^$" | while read dev major
  minor ; do
          if [ -z "$minor" ]; then
                  echo "ACTION==\"add\", KERNEL==\"${major##/dev/}\",
  RUN+=\"/usr/bin/raw $dev %N\""
          else
                  echo "ACTION==\"add\", ENV{MAJOR}==\"$major\",
  ENV{MINOR}==\"$minor\", RUN+=\"/usr/bin/raw $dev %M %m\""
          fi
  done
  

• Creating persistent raw devices for single path LUNs:

  If using unpartitioned LUNs, to create a single raw device for the whole LUN use this rule format:

  ACTION=="add", KERNEL=="sd*[!0-9]", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601601bd2180072193a9242c3dc11", RUN+="/bin/raw /dev/raw/raw1 %N"
  

  Set the RESULT value to the output of scsi_id -g -u -s /block/sdX (where sdX is the current path to the LUN).
  This will create the raw device /dev/raw/raw1 that will be persistently bound to the LUN with WWID 3600601601bd2180072193a9242c3dc11.

  If using partitioned LUNs, where raw devices are created for each of the partitions on the LUN, use this rule format:

  ACTION=="add", KERNEL=="sd*[0-9]", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601601bd2180072193a9242c3dc11", RUN+="/bin/raw /dev/raw/raw%n %N"
  

  Again, set RESULT to the output of scsi_id -g -u -s /block/sdX. This will create the raw device(s) /dev/raw/raw1, /dev/raw/raw2, etc. for each partition on the LUN and they will be persistently bound to the LUN with WWID 3600601601bd2180072193a9242c3dc11.

• Setting ownership and permissions on the raw devices:

  To set specific ownership and/or permissions for the raw devices, add entries to /etc/udev/rules.d/60-raw.rules in the following format:

  ACTION=="add", KERNEL=="raw*", OWNER=="root", GROUP=="disk", MODE=="0660"
  

• Testing and implementing the udev rules:

  Before implementing them, use the udevtest command to verify the udev rules work as expected. To verify that the raw device is created for a specific disk or partition, eg /dev/sdb1:

  [root@rhel5 rules.d]# udevtest /block/sdb/sdb1 | grep raw
  main: run: '/bin/raw /dev/raw/raw1 /dev/.tmp-8-17'
  

  To check ownership/permission settings for a particular raw device, eg /dev/raw/raw1:

  [root@rhel5 rules.d]# udevtest /class/raw/raw1 | grep mode
  udev_node_add: creating device node '/dev/raw/raw1', major = '162', minor = '1', mode = '0600', uid = '0', gid = '0'
  

  Finally, to actually create the raw device(s), use the start_udev command:

  [root@rhel5 rules.d]# start_udev
  Starting udev:                                             [  OK  ]
  

  Check that the raw device(s) have been created:

  [root@rhel5 rules.d]# raw -qa
  /dev/raw/raw1:  bound to major 8, minor 17
  [root@rhel5 rules.d]# ls -l /dev/raw
  total 0
  crw-rw---- 1 root   disk 162,  1 Jan 29 02:47 raw1
  

• Creating persistent raw devices for multipathed LUNs:

  Unfortunately it is not possible to write udev rules for creating raw devices on multipath devices (/dev/dm-*) without manipulating existing udev rules. Modifying existing rules for this purpose could cause unforeseen problems and is not supported by Red Hat Global Support Services.
  If absolutely necessary, an alternate method for creating raw devices on top of multipath devices could be to create the raw devices in /etc/rc.d/rc.local, so long as the raw device is not required before rc.local is executed. For example:

  /bin/raw /dev/raw/raw1 /dev/mpath/mpath1p1
  /bin/raw /dev/raw/raw2 /dev/mpath/mpath1p2
  

Note: Raw device support is not enabled on the s390 architecture.

https://bugzilla.redhat.com/show_bug.cgi?id=452534

Red Hat’s customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

You’ve seen him here before, but it’s been a while since he popped in for a visit. You can enjoy his earlier work here, here, here or here. (Or check out his entire RHM collection). Who is that masked man? It’s the Fedora Project’s Greg DeKoenigsberg. And who better to talk about this history of the Fedora than someone who has been involved nearly every step of the way…

By adding entries using devlabel, users can instead reference all devices by their symlink and no longer have to worry about what the true name of their device is. Devlabel works with both IDE and SCSI storage.

Devlabel first tries to find the uuid of the partition that you are attempting to add to devlabel by using the /usr/bin/partition_uuid program. Partition UUIDs are supported currently under ext2, ext3, xfs, jfs and ocfs. If no partition UUID can be found (or if it is not a partition you are adding) devlabel will then attempt to find either a SCSI uuid or IDE identifier.

To determine the unique identifier associated with a SCSI device, devlabel uses the program /usr/bin/scsi_unique_id. If this program cannot determine a unique identifier for your block device then the device cannot be used with devlabel.

Example usage of the devlabel command can be found below:

• Add a device to be managed by devlabel

  devlabel add -d <device> -s <link>
  

• Remove a devlabel symlink

  
  devlabel remove -s <link>
  

• Report the status of all devlabel symlinks

  devlabel status
  

• Get the UUID of a particular device

  devlabel printid -d 
  

Once a devlabel has been configured files such as /etc/fstab can be modified to reflect these changes and point to the symlink rather than the physical device itself.

The configuration file for devlabel is /etc/sysconfig/devlabel. This file tracks all the current device symlinks. It is not recommended to edit this file by hand.

The devlabel command is available from the devlabel package for Red Hat Enterprise Linux 3 and onwards.
For further information see the Red Hat Enterprise Linux System Administration Guide http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-devlabel.html

Red Hat’s customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.