LINUX - blktrace, blkparse, seekwatcher - how to run and install - and example script to run all nicely

SEEKWATCHER and BLKTRACE and BLKPARSE and example script 
############################################################
############################################################

blktrace helps analyze io to a drive while you run any command/script. blkparse helps put that in human readable output. Seekwatcher helps visualize the IO in a movie or picture (look at 5th and 6th links below).

NOTEseekwatcher generates visuals(pic/movie) from output traces generated by blktrace. it uses blkparse to make sense of it all and generate human readable output it can parse for times,read/writes, and locations on disk and to generate the visuals

Good articles for checking out first on blktrace:
Good articles for checking out first on seekwatcher:

Note: blktrace needs debugfs to work (more on this below) it allows trace information to be accessed by system (which is how blktrace runs its course). blktrace is first run to generate unreadable by human trace file, then blkparse can optionally be run on the tracefile to analyze the work (if you know how to read it - articles above describe it). then seekwatcher is run on output trace file to generate picture visualization or movie. OR you can use seekwatcher to run the blktrace program for you and then it will movie the picture/movie.

Note: when I used the word shell below that refers to the putty or xterm or teraterm etc.. terminal that you have open

PREREQS
########

apt-get install python
apt-get install python-dev
apt-get install blktrace
apt-get install python-matplotlib
apt-get install cython
apt-get install gcc

FOR THE MOVIES TO WORK:

apt-get install mencoder
or:
apt-get install png2theora 

NOTE: I used mencoder

CHECK PYTHON VERSION
=====================

Note: im sure this will work with other pythons

# python -V
Python 2.7.3

OTHER INFO ON MY PYTHON:
Python 2.7.3 (default, Jan  2 2013, 16:53:07)
[GCC 4.7.2] on linux2

GCC VERSION
============

Note: im sure this will work with other gccs

# gcc --version
gcc (Debian 4.7.2-5) 4.7.2

OTHER RELATED PROGRAMS I HAD INSTALLED (if curious)
===================================================

I noticed that seekwatcher was hard to get working with all of its python dependencies (im just a noob), so if your having troubles maybe try to match what I had installed python related. Those above are the ones that count, here is every python package I have just incase your curious:

# dpkg -l | egrep python | awk '{print $2}'
libboost-python1.42.0     libpython2.6     libpython2.7     python     python-apt     python-apt-common     python-aptdaemon     python-aptdaemon.gtk3widgets     python-aptdaemon.pkcompat     python-axiom     python-cairo     python-central     python-chardet     python-dateutil     python-dbus     python-dbus-dev     python-debian     python-defer     python-dev     python-gi     python-glade2     python-gnupginterface     python-gobject     python-gobject-2     python-gtk2     python-magic     python-matplotlib     python-matplotlib-data     python-minimal     python-mysqldb     python-nevow     python-newt     python-numpy     python-packagekit     python-pkg-resources     python-pycurl     python-pyparsing     python-software-properties     python-support     python-tk     python-twisted-conch     python-twisted-core     python-tz     python-xapian     python2.6     python2.6-minimal     python2.7     python2.7-dev     python2.7-minimal



DEBUGFS needs to be mounted
============================

Make sure you have this debugfs mount setup on your system - check like this:

# mount | egrep debugfs
OUTPUT SHOULD BE: debugfs on /sys/kernel/debug type debugfs (rw,relatime)

IF THIS OUTPUT IS MISSING:
# mkdir /sys/kernel/debug
# mount -t debugfs debugfs /sys/kernel/debug

INSTALLING
############

LATEST COMMIT all 3 should be same
https://oss.oracle.com/mercurial/mason/seekwatcher/archive/tip.tar.gz
http://oss.oracle.com/mercurial/mason/seekwatcher/archive/tip.tar.gz
https://oss.oracle.com/mercurial/mason/seekwatcher/file/b392aeaf693b/

mkdir /root/src
cd /root/src
wget http://oss.oracle.com/mercurial/mason/seekwatcher/archive/tip.tar.gz
tar -zxvf tip.tar.gz
mv seekwatcher* seekwatcher
cd seekwatcher
python setup.py install

Now seekwatcher should be installed into one your PATH directories so to run it you can just type "seekwatcher" without being in the correct directory and running "./seekwatcher" <--- (because thats annoying)

NOTES ON RUNNING BLKTRACE AND SEEKWATCHER
##########################################

Can run on disk where testing just dont do tests onto same spot as saving (irroneous data might come up maybe)
Do not run from directory where debugfs is mounted (debugfs being mounted just gives the right tools for blktrace)

blktrace needs to be specified the disks/device with -d, I see examples pointing to the /dev/sda instead of the working partition. In my example below I point to the partition ex: /dev/sda5, i still get results. Either way it should work. blktrace is on a higher layer as i understand from the reading.

METHOD1: RUN A TRACE WITH SEEKWATCHER AND GET OUTPUT IMAGE/MOVIE
#################################################################

The easiest way to run seekwatcher is to make it run blktrace for you.

Seekwatcher options:
-t <tracefile> saves output blktrace output of the command ran with -p, or uses the blktrace file specified as the input trace file
-o <outputfile> the image file or movie we will save
-p "command to run that happens on device pointed by -d"
-d <device to monitor> the harddrive we are monitoring
--movie: makes moves in .mpg format (use vlc to run them, and speed them up with vlc they look better)
-l label the traces (if doing multiple traces, each l corresponds to each t in order, 1st t corresponds to 1st l)

2 examples (What they do should be self evident based on the parameter descriptions I explained above):
seekwatcher -t find.trace -o find.png -p 'find / > /dev/null' -d /dev/sda5
seekwatcher -t dd.trace -o dd.png -p 'dd if=/dev/zero of=/mnt/file bs=1M count=4096' -d /dev/sda1 -d /dev/sdb1

2 more examples:
seekwatcher -t findallroot.trace -o findallroot.png -p 'find / -mount > /dev/null' -d /dev/sda5
seekwatcher -t findetcroot.trace -o findetcroot.png -p 'find /etc > /dev/null' -d /dev/sda5

METHOD2: BLKTRACE followed by SEEKWATCHER
###########################################

blktrace args that I use (the common ones are -d and -o listed below for device and output file, there is more like adjust the buffer with -b or -n which might be useful on big traces, the program will warn you if your buffer is too small at the end of the trace so you can rerun your test - it will tell you if you lost any events and how many percent of events were lost - i lost 0.1% of events once so I didnt redo the test for that small lost #)

-d <device> the device that will be read with blktrace for io. note blktrace goes between userland and devices so you can put the partition or device to analyze here like -d /dev/sda5 or -d /dev/sda in the parse of the trace (with blkparse) you can see when data goes from sda5 to sda - however going from sda5 to sda or seeing from sda that it got io coming from sda5 is irellevant for seekwatcher (it will still look like a write, or in the other direction a read)
-o <output file> the save file of the trace, note best not to save to the same device as reading from with -d option or else you get extra writes (if you dont mind seeing the writes on your results then its okay to leave it).

Noteblktrace - analyzes IO with very little overhead (2% CPU hit) output by itself is not human readable (some pieces are) - doing work with blktrace requires 2 shells (one for running blktrace and one for running the commands)
blkparse helps make blktrace output human readable

NoteThe above seekwatcher method used -p to give the command that was traced, however we can use blktrace to generate that command and skip the -p in seekwatcher.

Notefirst we need some blktrace output for seekwatcher if we are not using -p. that way we can do alot of operations in a shell rather then specify everything in a -p argument within seekwatcher

Note: if using seekwatcher with -p, dont forget -d for the device name (remember blktrace needs a -d all the time)

OUTPUT TO SCREEN (not useful with seekwatcher)  - METHOD1 ( no seekwatcher)
=======================================================================
# blktrace -d /dev/sda -o - | blkparse -i -
then in another shell run commands and you will see your io on blktrace shell. not useful for seekwatcher (Because there isnt a file seekwatcher can be point at)

OUTPUT TO FILE (can be used by seekwatcher) - METHOD2a (no seekwatcher)
=====================================================================
# blktrace -d /dev/sda -o trace123
Then do your operations in antoher shell when done go back to blktrace shell and press CONTROL-C
# ls -lisah

When done you will have a file called trace123 (it might have an extra extention added on, like trace123 will become trace123.blktrace.0)

To convert the output to humanreadable text
# blkparse -i trace123.blktrace.0

OR TO TAKE CARE OF EXTRA SUFFIX use *:
# blkparse -i trace123*

OR instead of running blkparse you can just run seekwatcher

BLKTRACE WITH SEEKWATCHER - METHOD2b (With seekwatcher)
=========================================================

First run blktrace (as described in above "OUTPUT TO FILE" section)
# blktrace -d /dev/sda -o trace123

To end the capture on blktrace go to the shell where its dumping all of the info and press CONTROL-C (it will show summary info) (or read Note and you can just kill the operation, just dont kill -9 it)

Note: you can kill blktrace (aka: properly stop blktrace) with SIGTERM instead of doing the CONTROL-C button press - it will have the same effect, find out its pid with "ps aux | grep blktrace" then "kill -15 PID" or "kill PID" (put in the actual pid # inplace of PID). When you kill with SIGTERM (signal 15) or CONTROL-C blktrace will spit out some summary information (im assuming if you used kill -9 you would cut blktrace and it wouldnt show you that information, so dont kill -9 the blktrace). Note kill by default (without any parameters) uses a -15 signal (SIGTERM signal). 

Note: Since we can use kill to properly stop blktrace, that means you can stick to using one shell by backgrounding blktrace with &. "blktrace -o trace123 -d /dev/sda &" (hitting enter after this will give you the pid of the backgrounded blktrace app on the terminal output so you dont have to look it up) and then do your operations you want to trace, then when done just "kill -15 PID" or "kill PID" (note the pid was shown to you, if you missed it, just type "jobs -l" to see all background processes started by you and their pids with -l argument - or of course just use "ps aux | grep blktrace" or a similar method)

3 examples (What they do should be self evident based on the parameter descriptions I explained above):
# seekwatcher -t find.trace -o find.png
# seekwatcher -t ext3.trace -t xfs.trace -l ext3 -l xfs -o ext3_vs_xfs.png
# seekwatcher -t ext3.trace -o ext3-trace.mpg --movie

Seekwatcher creates 3 graphs by default: a plot of each sector read or written, the number of seeks per second, and the throughput per second during the run. Sometimes the plot of each sector can be difficult to read, especially when comparing two runs with very different IO patterns. seekwatcher --no-io-graph will disable this plot. Another option is to use seekwatcher --zoom start:end to specify the range on the yaxis you want included in the graph. The range is specified in MB, and using a range of 0:0 will turn off seekwatcher's attempt at autoscaling the plot.

IF GET ERROR “bad fd error”:
=============================

ATTACHED: below is seekwatcher.py which has this fix applied to it - NOT TAKING ANY CREDIT FOR THE PROGRAMMING, Im just distributing it with the fix.

If you get this error - when running seekwatcher:

sh: 1: Syntax error: Bad fd number
Empty blktrace run found, exiting
The problem arises because /bin/sh is linked to “dash” and they use the wrong redirections. The fix is described here:

THIS LINK:

OR THIS LINK:

You have to change the line (note this diff might be showing it backwards):
613c613
<                 ‘ -d ‘ + fh.name + ‘ -O > /dev/null 2>&1′)
>                 ‘ -d ‘ + fh.name + ‘ -O >& /dev/null’)

The error comes about that blkparse is run with ">& /dev/null" so that fd1 and fd2 go to /dev/null, but in bash its done differently (with bash3 and bash4 this method works "> /dev/null 2>&1")

TO FIX:
Open the file/program seekwatcher with vi (or your fav editor) and edit the code since its python you can do that:

# which seekwatcher
OUTPUT: /usr/local/bin/seekwatcher
# vi /usr/local/bin/seekwatcher

OR can open like this:

# vi `which seekwatcher`

Then type "/fh.name" without quotes to find the line that says this:
‘ -d ‘ + fh.name + ‘ -O >& /dev/null’

And change it to say:
' -d ' + fh.name + ' -O > /dev/null 2>&1'

It just the last part of the line that you need to chang after the  -O >
NOTE: do not mess up with the trailing spaces on the left (leave the tabs/spaces on the left alone) or else the program wont work. Im not showing the spaces in my fix instructions, but the diff file output above shows them (you will see them in the code)

Now save and exit and it should work

IF GET OTHER ERROR:
===================

Use google

My example script: makes a trace of reading every file in a dir and makes a seekwatcher picture and movie
###########################################################################################################

Read the comments at the top

#!/bin/bash
#
# run this inside a directory where you want your traces and pic/vids to save
# (prefereablly outside of the readlocation <- the place that will be read for the trace file)
#
# My example script: makes a trace of reading every file in a dir and makes a seekwatcher picture and movie
# You can just copy paste this script into a shell to run it, or just make a script file and run that. Just change the first variables:
#
# make sure you have this debugfs mount setup on your system - check like this;
# mount | egrep debugfs
# OUTPUT SHOULD BE: debugfs on /sys/kernel/debug type debugfs (rw,relatime)
# IF THIS OUTPUT IS MISSING
# mkdir /sys/kernel/debug
# mount -t debugfs debugfs /sys/kernel/debug
#
# If you run this script as a script: then unhash the below #### hash marks you see (its a check to see if blktrace started)
# if blktrace didnt start then exit out of the script as the rest is useless (the troubleshoot why blktrace didnt start)
#
# If you run this script as a copy paste: then DONT unhash the below #### or else if blktrace didnt start, the exit 1 command
# will exit your shell (which is just annoying - but then troubleshoot why blktrace didnt start)
#
# about picking device:
# blktrace needs to be specified the disks/device with -d, I see examples pointing to the /dev/sda instead of the working partition.
# In my example below I point to the partition ex: /dev/sda5, i still get results.
# Either way it should work. blktrace is on a higher layer as i understand from the reading.
#
# change these 4 variables to meet your needs DESCRIPTION,DEVICE,SAVE and READ LOCATION
DESCRIPTION=opt-deleteme-dd-sda5
DEVICE=/dev/sda5
READLOCATION=/opt/
SAVELOCATION=/root/traces/deleteme # ***SAVE DIR SHOULD EXIST*** -/dev/null works too - saving to same disk i want to see read and write
## NOTE: best not to trace from the same DEVICE and save to same location SAVELOCATION, in my case I did (so you should see extra writes...
## when it saves the trace to the device)
DATE=`date +%Y-%m-%d_%H-%M-%S`
NAMEFILE=${DESCRIPTION}_${DATE}
echo "FILENAME WILL HAVE: $NAMEFILE"
sleep 1
##
echo "STARTING TRACE ON $DEVICE"
##
blktrace -d ${DEVICE} -o ${NAMEFILE} &
PID123=$!
echo "PID IS:" $PID123
######
######
###### CHECK THAT BLKPARSE IS RUNNING IF NOT EXIT <-leave this commented out with #s marks
###### unhash below 4 hashmarks #### only if running as script (not as copy paste, if run below hashed out code..
###### with copy paste the exit 1 command will quit out of your shell which is annoying)
######
#### PIDS=`ps cax | grep ${PID123} | grep -o '^[ ]*[0-9]*'`
#### if [ -z "$PIDS" ]; then
#### echo "Process not running." 1>&2
#### exit 1
#### fi
sleep 2
echo "RUNNING DD COMMAND SOON!"
sleep 2
echo "RUNNING IN 2 SECOND!"
sleep 2
##
##
## if you want to use my script for your own traces, this is where you would change what program you want to run for the trace
## make sure to not use any backgrounded apps here, or else the blktrace would be killed immediatedly after by the kill -15
## why would it be killed? because backgrounded apps give way for the rest of the script to run. without backgrounding your program
## the script would wait for your program to finish before continuing to the kill command
## (i would also remove the annoying sleeps :) )
##
time (for i in `find ${READLOCATION} -type f`;
do
echo READING $i
dd if=$i of=${SAVELOCATION} bs=64k
done;)
##
##
##
##
sleep 1
echo "DONE"
echo "EXITING BLKTRACE NOW"
sleep 1
kill -15 $PID123
# Note: regular kill $PID123 would work as well as they use the same SIGTERM signal (its equivalent of CONTROL-C - a proper close).
# Note: kill -9 would be too powerful so dont use it
##
TRACEFILE="${NAMEFILE}*"
##
sleep 1
echo "HERE IS THE TRACE FILE:"
ls -lisah ${TRACEFILE}
sleep 1
##
echo "BLKPARSE SAYS:"
blkparse -i ${TRACEFILE}
sleep 2
echo "GOING TO RUN SEEKWATCHER TO MAKE AN IMAGE:"
sleep 1
seekwatcher -t ${TRACEFILE} -o ${NAMEFILE}.png
sleep 1
echo "MADE A PICTURE FILE WITH SEEKWATCHER:"
ls -lisah ${NAMEFILE}.png
sleep 1
echo "GOING TO RUN SEEKWATCHER TO MAKE A MOVIE:"
sleep 1
seekwatcher -t ${TRACEFILE} -o ${NAMEFILE}.mpg --movie
sleep 1
echo "MADE A MOVIE FILE WITH SEEKWATCHER:"
ls -lisah ${NAMEFILE}.mpg
sleep 1
echo "THE END"


ATTACHEMENTS
===============

some of the traces that i did on my ext3 system with the above script
note: deleteme means i saved to disk and i called the file deleteme so i didnt forget to delete it
also the edit of seekwatcher is below the one containing the fix for fd error mentioned above

FINAL WORDS
=============

Seekwatcher is great and seems to be the only tool available to visualize IO avail to the public without writing your own program to parse thru traces.

IM LOOKING FOR MORE VISUALIZATION PROGRAMS BUT COULDNT FIND ANY SO I POSTED ON SERVERFAULT:

SelectionFile type iconFile nameDescriptionSizeRevisionTimeUser
č

Download
  3628k v. 1 Jan 12, 2014, 4:29 AM koss boss
ą

View Download
  99k v. 1 Jan 12, 2014, 4:29 AM koss boss
č

Download
  1354k v. 1 Jan 12, 2014, 2:19 AM koss boss
ą

View Download
  82k v. 1 Jan 12, 2014, 2:19 AM koss boss
č

Download
  1750k v. 1 Jan 12, 2014, 2:19 AM koss boss
ą

View Download
  96k v. 1 Jan 12, 2014, 2:19 AM koss boss
ą

View Download
  82k v. 1 Jan 12, 2014, 2:19 AM koss boss
č

Download
  2418k v. 1 Jan 12, 2014, 4:29 AM koss boss
ą

View Download
  105k v. 1 Jan 12, 2014, 4:29 AM koss boss
č

Download
  2874k v. 1 Jan 12, 2014, 2:19 AM koss boss
ą

View Download
  99k v. 1 Jan 12, 2014, 2:19 AM koss boss
č

Download
  3136k v. 1 Jan 12, 2014, 2:19 AM koss boss
ą

View Download
  101k v. 1 Jan 12, 2014, 2:20 AM koss boss
č

Download
  3868k v. 1 Jan 12, 2014, 4:29 AM koss boss
ą

View Download
  100k v. 1 Jan 12, 2014, 4:29 AM koss boss
ċ

Download
  43k v. 1 Jan 12, 2014, 2:20 AM koss boss
Comments