pbench: A Benchmarking and Performance Analysis Framework

  1. What is pbench?

Pbench is a harness that allows data collection from a variety of tools while running a benchmark.

Pbench has some built-in script that run some common benchmarks, but the data collection can be run separately as well

with a benchmark that is not built-in to pbench, or a pbench script can be written for the benchmark. Such contributions are more than welcome!

2.   Available tools:

      The configured default set of tools (what you would get by running pbench-register-tool-set) is:

  • sar, iostat, mpstat, pidstat, proc-vmstat, proc-interrupts, perf

In addition, there are tools that can be added to the default set with pbench-register-tool:

  • blktrace, cpuacct, dm-cache, docker, kvmstat, kvmtrace, lockstat, numastat, perf, porc-sched_debug, proc-vmstat, qemu-migrate, rabbit, strace, sysfs, systemtap, tcpdump, turbostat, virsh-migrate, vmstat

3. Available benchmark scripts

  • pbench_dbench
  • pbench_fio
  • pbench_linpack
  • pbench_migrate
  • pbench_tpcc
  • pbench_uperf
  • user-benchmark ( to run any job)

5. How to install?

http://distributed-system-analysis.github.io/pbench/doc/agent/installation.html

6. Repos:

https://copr.fedorainfracloud.org/coprs/ndokos/pbench/

7. User Guide

http://distributed-system-analysis.github.io/pbench/doc/agent/user-guide.html

8. source

https://github.com/distributed-system-analysis/pbench

9. Mailing list

https://groups.google.com/forum/#!forum/pbench

10) Example pbench_fio run from host on clients ( Could be remote machines or guests)

pbench config settings:

/opt/pbench-agent/bench-scripts/pbench_fio ( can customize ioengine, ..etc)

Ex:

block_sizes=”4,64,1024″
targets=”/tmp/fio”
runtime=30
ramptime=5
#iodepth=32
iodepth=1
#ioengine=”libaio”
ioengine=”sync”

Clear previously registered tools (if any)

source /opt/pbench-agent/profile
clear-results
kill-tools
clear-tools

#register tool set

register-tool-set ( which covers all default tools)

#If any other tools needed

register-tool –name=perf — –record-opts=”record -g –pid=14355 -e kvm:kvm_inj_virq -e kvm:kvm_entry -e kvm:kvm_exit”

register-tool –name kvmstat

register-tool –name tcpdump

register-tool –name blktrace  — –devices=/dev/nvme1n1p1

#run benchmark

pbench_fio –clients=<clients on which tests need to be done. For ex: VM’s>  –test-types=<test types>  –block-sizes=<different sizes with a comma> –targets=<disks> –samples=<num of samples>   –config=<config name>

pbench_fio –clients=$clients –test-types=write,read,randread,randwrite,randrw  –block-sizes=4,16,64,256,1024,4096,16384  –targets=/dev/vdc  –samples=5  –config=fio-config

Can add more disks if more jobs need to be run.

samples=n: Run 5 samples and pbench calclulate std deviation  and gives you closer match.

#kill tools and move results

move-results
kill-tools
clear-tools

Results:

 

      iteration summary tools       readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
                1-write-4KiB    201319.13[+/-2.3%]   805276.42[+/-2.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   201319.13[+/-2.3%]   805276.42[+/-2.3%]  2486540.80[+/-2.5%]
               2-write-16KiB    59416.95[+/-3.8%]   950669.97[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]    59416.95[+/-3.8%]   950669.97[+/-3.8%]  6065548.80[+/-2.7%]
               3-write-64KiB    13235.63[+/-13.2%]  847083.63[+/-13.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   13235.63[+/-13.2%]  847083.63[+/-13.2%]21400166.40[+/-20.8%]
              4-write-256KiB    3755.32[+/-3.8%]   961464.90[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]     3755.32[+/-3.8%]   961464.90[+/-3.8%] 79495168.00[+/-5.2%]
             5-write-1024KiB    988.33[+/-2.1%]  1012008.91[+/-2.1%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      988.33[+/-2.1%]  1012008.91[+/-2.1%]314354892.80[+/-5.4%]
             6-write-4096KiB    272.84[+/-1.7%]  1117408.43[+/-1.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      272.84[+/-1.7%]  1117408.43[+/-1.6%]1635133030.40[+/-2.7%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
                 7-read-4KiB     443081.35[+/-1.5%]  1772325.42[+/-1.5%]   443081.35[+/-1.5%]  1772325.42[+/-1.5%]  1569043.20[+/-2.5%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
                8-read-16KiB     145205.19[+/-0.4%]  2323284.26[+/-0.4%]   145205.19[+/-0.4%]  2323284.26[+/-0.4%] 5470771.20[+/-12.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
                9-read-64KiB     33796.18[+/-5.2%]  2162961.77[+/-5.2%]    33796.18[+/-5.2%]  2162961.77[+/-5.2%]21228390.40[+/-11.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
              10-read-256KiB     9030.00[+/-1.4%]  2311667.51[+/-1.4%]     9030.00[+/-1.4%]  2311667.51[+/-1.4%]103413760.00[+/-4.4%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
             11-read-1024KiB     2358.56[+/-1.3%]  2415179.90[+/-1.3%]     2358.56[+/-1.3%]  2415179.90[+/-1.3%]185453772.80[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
             12-read-4096KiB     597.96[+/-2.0%]  2449138.14[+/-2.0%]      597.96[+/-2.0%]  2449138.14[+/-2.0%]746619699.20[+/-1.5%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
            13-randread-4KiB    461194.31[+/-0.9%]  1844777.40[+/-0.9%]   461194.31[+/-0.9%]  1844777.40[+/-0.9%]  1369676.80[+/-2.9%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
           14-randread-16KiB    143443.52[+/-0.1%]  2295097.46[+/-0.1%]   143443.52[+/-0.1%]  2295097.46[+/-0.1%]  5756672.00[+/-2.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
           15-randread-64KiB    30334.00[+/-11.2%] 1941378.17[+/-11.2%]   30334.00[+/-11.2%] 1941378.17[+/-11.2%]18773094.40[+/-14.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
          16-randread-256KiB    8990.84[+/-0.9%]  2301671.35[+/-0.9%]     8990.84[+/-0.9%]  2301671.35[+/-0.9%]100553113.60[+/-3.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
         17-randread-1024KiB    2301.50[+/-1.7%]  2356708.83[+/-1.7%]     2301.50[+/-1.7%]  2356708.83[+/-1.7%]184976998.40[+/-3.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
         18-randread-4096KiB    587.94[+/-1.3%]  2408056.26[+/-1.3%]      587.94[+/-1.3%]  2408056.26[+/-1.3%]757801779.20[+/-1.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
           19-randwrite-4KiB   227753.59[+/-4.2%]   911014.29[+/-4.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   227753.59[+/-4.2%]   911014.29[+/-4.2%]  2133932.80[+/-5.6%]
          20-randwrite-16KiB   59156.56[+/-7.7%]   946503.20[+/-7.7%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]    59156.56[+/-7.7%]   946503.20[+/-7.7%] 6273779.20[+/-13.1%]
          21-randwrite-64KiB   13194.78[+/-11.4%]  844464.84[+/-11.4%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   13194.78[+/-11.4%]  844464.84[+/-11.4%]22670233.60[+/-22.6%]
         22-randwrite-256KiB   3626.32[+/-9.8%]   928403.87[+/-9.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]     3626.32[+/-9.8%]   928403.87[+/-9.8%]89994444.80[+/-15.1%]
        23-randwrite-1024KiB   961.35[+/-9.6%]   984385.43[+/-9.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      961.35[+/-9.6%]   984385.43[+/-9.6%]315842150.40[+/-21.3%]
        24-randwrite-4096KiB   198.75[+/-8.9%]   814045.25[+/-8.9%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      198.75[+/-8.9%]   814045.25[+/-8.9%]2429529497.60[+/-13.7%]

                   iteration       readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
              25-randrw-4KiB   209010.84[+/-3.6%]   836043.29[+/-3.6%]   103595.54[+/-3.5%]   414382.20[+/-3.5%]  2793664.00[+/-3.0%]   105415.30[+/-3.7%]   421661.09[+/-3.7%]  2113689.60[+/-4.4%]
             26-randrw-16KiB   66672.72[+/-6.8%]  1066765.22[+/-6.8%]    33757.08[+/-6.6%]   540113.54[+/-6.6%]  6673433.60[+/-8.8%]    32915.65[+/-6.9%]   526651.68[+/-6.9%] 6249600.00[+/-10.3%]
             27-randrw-64KiB   13629.35[+/-14.3%]  872278.02[+/-14.3%]    7258.89[+/-14.4%]  464570.25[+/-14.4%] 20632985.60[+/-9.7%]    6370.46[+/-14.1%]  407707.76[+/-14.1%] 21101363.20[+/-9.7%]
            28-randrw-256KiB   4991.29[+/-11.3%] 1277774.89[+/-11.3%]    2732.44[+/-10.0%]  699505.92[+/-10.0%]84772556.80[+/-19.5%]    2258.84[+/-12.7%]  578268.96[+/-12.7%]89707827.20[+/-23.2%]
           29-randrw-1024KiB   1215.91[+/-6.0%]  1245092.34[+/-6.0%]      606.65[+/-5.9%]   621245.43[+/-5.9%]326084608.00[+/-9.1%]      609.25[+/-6.1%]   623846.91[+/-6.1%]337001267.20[+/-10.2%]
           30-randrw-4096KiB   328.24[+/-7.1%]  1344195.53[+/-7.2%]     148.70[+/-14.2%]  608808.18[+/-14.2%]1185399603.20[+/-1.1%]      179.54[+/-1.4%]   735387.35[+/-1.4%]1534726963.20[+/-4.4%]

 

Summary, tools of each run provide respective debug info details.

For ex:

1-write-4KiB             clients     details      readwrite-IOPS    readwrite-kB/sec           read-IOPS         read-kB/sec    read-95th-lat-ms          write-IOPS        write-kB/sec   write-95th-lat-ms
                             all_clients                       202003.10           808011.98                0.00                0.00                0.00           202003.10           808011.98                4.88
                          virbr0-122-233           12391.71            49566.75                0.00                0.00                0.00            12391.71            49566.75                4.93
                           virbr0-122-26           12218.42            48873.65                0.00                0.00                0.00            12218.42            48873.65                4.96
                          virbr0-122-123           12835.02            51339.93                0.00                0.00                0.00            12835.02            51339.93                4.77
                           virbr0-122-84           12515.69            50062.74                0.00                0.00                0.00            12515.69            50062.74                4.90
                          virbr0-122-242           12689.89            50759.62                0.00                0.00                0.00            12689.89            50759.62                4.89
                          virbr0-122-184           12473.84            49895.40                0.00                0.00                0.00            12473.84            49895.40                4.93
                           virbr0-122-34           12413.58            49654.27                0.00                0.00                0.00            12413.58            49654.27                4.93
                          virbr0-122-170           12806.87            51227.44                0.00                0.00                0.00            12806.87            51227.44                4.82
                          virbr0-122-152           12745.22            50980.79                0.00                0.00                0.00            12745.22            50980.79                4.85
                          virbr0-122-236           12553.71            50214.89                0.00                0.00                0.00            12553.71            50214.89                4.95
                          virbr0-122-237           12840.07            51360.35                0.00                0.00                0.00            12840.07            51360.35                4.83
                          virbr0-122-186           12629.69            50518.86                0.00                0.00                0.00            12629.69            50518.86                4.91
                           virbr0-122-40           13056.13            52224.43                0.00                0.00                0.00            13056.13            52224.43                4.71
                          virbr0-122-223           12946.96            51787.85                0.00                0.00                0.00            12946.96            51787.85                4.75
                          virbr0-122-108           12421.49            49685.84                0.00                0.00                0.00            12421.49            49685.84                4.94
                           virbr0-122-90           12464.81            49859.18                0.00                0.00                0.00            12464.81            49859.18                4.94

Tools provide all graphical representation of every debug tool registered.

 

Thanks to Andrew, Peter, ndk, archit and other perf folks in Redhat who developed tool

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s