SPEC Cloud™ IaaS 2016

The SPEC Cloud IaaS 2016 benchmark is SPEC’s first benchmark suite to measure cloud performance. The benchmark suite’s use is targeted at cloud providers, cloud consumers, hardware vendors, virtualization software vendors, application software vendors, and academic researchers.

The SPEC Cloud IaaS 2016 Benchmark addresses the performance of infrastructure-as-a-service (IaaS) cloud platforms. IaaS cloud platforms can either be public or private.

The SPEC Cloud IaaS 2016 benchmark is available for purchase via the SPEC order form.

Austin Openstack summit talk

https://www.openstack.org/videos/video/dell-performance-measuring-with-spec-cloud

Other links

https://www.brighttalk.com/webcast/10773/204933/innovating-cloud-benchmarking-with-the-new-spec-cloud-benchmark

http://www.reuters.com/article/idUSnMKWyNDfRa+1da+MKW20160503

Advertisements

pbench: A Benchmarking and Performance Analysis Framework

  1. What is pbench?

Pbench is a harness that allows data collection from a variety of tools while running a benchmark.

Pbench has some built-in script that run some common benchmarks, but the data collection can be run separately as well

with a benchmark that is not built-in to pbench, or a pbench script can be written for the benchmark. Such contributions are more than welcome!

2.   Available tools:

      The configured default set of tools (what you would get by running pbench-register-tool-set) is:

  • sar, iostat, mpstat, pidstat, proc-vmstat, proc-interrupts, perf

In addition, there are tools that can be added to the default set with pbench-register-tool:

  • blktrace, cpuacct, dm-cache, docker, kvmstat, kvmtrace, lockstat, numastat, perf, porc-sched_debug, proc-vmstat, qemu-migrate, rabbit, strace, sysfs, systemtap, tcpdump, turbostat, virsh-migrate, vmstat

3. Available benchmark scripts

  • pbench_dbench
  • pbench_fio
  • pbench_linpack
  • pbench_migrate
  • pbench_tpcc
  • pbench_uperf
  • user-benchmark ( to run any job)

5. How to install?

http://distributed-system-analysis.github.io/pbench/doc/agent/installation.html

6. Repos:

https://copr.fedorainfracloud.org/coprs/ndokos/pbench/

7. User Guide

http://distributed-system-analysis.github.io/pbench/doc/agent/user-guide.html

8. source

https://github.com/distributed-system-analysis/pbench

9. Mailing list

https://groups.google.com/forum/#!forum/pbench

10) Example pbench_fio run from host on clients ( Could be remote machines or guests)

pbench config settings:

/opt/pbench-agent/bench-scripts/pbench_fio ( can customize ioengine, ..etc)

Ex:

block_sizes=”4,64,1024″
targets=”/tmp/fio”
runtime=30
ramptime=5
#iodepth=32
iodepth=1
#ioengine=”libaio”
ioengine=”sync”

Clear previously registered tools (if any)

source /opt/pbench-agent/profile
clear-results
kill-tools
clear-tools

#register tool set

register-tool-set ( which covers all default tools)

#If any other tools needed

register-tool –name=perf — –record-opts=”record -g –pid=14355 -e kvm:kvm_inj_virq -e kvm:kvm_entry -e kvm:kvm_exit”

register-tool –name kvmstat

register-tool –name tcpdump

register-tool –name blktrace  — –devices=/dev/nvme1n1p1

#run benchmark

pbench_fio –clients=<clients on which tests need to be done. For ex: VM’s>  –test-types=<test types>  –block-sizes=<different sizes with a comma> –targets=<disks> –samples=<num of samples>   –config=<config name>

pbench_fio –clients=$clients –test-types=write,read,randread,randwrite,randrw  –block-sizes=4,16,64,256,1024,4096,16384  –targets=/dev/vdc  –samples=5  –config=fio-config

Can add more disks if more jobs need to be run.

samples=n: Run 5 samples and pbench calclulate std deviation  and gives you closer match.

#kill tools and move results

move-results
kill-tools
clear-tools

Results:

 

      iteration summary tools       readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
                1-write-4KiB    201319.13[+/-2.3%]   805276.42[+/-2.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   201319.13[+/-2.3%]   805276.42[+/-2.3%]  2486540.80[+/-2.5%]
               2-write-16KiB    59416.95[+/-3.8%]   950669.97[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]    59416.95[+/-3.8%]   950669.97[+/-3.8%]  6065548.80[+/-2.7%]
               3-write-64KiB    13235.63[+/-13.2%]  847083.63[+/-13.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   13235.63[+/-13.2%]  847083.63[+/-13.2%]21400166.40[+/-20.8%]
              4-write-256KiB    3755.32[+/-3.8%]   961464.90[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]     3755.32[+/-3.8%]   961464.90[+/-3.8%] 79495168.00[+/-5.2%]
             5-write-1024KiB    988.33[+/-2.1%]  1012008.91[+/-2.1%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      988.33[+/-2.1%]  1012008.91[+/-2.1%]314354892.80[+/-5.4%]
             6-write-4096KiB    272.84[+/-1.7%]  1117408.43[+/-1.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      272.84[+/-1.7%]  1117408.43[+/-1.6%]1635133030.40[+/-2.7%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
                 7-read-4KiB     443081.35[+/-1.5%]  1772325.42[+/-1.5%]   443081.35[+/-1.5%]  1772325.42[+/-1.5%]  1569043.20[+/-2.5%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
                8-read-16KiB     145205.19[+/-0.4%]  2323284.26[+/-0.4%]   145205.19[+/-0.4%]  2323284.26[+/-0.4%] 5470771.20[+/-12.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
                9-read-64KiB     33796.18[+/-5.2%]  2162961.77[+/-5.2%]    33796.18[+/-5.2%]  2162961.77[+/-5.2%]21228390.40[+/-11.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
              10-read-256KiB     9030.00[+/-1.4%]  2311667.51[+/-1.4%]     9030.00[+/-1.4%]  2311667.51[+/-1.4%]103413760.00[+/-4.4%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
             11-read-1024KiB     2358.56[+/-1.3%]  2415179.90[+/-1.3%]     2358.56[+/-1.3%]  2415179.90[+/-1.3%]185453772.80[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
             12-read-4096KiB     597.96[+/-2.0%]  2449138.14[+/-2.0%]      597.96[+/-2.0%]  2449138.14[+/-2.0%]746619699.20[+/-1.5%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
            13-randread-4KiB    461194.31[+/-0.9%]  1844777.40[+/-0.9%]   461194.31[+/-0.9%]  1844777.40[+/-0.9%]  1369676.80[+/-2.9%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
           14-randread-16KiB    143443.52[+/-0.1%]  2295097.46[+/-0.1%]   143443.52[+/-0.1%]  2295097.46[+/-0.1%]  5756672.00[+/-2.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
           15-randread-64KiB    30334.00[+/-11.2%] 1941378.17[+/-11.2%]   30334.00[+/-11.2%] 1941378.17[+/-11.2%]18773094.40[+/-14.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
          16-randread-256KiB    8990.84[+/-0.9%]  2301671.35[+/-0.9%]     8990.84[+/-0.9%]  2301671.35[+/-0.9%]100553113.60[+/-3.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
         17-randread-1024KiB    2301.50[+/-1.7%]  2356708.83[+/-1.7%]     2301.50[+/-1.7%]  2356708.83[+/-1.7%]184976998.40[+/-3.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
         18-randread-4096KiB    587.94[+/-1.3%]  2408056.26[+/-1.3%]      587.94[+/-1.3%]  2408056.26[+/-1.3%]757801779.20[+/-1.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
           19-randwrite-4KiB   227753.59[+/-4.2%]   911014.29[+/-4.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   227753.59[+/-4.2%]   911014.29[+/-4.2%]  2133932.80[+/-5.6%]
          20-randwrite-16KiB   59156.56[+/-7.7%]   946503.20[+/-7.7%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]    59156.56[+/-7.7%]   946503.20[+/-7.7%] 6273779.20[+/-13.1%]
          21-randwrite-64KiB   13194.78[+/-11.4%]  844464.84[+/-11.4%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   13194.78[+/-11.4%]  844464.84[+/-11.4%]22670233.60[+/-22.6%]
         22-randwrite-256KiB   3626.32[+/-9.8%]   928403.87[+/-9.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]     3626.32[+/-9.8%]   928403.87[+/-9.8%]89994444.80[+/-15.1%]
        23-randwrite-1024KiB   961.35[+/-9.6%]   984385.43[+/-9.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      961.35[+/-9.6%]   984385.43[+/-9.6%]315842150.40[+/-21.3%]
        24-randwrite-4096KiB   198.75[+/-8.9%]   814045.25[+/-8.9%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      198.75[+/-8.9%]   814045.25[+/-8.9%]2429529497.60[+/-13.7%]

                   iteration       readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
              25-randrw-4KiB   209010.84[+/-3.6%]   836043.29[+/-3.6%]   103595.54[+/-3.5%]   414382.20[+/-3.5%]  2793664.00[+/-3.0%]   105415.30[+/-3.7%]   421661.09[+/-3.7%]  2113689.60[+/-4.4%]
             26-randrw-16KiB   66672.72[+/-6.8%]  1066765.22[+/-6.8%]    33757.08[+/-6.6%]   540113.54[+/-6.6%]  6673433.60[+/-8.8%]    32915.65[+/-6.9%]   526651.68[+/-6.9%] 6249600.00[+/-10.3%]
             27-randrw-64KiB   13629.35[+/-14.3%]  872278.02[+/-14.3%]    7258.89[+/-14.4%]  464570.25[+/-14.4%] 20632985.60[+/-9.7%]    6370.46[+/-14.1%]  407707.76[+/-14.1%] 21101363.20[+/-9.7%]
            28-randrw-256KiB   4991.29[+/-11.3%] 1277774.89[+/-11.3%]    2732.44[+/-10.0%]  699505.92[+/-10.0%]84772556.80[+/-19.5%]    2258.84[+/-12.7%]  578268.96[+/-12.7%]89707827.20[+/-23.2%]
           29-randrw-1024KiB   1215.91[+/-6.0%]  1245092.34[+/-6.0%]      606.65[+/-5.9%]   621245.43[+/-5.9%]326084608.00[+/-9.1%]      609.25[+/-6.1%]   623846.91[+/-6.1%]337001267.20[+/-10.2%]
           30-randrw-4096KiB   328.24[+/-7.1%]  1344195.53[+/-7.2%]     148.70[+/-14.2%]  608808.18[+/-14.2%]1185399603.20[+/-1.1%]      179.54[+/-1.4%]   735387.35[+/-1.4%]1534726963.20[+/-4.4%]

 

Summary, tools of each run provide respective debug info details.

For ex:

1-write-4KiB             clients     details      readwrite-IOPS    readwrite-kB/sec           read-IOPS         read-kB/sec    read-95th-lat-ms          write-IOPS        write-kB/sec   write-95th-lat-ms
                             all_clients                       202003.10           808011.98                0.00                0.00                0.00           202003.10           808011.98                4.88
                          virbr0-122-233           12391.71            49566.75                0.00                0.00                0.00            12391.71            49566.75                4.93
                           virbr0-122-26           12218.42            48873.65                0.00                0.00                0.00            12218.42            48873.65                4.96
                          virbr0-122-123           12835.02            51339.93                0.00                0.00                0.00            12835.02            51339.93                4.77
                           virbr0-122-84           12515.69            50062.74                0.00                0.00                0.00            12515.69            50062.74                4.90
                          virbr0-122-242           12689.89            50759.62                0.00                0.00                0.00            12689.89            50759.62                4.89
                          virbr0-122-184           12473.84            49895.40                0.00                0.00                0.00            12473.84            49895.40                4.93
                           virbr0-122-34           12413.58            49654.27                0.00                0.00                0.00            12413.58            49654.27                4.93
                          virbr0-122-170           12806.87            51227.44                0.00                0.00                0.00            12806.87            51227.44                4.82
                          virbr0-122-152           12745.22            50980.79                0.00                0.00                0.00            12745.22            50980.79                4.85
                          virbr0-122-236           12553.71            50214.89                0.00                0.00                0.00            12553.71            50214.89                4.95
                          virbr0-122-237           12840.07            51360.35                0.00                0.00                0.00            12840.07            51360.35                4.83
                          virbr0-122-186           12629.69            50518.86                0.00                0.00                0.00            12629.69            50518.86                4.91
                           virbr0-122-40           13056.13            52224.43                0.00                0.00                0.00            13056.13            52224.43                4.71
                          virbr0-122-223           12946.96            51787.85                0.00                0.00                0.00            12946.96            51787.85                4.75
                          virbr0-122-108           12421.49            49685.84                0.00                0.00                0.00            12421.49            49685.84                4.94
                           virbr0-122-90           12464.81            49859.18                0.00                0.00                0.00            12464.81            49859.18                4.94

Tools provide all graphical representation of every debug tool registered.

 

Thanks to Andrew, Peter, ndk, archit and other perf folks in Redhat who developed tool