External Snapshot of raw images

When external snapshot of raw image is taken, delta is taken into qcow2 files.

virsh # list
Id    Name                           State
—————————————————-
4     cbtool                         running
6     master                         running

virsh # snapshot-create-as master snap1-master “snap1” –diskspec vda,file=/home/snap1.qcow2   –disk-only –atomic

Domain snapshot snap2-master created

snapshots tree :

virsh # snapshot-list  master –tree
snap1-master

virsh # snapshot-create-as master snap2-master “snap2” –diskspec vda,file=/home/snap2.qcow2   –disk-only  –atomic
Domain snapshot snap2-master created
virsh # snapshot-list  master –tree
snap1-master
|
+- snap2-master

Image info:

qemu-img info  /home/snap2.qcow2
image: /home/snap2.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 196K
cluster_size: 65536
backing file: /home/snap1.qcow2
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false

How to Delete:

virsh # snapshot-list master
Name                 Creation Time             State
————————————————————
snap2-master         2016-01-07 03:38:10 -0500 disk-snapshot

virsh # snapshot-delete master snap2-master –metadata
Domain snapshot snap2-master deleted

 

 

 

Advertisements

Starting MongoDB on CentOS with NUMA disabled

Have been noticing this every time i run mongodb on centos/RHEL or any other.

Error:
Sun Dec 20 06:26:16.832 [initandlisten] ** WARNING: You are running on a NUMA machine.
Sun Dec 20 06:26:16.832 [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
Sun Dec 20 06:26:16.832 [initandlisten] ** numactl –interleave=all mongod [other options]
Sun Dec 20 06:26:16.832 [initandlisten]

TO avoid performance issues, its recommended to run mongodb with interleaving memory across all NUMA nodes.

Dint find any way to solve this.
Kill existing mongod and restart mongodb with below.

numactl –interleave=all runuser -s /bin/bash mongodb -c “/usr/bin/mongod –dbpath /var/lib/mongodb”

 

pbench: A Benchmarking and Performance Analysis Framework

  1. What is pbench?

Pbench is a harness that allows data collection from a variety of tools while running a benchmark.

Pbench has some built-in script that run some common benchmarks, but the data collection can be run separately as well

with a benchmark that is not built-in to pbench, or a pbench script can be written for the benchmark. Such contributions are more than welcome!

2.   Available tools:

      The configured default set of tools (what you would get by running pbench-register-tool-set) is:

  • sar, iostat, mpstat, pidstat, proc-vmstat, proc-interrupts, perf

In addition, there are tools that can be added to the default set with pbench-register-tool:

  • blktrace, cpuacct, dm-cache, docker, kvmstat, kvmtrace, lockstat, numastat, perf, porc-sched_debug, proc-vmstat, qemu-migrate, rabbit, strace, sysfs, systemtap, tcpdump, turbostat, virsh-migrate, vmstat

3. Available benchmark scripts

  • pbench_dbench
  • pbench_fio
  • pbench_linpack
  • pbench_migrate
  • pbench_tpcc
  • pbench_uperf
  • user-benchmark ( to run any job)

5. How to install?

http://distributed-system-analysis.github.io/pbench/doc/agent/installation.html

6. Repos:

https://copr.fedorainfracloud.org/coprs/ndokos/pbench/

7. User Guide

http://distributed-system-analysis.github.io/pbench/doc/agent/user-guide.html

8. source

https://github.com/distributed-system-analysis/pbench

9. Mailing list

https://groups.google.com/forum/#!forum/pbench

10) Example pbench_fio run from host on clients ( Could be remote machines or guests)

pbench config settings:

/opt/pbench-agent/bench-scripts/pbench_fio ( can customize ioengine, ..etc)

Ex:

block_sizes=”4,64,1024″
targets=”/tmp/fio”
runtime=30
ramptime=5
#iodepth=32
iodepth=1
#ioengine=”libaio”
ioengine=”sync”

Clear previously registered tools (if any)

source /opt/pbench-agent/profile
clear-results
kill-tools
clear-tools

#register tool set

register-tool-set ( which covers all default tools)

#If any other tools needed

register-tool –name=perf — –record-opts=”record -g –pid=14355 -e kvm:kvm_inj_virq -e kvm:kvm_entry -e kvm:kvm_exit”

register-tool –name kvmstat

register-tool –name tcpdump

register-tool –name blktrace  — –devices=/dev/nvme1n1p1

#run benchmark

pbench_fio –clients=<clients on which tests need to be done. For ex: VM’s>  –test-types=<test types>  –block-sizes=<different sizes with a comma> –targets=<disks> –samples=<num of samples>   –config=<config name>

pbench_fio –clients=$clients –test-types=write,read,randread,randwrite,randrw  –block-sizes=4,16,64,256,1024,4096,16384  –targets=/dev/vdc  –samples=5  –config=fio-config

Can add more disks if more jobs need to be run.

samples=n: Run 5 samples and pbench calclulate std deviation  and gives you closer match.

#kill tools and move results

move-results
kill-tools
clear-tools

Results:

 

      iteration summary tools       readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
                1-write-4KiB    201319.13[+/-2.3%]   805276.42[+/-2.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   201319.13[+/-2.3%]   805276.42[+/-2.3%]  2486540.80[+/-2.5%]
               2-write-16KiB    59416.95[+/-3.8%]   950669.97[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]    59416.95[+/-3.8%]   950669.97[+/-3.8%]  6065548.80[+/-2.7%]
               3-write-64KiB    13235.63[+/-13.2%]  847083.63[+/-13.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   13235.63[+/-13.2%]  847083.63[+/-13.2%]21400166.40[+/-20.8%]
              4-write-256KiB    3755.32[+/-3.8%]   961464.90[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]     3755.32[+/-3.8%]   961464.90[+/-3.8%] 79495168.00[+/-5.2%]
             5-write-1024KiB    988.33[+/-2.1%]  1012008.91[+/-2.1%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      988.33[+/-2.1%]  1012008.91[+/-2.1%]314354892.80[+/-5.4%]
             6-write-4096KiB    272.84[+/-1.7%]  1117408.43[+/-1.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      272.84[+/-1.7%]  1117408.43[+/-1.6%]1635133030.40[+/-2.7%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
                 7-read-4KiB     443081.35[+/-1.5%]  1772325.42[+/-1.5%]   443081.35[+/-1.5%]  1772325.42[+/-1.5%]  1569043.20[+/-2.5%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
                8-read-16KiB     145205.19[+/-0.4%]  2323284.26[+/-0.4%]   145205.19[+/-0.4%]  2323284.26[+/-0.4%] 5470771.20[+/-12.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
                9-read-64KiB     33796.18[+/-5.2%]  2162961.77[+/-5.2%]    33796.18[+/-5.2%]  2162961.77[+/-5.2%]21228390.40[+/-11.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
              10-read-256KiB     9030.00[+/-1.4%]  2311667.51[+/-1.4%]     9030.00[+/-1.4%]  2311667.51[+/-1.4%]103413760.00[+/-4.4%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
             11-read-1024KiB     2358.56[+/-1.3%]  2415179.90[+/-1.3%]     2358.56[+/-1.3%]  2415179.90[+/-1.3%]185453772.80[+/-3.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
             12-read-4096KiB     597.96[+/-2.0%]  2449138.14[+/-2.0%]      597.96[+/-2.0%]  2449138.14[+/-2.0%]746619699.20[+/-1.5%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
            13-randread-4KiB    461194.31[+/-0.9%]  1844777.40[+/-0.9%]   461194.31[+/-0.9%]  1844777.40[+/-0.9%]  1369676.80[+/-2.9%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
           14-randread-16KiB    143443.52[+/-0.1%]  2295097.46[+/-0.1%]   143443.52[+/-0.1%]  2295097.46[+/-0.1%]  5756672.00[+/-2.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
           15-randread-64KiB    30334.00[+/-11.2%] 1941378.17[+/-11.2%]   30334.00[+/-11.2%] 1941378.17[+/-11.2%]18773094.40[+/-14.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
          16-randread-256KiB    8990.84[+/-0.9%]  2301671.35[+/-0.9%]     8990.84[+/-0.9%]  2301671.35[+/-0.9%]100553113.60[+/-3.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
         17-randread-1024KiB    2301.50[+/-1.7%]  2356708.83[+/-1.7%]     2301.50[+/-1.7%]  2356708.83[+/-1.7%]184976998.40[+/-3.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]
         18-randread-4096KiB    587.94[+/-1.3%]  2408056.26[+/-1.3%]      587.94[+/-1.3%]  2408056.26[+/-1.3%]757801779.20[+/-1.3%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]

                   iteration     readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
           19-randwrite-4KiB   227753.59[+/-4.2%]   911014.29[+/-4.2%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   227753.59[+/-4.2%]   911014.29[+/-4.2%]  2133932.80[+/-5.6%]
          20-randwrite-16KiB   59156.56[+/-7.7%]   946503.20[+/-7.7%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]    59156.56[+/-7.7%]   946503.20[+/-7.7%] 6273779.20[+/-13.1%]
          21-randwrite-64KiB   13194.78[+/-11.4%]  844464.84[+/-11.4%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]   13194.78[+/-11.4%]  844464.84[+/-11.4%]22670233.60[+/-22.6%]
         22-randwrite-256KiB   3626.32[+/-9.8%]   928403.87[+/-9.8%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]     3626.32[+/-9.8%]   928403.87[+/-9.8%]89994444.80[+/-15.1%]
        23-randwrite-1024KiB   961.35[+/-9.6%]   984385.43[+/-9.6%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      961.35[+/-9.6%]   984385.43[+/-9.6%]315842150.40[+/-21.3%]
        24-randwrite-4096KiB   198.75[+/-8.9%]   814045.25[+/-8.9%]        0.00[+/-0.0%]        0.00[+/-0.0%]        0.00[+/-0.0%]      198.75[+/-8.9%]   814045.25[+/-8.9%]2429529497.60[+/-13.7%]

                   iteration       readwrite_IOPS            rw_kB_sec            read_IOPS          read_kB_sec     read_95th_lat_us           write_IOPS         write_kB_sec    write_95th_lat_us
              25-randrw-4KiB   209010.84[+/-3.6%]   836043.29[+/-3.6%]   103595.54[+/-3.5%]   414382.20[+/-3.5%]  2793664.00[+/-3.0%]   105415.30[+/-3.7%]   421661.09[+/-3.7%]  2113689.60[+/-4.4%]
             26-randrw-16KiB   66672.72[+/-6.8%]  1066765.22[+/-6.8%]    33757.08[+/-6.6%]   540113.54[+/-6.6%]  6673433.60[+/-8.8%]    32915.65[+/-6.9%]   526651.68[+/-6.9%] 6249600.00[+/-10.3%]
             27-randrw-64KiB   13629.35[+/-14.3%]  872278.02[+/-14.3%]    7258.89[+/-14.4%]  464570.25[+/-14.4%] 20632985.60[+/-9.7%]    6370.46[+/-14.1%]  407707.76[+/-14.1%] 21101363.20[+/-9.7%]
            28-randrw-256KiB   4991.29[+/-11.3%] 1277774.89[+/-11.3%]    2732.44[+/-10.0%]  699505.92[+/-10.0%]84772556.80[+/-19.5%]    2258.84[+/-12.7%]  578268.96[+/-12.7%]89707827.20[+/-23.2%]
           29-randrw-1024KiB   1215.91[+/-6.0%]  1245092.34[+/-6.0%]      606.65[+/-5.9%]   621245.43[+/-5.9%]326084608.00[+/-9.1%]      609.25[+/-6.1%]   623846.91[+/-6.1%]337001267.20[+/-10.2%]
           30-randrw-4096KiB   328.24[+/-7.1%]  1344195.53[+/-7.2%]     148.70[+/-14.2%]  608808.18[+/-14.2%]1185399603.20[+/-1.1%]      179.54[+/-1.4%]   735387.35[+/-1.4%]1534726963.20[+/-4.4%]

 

Summary, tools of each run provide respective debug info details.

For ex:

1-write-4KiB             clients     details      readwrite-IOPS    readwrite-kB/sec           read-IOPS         read-kB/sec    read-95th-lat-ms          write-IOPS        write-kB/sec   write-95th-lat-ms
                             all_clients                       202003.10           808011.98                0.00                0.00                0.00           202003.10           808011.98                4.88
                          virbr0-122-233           12391.71            49566.75                0.00                0.00                0.00            12391.71            49566.75                4.93
                           virbr0-122-26           12218.42            48873.65                0.00                0.00                0.00            12218.42            48873.65                4.96
                          virbr0-122-123           12835.02            51339.93                0.00                0.00                0.00            12835.02            51339.93                4.77
                           virbr0-122-84           12515.69            50062.74                0.00                0.00                0.00            12515.69            50062.74                4.90
                          virbr0-122-242           12689.89            50759.62                0.00                0.00                0.00            12689.89            50759.62                4.89
                          virbr0-122-184           12473.84            49895.40                0.00                0.00                0.00            12473.84            49895.40                4.93
                           virbr0-122-34           12413.58            49654.27                0.00                0.00                0.00            12413.58            49654.27                4.93
                          virbr0-122-170           12806.87            51227.44                0.00                0.00                0.00            12806.87            51227.44                4.82
                          virbr0-122-152           12745.22            50980.79                0.00                0.00                0.00            12745.22            50980.79                4.85
                          virbr0-122-236           12553.71            50214.89                0.00                0.00                0.00            12553.71            50214.89                4.95
                          virbr0-122-237           12840.07            51360.35                0.00                0.00                0.00            12840.07            51360.35                4.83
                          virbr0-122-186           12629.69            50518.86                0.00                0.00                0.00            12629.69            50518.86                4.91
                           virbr0-122-40           13056.13            52224.43                0.00                0.00                0.00            13056.13            52224.43                4.71
                          virbr0-122-223           12946.96            51787.85                0.00                0.00                0.00            12946.96            51787.85                4.75
                          virbr0-122-108           12421.49            49685.84                0.00                0.00                0.00            12421.49            49685.84                4.94
                           virbr0-122-90           12464.81            49859.18                0.00                0.00                0.00            12464.81            49859.18                4.94

Tools provide all graphical representation of every debug tool registered.

 

Thanks to Andrew, Peter, ndk, archit and other perf folks in Redhat who developed tool

 

IT monitoring 101: How to select the right tools for your organization ?

IT monitoring 101: How to select the right tools for your organization ?

By Pradipta