Docker at insane scale on IBM Power Systems

https://developer.ibm.com/bluemix/2015/11/13/docker-insane-scale-on-ibm-power-systems/

Number of io requests for each io_submit

Async io use io_submit calls. aio=native is used for async io. To get number of io requests for each io_submit from KVM VM,
here you go. My seq write 4k run on SSD. While capturing IOPS, make sure to trace io_submit perf events using below. 
sys_enter_io_submit, sys_exit_io_submit are mandatory. Essentially each io_submit call followed by *_io_getevents
which are irrelevant to present topic. 

 -e syscalls:sys_enter_io_submit -e syscalls:sys_exit_io_submit -e syscalls:sys_enter_io_getevents -e syscalls:sys_exit_io_getevents

Get Iops for this run.

write-4KiBIOPS 76971.9

Get Number of io_submits from captured perf.data. Number of enter_io_submits are fine. Ofcourse same number of exits will be there

[root@perf io-submit-write-4k]# perf script | grep io_submit | grep enter | wc -l
493370
[root@perf io-submit-write-4k]#
Get timestamp of io_submit events. (first and last)
First: 
qemu-kvm  3693 [025]  1914.589390: syscalls:sys_enter_io_submit: ctx_id: 0x7f3f18a61000, nr: 0x000000d1, iocbpp:
                     697 io_submit (/usr/lib64/libaio.so.1.0.1)
                       8 [unknown] ([unknown])
                       0 [unknown] ([unknown])

Last: 

qemu-kvm  3693 [001]  1949.737723: syscalls:sys_enter_io_submit: ctx_id: 0x7f3f18a61000, nr: 0x000000d1, iocbpp: 0x7ffd4e50b7b0
                 697 io_submit (/usr/lib64/libaio.so.1.0.1)
        7f3f1c6c9250 [unknown] ([unknown])
                   0 [unknown] ([unknown])

Number of submits per sec.
Time stamp diff: 1949.737723- 1914.589390 = 35.15
Number of submits:  493370

Submits/sec = 493370/35.15 = 14036.13086771
 IOPS metric is requests/second. We got submits per second. 
Number of requests per submit
requets/submission = (requests/sec) / (submits/sec)
                   =  76971.9 / 14036.130
                   =  5.483840631

So for my 4k write run number of requests per each submit are 5.48

iostat analysis: time spent for each IO request

These are one of the 4K write results to disk vdb (lvm volume on SSD which is irrelevent for present discussion).

“The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them

Wait_Time-vdb-write=0.042727

Throughput :

Throughput-vdb-write=65.902727

“svctm – The average service time (in milliseconds) for I/O requests that were issued to the device.

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
vdb               0.00     0.00 49409.00    0.00   193.00     0.00     8.00     7.04    0.14    0.14    0.00   0.02 100.00

Disk Utilization

Utilization-vdb=74.195455

Frame size; 4K

0.0427 ms = 42.7 microseconds.   42.7 microseconds per request
Throughput is 65 MB/s so 65 * 1024 / 4

= 16640 requests/s

1000000 microseconds/s / 16640 requests/s = 60 ms/request

60 ms/request * 0.75% disk utilization = 45 microseconds/request

So Time spend on each IO request is 45 microseconds/request

Thanks to Stefan

 

 

Qcow2 image creation with preallocation options to boost IO performance

While i started evaluating KVM IO performance, i leveraged qcow2 preallocation capabilities which helped me to boot IO performance. Its always good to preallocate image for better performance. For ex, while attaching disks to instances in openstack.

Earlier if we want to preallocate qcow2 image we had to do manually using falloc as shown below.

qemu-img create -f qcow2 -o preallocation=metadata /tmp/test.qcow2 8G
fallocate -l 8591507456 /tmp/test.qcow2

Now we have these options very useful.

qemu-img create -f qcow2 /tmp/test.qcow2 -o preallocation=falloc 1G

preallocation=falloc
Formatting ‘/tmp/test.qcow2′, fmt=qcow2 size=1073741824 encryption=off cluster_size=65536 preallocation=’falloc’ lazy_refcounts=off refcount_bits=16

“falloc” mode preallocated space for image by calling posic_fallocate()

preallocation=full

qemu-img create -f qcow2 /tmp/test.qcow2 -o preallocation=full 1G
Formatting ‘/tmp/test.qcow2′, fmt=qcow2 size=1073741824 encryption=off cluster_size=65536 preallocation=’full’ lazy_refcounts=off refcount_bits=16

“full” mode preallocates space for image by writing zeros to underlying storage. Similing to dd

Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1232570

Thanks to my IO Guru Stefan Hajnoczi for clarifying some of the IO part internals.

Redhat Satelitte health check

What is satelitte server?

Satellite 6 is the next generation Linux systems management tool and is the combined product of many open source projects, including Foreman, Katello, Pulp, Candlepin, and Puppet. Since all of these projects bring different featuresand functionality, the performance of your Satellite 6 server is essential.

What is capsule server?

The Red Satellite Capsule Server is a Satellite component that provides federated services to discover, provision, and configure hosts outside of the primary Satellite server. A Satellite Capsule Server provides the following features:
  • Pulp Server/Content Node features, including:
    • Repository synchronization
    • Content delivery

Why Healthcheck ?

Once we start scaling up datacenter, so as capsule server and managed nodes, its essential to have satelitte healthcheck tool.   It make sure to check all basic services, network ports..etc checked.

Any other tools?

There might be other tools. I find this useful and i customized it for my satelitte scale work. This is not officially supported one.

Usage:

 

  1. git clone https://github.com/boogiespook/sat6_healthCheck.git
  2. cd sat6_healthCheck
  3.  chmod 655 sat6_healthCheck.sh
  4.  Update credentials. mkdir ~/.hammer
    chmod 600 ~/.hammer
    echo << EOF >> /root/.hammer/cli_config.yml
    :foreman:
    :host: ‘https://$(hostname -f)’
    :username: ‘admin’
    :password: ‘password’
    EOF”
  5.  ./sat6_healthCheck.sh

Example Output:

  1. #######################################
    Satellite 6 Health Check Report
    #######################################
  2. ####################
    ## Checking umask ##
    ####################
  3. #######################
    ## Connection Status ##
    #######################
  4. ############################
    Checking Disk Partitions
    ############################”
  5. ###########################
    Checking Firewall Rules
    ###########################”
  6. #######################################
    ## Checking Satellite Configuration  ##
    #######################################
  7. #######################
    Checking OS Services
    #######################”
  8. #######################################
    Checking Katello/Satellite Services
    ######################################