Performance Monitoring of Red Hat Satellite 6 using satellite-performance

Continuous time-series metric collection of Satellite  & all Capsules are essential while satellite running at scale.
This post is helps to configure & monitor metrics using satellite-performance
1) Tools:
  • Collectd – Daemon to collect System Performance Statistics
    • Collects CPU, Memory, Disk, Network, Per Process stats (Regex), Postgresql, mongodb, turbostat, qpid, foreman, DynFlow, Passenger, Puppet, Tomcat, collectd..etc
  • Graphite/Carbon
    • Carbon receives metrics, and flushes them to whisper database files
    • Graphite is webapp frontend to carbon
  • Grafana – Visualize metrics from multiple backends.
    • Dashboards saved in json and customized by Ansible during deployment

2) Architecture

monitoring

3) How do i configure performance?

Archit has come up with a nice blog for configuration

Description of metrics collected in satperf:
http://arcolife.github.io/blog/2016/10/05/monitoring-in-satperf-metrics-collection

4)  Example Graphs
4.1 )  Passenger Mem
foreman
4.2) Postgresql DB (candlepin & foreman)
db
4.3) Candlepin DB
candlepindb
4.4) Puppet Registrations
pupper_registrations
4.2) Dynflow Mem
dynflow
Thanks to Archit, Jhutar for providing inputs & help!
Advertisements

RedHat Satellite 6.2 Considerations for Large scale deployments

Red Hat Satellite is a complete system management product that allows system administrators to manage the full life cycle of Red Hat deployments across physical, virtual, and private clouds. Red Hat Satellite delivers system provisioning, configuration management, software management, and subscription management- all while maintaining high scalability and security.  Satellite 6.2 is third major release of the next generation Satellite with a raft of improvements that continue to narrow the gaps in functionality found in Satellite 5 in many critical areas of the product.  This Blog  provides basic guidelines and considerations for tuning Red Hat Satellite 6.2 & capsule for Large scale deployments

1) Increase open-files-limit for Apache with systemd on satellite & Capsule server

# cat /etc/systemd/system/httpd.service.d/limits.conf

[Service]

LimitNOFILE=1000000

# systemctl daemon-reload

# katello-service restart

2) Increase open-files-limit for Qpid with systemd on satellite & Capsule server

# cat /etc/systemd/system/qpidd.service.d/limits.conf

[Service]

LimitNOFILE=1000000

# systemctl daemon-reload

# katello-service restart

3) Increase postgresql shared_buffer

While registering content hosts at scale to Satellite server, shared_buffers needs to be set appropriately in postgresql.conf. Recommended: 256 MB

4) Increase postgresql max_connections

When registering content hosts at scale, it is recommended to increase max_connections setting (set to 100 by default) as per your needs and HW profile. For example, you might need to set the value to 200 when you are registering 200 content hosts in parallel.

5) Storage planning for qpid

When you use katello-agent extensively, plan storage capacity for /var/lib/qpidd in advance. Currently, in Satellite 6.2 /var/lib/qpidd requires 2MB disk space per a content hos.

6) Increase open-files-limit for Qpid Dispatch Router with systemd on satellite & Capsule server

# cat /etc/systemd/system/qdrouterd.service.d/limits.conf

[Service]

LimitNOFILE=1000000

# systemctl daemon-reload

# katello-service restart

 

Special Thanks to Jan Jutar & Archit Sarma for the help to get scale numbers.

 

 

 

 

 

 

 

Red Hat Satellite 6.2 is available

Red Hat Satellite is a system management solution that enables you to deploy, configure, and maintain your systems across physical, virtual, and cloud environments. Satellite provides provisioning, remote management and monitoring of multiple Red Hat Enterprise Linux deployments with a single, centralized tool. Red Hat Satellite Server synchronizes the content from Red Hat Customer Portal and other sources, and provides functionality including fine-grained life cycle management, user and group role-based access control, integrated subscription management, as well as advanced GUI, CLI, or API access.

Red Hat Satellite 6.2 is available now :  https://access.redhat.com/blogs/1169563/posts/2464761

Satellite 6.2 Feature Overview:

Remote Execution

Demo: https://www.redhat.com/en/about/videos/satellite-feature-overview-remote-execution

Details: https://access.redhat.com/articles/2464671

 Inter-Satellite Sync

Demo: https://www.redhat.com/en/about/videos/satellite-feature-intersatellite-sync

Details:

Importing Existing Hosts via the Bootstrap Script

demo: https://www.redhat.com/en/about/videos/satellite-feature-overview-bootstrap-script

Details: https://access.redhat.com/articles/2280691

PXE-Less Discovery

Demo: https://www.redhat.com/en/about/videos/satellite-feature-overview-pxeless-discovery

Details: https://access.redhat.com/articles/2464701

 

SPEC Cloud™ IaaS 2016

The SPEC Cloud IaaS 2016 benchmark is SPEC’s first benchmark suite to measure cloud performance. The benchmark suite’s use is targeted at cloud providers, cloud consumers, hardware vendors, virtualization software vendors, application software vendors, and academic researchers.

The SPEC Cloud IaaS 2016 Benchmark addresses the performance of infrastructure-as-a-service (IaaS) cloud platforms. IaaS cloud platforms can either be public or private.

The SPEC Cloud IaaS 2016 benchmark is available for purchase via the SPEC order form.

Austin Openstack summit talk

https://www.openstack.org/videos/video/dell-performance-measuring-with-spec-cloud

Other links

https://www.brighttalk.com/webcast/10773/204933/innovating-cloud-benchmarking-with-the-new-spec-cloud-benchmark

http://www.reuters.com/article/idUSnMKWyNDfRa+1da+MKW20160503

What’s the difference between two vendors selling the same open source project?

Ahh this is a great read. Chapter 3 could have been named “The Red Hat Way”. Thanks to Jeremy for sharing this

Open Source for Business People