title: OPOSS zpool iostat: I/O Performance Monitoring
agents: linux
catalog: os/storage
license: GPL
distribution: check_mk
description:
 This check monitors ZFS storage pool I/O performance and statistics using the
 {zpool iostat} command. It tracks comprehensive metrics including I/O operations,
 throughput, latency, storage utilization, and queue depths.

 The check provides detailed monitoring of ZFS pool performance characteristics:
 - I/O operations per second (read/write)
 - Data throughput (read/write bandwidth)
 - I/O latency and wait times
 - Storage pool capacity utilization
 - Queue depths and active operations
 - Synchronous and asynchronous I/O statistics
 - Scrub and trim operation statistics

 The plugin uses the {zpool iostat -Hylpq} command to collect detailed statistics
 with parseable output, latency information, and queue statistics. Data is
 collected over a configurable sampling interval to provide meaningful averages.

 This check requires ZFS to be installed and at least one zpool to be configured
 on the monitored system. The check automatically discovers all available zpools
 and creates individual services for each pool.

 The agent plugin is implemented in Python 3 with comprehensive error handling,
 configurable timeouts, and support for Agent Bakery deployment. Configuration
 options include sampling intervals, command timeouts, and detailed statistics
 collection.

 Storage utilization monitoring includes configurable thresholds for warning
 and critical states based on pool capacity usage. I/O performance monitoring
 supports optional thresholds for operations per second, throughput rates,
 and latency measurements.

 Advanced features include monitoring of ZFS-specific queue statistics:
 - Sync and async I/O queue depths
 - Per-operation type latencies
 - Scrub and trim operation queues
 - Active vs pending operation counts

item:
 The ZFS pool name (e.g. {tank}, {data}, {rpool})

discovery:
 One service is created for each ZFS pool that is available and accessible
 through the {zpool iostat} command. Services are named "ZPool I/O <pool_name>"
 (e.g. "ZPool I/O tank").

 Discovery requires the zpool command to be available and at least one pool
 to be imported and accessible. Pools that are not imported or are in a
 degraded state may not be discovered depending on their accessibility.

perfdata:
 The check provides comprehensive performance data including:

 All metric names carry the {oposs_zpool_} prefix. This avoids collisions
 with the built-in CheckMK metrics of the same base name. Wait times are
 stored in seconds and carry an {_s} suffix, while the corresponding
 thresholds are configured in milliseconds.

 Basic I/O metrics:
 - {oposs_zpool_read_ops} - Read operations per second
 - {oposs_zpool_write_ops} - Write operations per second
 - {oposs_zpool_read_throughput} - Read throughput in bytes per second
 - {oposs_zpool_write_throughput} - Write throughput in bytes per second

 Storage capacity:
 - {oposs_zpool_allocated} - Allocated storage space in bytes
 - {oposs_zpool_free} - Free storage space in bytes
 - {oposs_zpool_storage_used_percent} - Storage utilization percentage

 Latency metrics:
 - {oposs_zpool_read_wait_s} - Average read I/O wait time in seconds
 - {oposs_zpool_write_wait_s} - Average write I/O wait time in seconds
 - {oposs_zpool_disk_read_wait_s} - Disk-level read wait time
 - {oposs_zpool_disk_write_wait_s} - Disk-level write wait time
 - {oposs_zpool_disk_wait_max_s} - Higher of the two disk wait times, only
   emitted when disk wait levels are configured

 Queue statistics (when available):
 - {oposs_zpool_syncq_read_wait_s} - Synchronous read queue wait time
 - {oposs_zpool_syncq_write_wait_s} - Synchronous write queue wait time
 - {oposs_zpool_asyncq_read_wait_s} - Asynchronous read queue wait time
 - {oposs_zpool_asyncq_write_wait_s} - Asynchronous write queue wait time
 - {oposs_zpool_scrub_wait_s} - Scrub operation wait time
 - {oposs_zpool_trim_wait_s} - Trim operation wait time
 - {oposs_zpool_rebuild_wait_s} - Rebuild operation wait time

 Queue depths:
 - {oposs_zpool_syncq_read_pend} - Pending synchronous read operations
 - {oposs_zpool_syncq_read_activ} - Active synchronous read operations
 - {oposs_zpool_syncq_write_pend} - Pending synchronous write operations
 - {oposs_zpool_syncq_write_activ} - Active synchronous write operations
 - {oposs_zpool_asyncq_read_pend} - Pending asynchronous read operations
 - {oposs_zpool_asyncq_read_activ} - Active asynchronous read operations
 - {oposs_zpool_asyncq_write_pend} - Pending asynchronous write operations
 - {oposs_zpool_asyncq_write_activ} - Active asynchronous write operations

 Special operation queues:
 - {oposs_zpool_scrubq_read_pend} - Pending scrub read operations
 - {oposs_zpool_scrubq_read_activ} - Active scrub read operations
 - {oposs_zpool_trimq_write_pend} - Pending trim write operations
 - {oposs_zpool_trimq_write_activ} - Active trim write operations
 - {oposs_zpool_rebuildq_write_pend} - Pending rebuild write operations
 - {oposs_zpool_rebuildq_write_activ} - Active rebuild write operations

parameters:
 Storage utilization levels:
 - {storage_levels} - Warning/Critical thresholds for storage utilization percentage

 I/O operation thresholds:
 - {read_ops_levels} - Warning/Critical thresholds for read operations per second
 - {write_ops_levels} - Warning/Critical thresholds for write operations per second

 Latency thresholds:
 - {read_wait_levels} - Warning/Critical thresholds for read wait time in milliseconds
 - {write_wait_levels} - Warning/Critical thresholds for write wait time in milliseconds

 Throughput monitoring:
 - {read_throughput_levels} - Warning/Critical thresholds for read throughput in bytes/second
 - {write_throughput_levels} - Warning/Critical thresholds for write throughput in bytes/second

 Advanced latency monitoring:
 - {disk_wait_levels} - Warning/Critical thresholds for disk-level I/O wait times

 Queue wait time monitoring:
 - {syncq_read_wait_levels} - Thresholds for synchronous read queue wait times
 - {syncq_write_wait_levels} - Thresholds for synchronous write queue wait times
 - {asyncq_read_wait_levels} - Thresholds for asynchronous read queue wait times
 - {asyncq_write_wait_levels} - Thresholds for asynchronous write queue wait times
 - {scrub_wait_levels} - Thresholds for scrub operation wait times
 - {trim_wait_levels} - Thresholds for trim operation wait times
 - {rebuild_wait_levels} - Thresholds for rebuild operation wait times

 Queue depth monitoring:
 - {syncq_read_pend_levels} - Thresholds for pending synchronous read operations
 - {syncq_read_activ_levels} - Thresholds for active synchronous read operations
 - {syncq_write_pend_levels} - Thresholds for pending synchronous write operations
 - {syncq_write_activ_levels} - Thresholds for active synchronous write operations
 - {asyncq_read_pend_levels} - Thresholds for pending asynchronous read operations
 - {asyncq_read_activ_levels} - Thresholds for active asynchronous read operations
 - {asyncq_write_pend_levels} - Thresholds for pending asynchronous write operations
 - {asyncq_write_activ_levels} - Thresholds for active asynchronous write operations
 - {scrubq_read_pend_levels} - Thresholds for pending scrub read operations
 - {scrubq_read_activ_levels} - Thresholds for active scrub read operations
 - {trimq_write_pend_levels} - Thresholds for pending trim write operations
 - {trimq_write_activ_levels} - Thresholds for active trim write operations
 - {rebuildq_write_pend_levels} - Thresholds for pending rebuild write operations
 - {rebuildq_write_activ_levels} - Thresholds for active rebuild write operations

cluster:
 In cluster environments, the check can be configured to monitor ZFS pools
 across multiple nodes. Each node reports its local pool statistics, and
 cluster-wide aggregation can be performed for shared storage configurations.

 For clustered ZFS setups, ensure that pool access is properly configured
 and that the monitoring does not interfere with cluster operations.

examples:
 Check a pool with default settings:
 Pool "tank" shows normal I/O activity with 150 read ops/s, 75 write ops/s,
 and 65% storage utilization.

 Check with custom thresholds:
 Pool "data" configured with storage thresholds at 85%/95% and I/O operation
 thresholds at 500/1000 ops/s for reads and 250/500 ops/s for writes.

 High-performance monitoring:
 Pool "nvme-pool" with detailed queue statistics enabled, showing sync/async
 I/O patterns and queue depths for performance optimization.

 Error conditions:
 - Pool not accessible: Service reports UNKNOWN state
 - ZFS command timeout: Service reports UNKNOWN with timeout message
 - Parse errors: Service reports UNKNOWN with parsing error details
 - High latency: Service reports WARNING/CRITICAL based on configured thresholds
 - Storage near capacity: Service reports WARNING/CRITICAL based on utilization levels