gwcli(8) | Ceph iSCSI Gateway Tools | gwcli(8) |
gwcli - manage iscsi gateway configuration from the command line
gwcli is a configuration shell interface used for viewing, editing and saving the configuration of a ceph/iSCSI gateway environment. It enables the administrator to define rbd devices, map them across gateway nodes and export them to various clients over iSCSI. In addition to managing the iSCSI related elements of the configuration, the shell provides an overview of the ceph cluster, describing the available pools and the capacity they provide. Since rbd images are thin provisioned, the capacity information also indicates the capacity over-commit of the pools, enabling the admin to make more informed choices when allocating new rbd images.
iSCSI services are implemented by the kernel's LIO target subsystem layer, with iSCSI settings enforced by the rbd-target-gw daemon. The targetcli command may still be used to view lower level detail of the LIO environment, but all changes must be made using the gwcli.
The gwcli shell is similar to the targetcli interface, and is also based on 'configshell'. The layout of the UI is a tree format, and is navigated in much the same way as a filesystem.
gwcli [-d | --debug]
The -d option provides additional verbosity within the shell
gwcli [cmd]
Invoke gwcli as root to enter the interactive shell, or supply a command to execute without entering the shell. Within the shell, us ls to list nodes beneath the current path. Moving around the tree is done using the cd command, or by simply entering the 'path' of the new location/node directly. Use help <cmd> for specific help information. The shell provides tab completion for commands and command arguments.
Configuration state is persisted within a rados object stored in the 'rbd' pool. gwcli orchestrates changes across all iscsi gateways via the rbd-target-api service running on each gateway. Once the change to the local LIO subsystem is complete, the change is committed to the rados configuration object. Although 'targetcli' is available, it can only really provide a view of the local LIO configuration.
gwcli interacts with an API service provided by each iSCSI gateway's rbd-target-api daemon. The API service is installed with the cli, and can be configured by updating the api_* related settings in '/etc/ceph/iscsi-gateway.cfg'.
Typically, the following options are regarded as site specific;
api_password = <password>
api_port = <port_number>
api_secure = <true or false>
NB. An example iscsi-gateway.cfg file is provided under /usr/share/doc/ceph-iscsi-config*
Access to the API is normally restricted to the IP's of the gateway nodes, but you may also define other IP addresses that should be granted access to the API by adding the following entry to the configuration file;
By default the API service is not running with TLS, so for a more secure environment ensure iscsi-gateway.cfg has "api_secure = true" defined. When using secure mode you will need to create the appropriate certificate and private key files, and place them in /etc/ceph as 'iscsi-gateway.crt' and 'iscsi-gateway.key' on each gateway node.
Once these files are inplace across the nodes, the rbd-target-api service can be started. Check that the API service is enabled and in the correct mode by looking at the output of 'systemctl status rbd-target-api'. You should see a message similar to
The example gwcli output below shows a small two-gateway configuration, supporting 2 iSCSI clients
$ sudo gwcli
/> ls
/> ls o- / ................................................................... [...]
o- clusters .................................................. [Clusters: 1]
| o- ceph ...................................................... [HEALTH_OK]
| o- pools .................................................... [Pools: 3]
| | o- ec ......................... [(2+1), Commit: 0b/40G (0%), Used: 0b]
| | o- iscsi ...................... [(x3), Commit: 0b/20G (0%), Used: 18b]
| | o- rbd ........................ [(x3), Commit: 8G/20G (40%), Used: 5K]
| o- topology .......................................... [OSDs: 3,MONs: 3]
o- disks .................................................... [8G, Disks: 5]
| o- rbd ........................................................ [rbd (8G)]
| o- disk_1 ............................................ [rbd/disk_1 (1G)]
| o- disk_2 ............................................ [rbd/disk_2 (2G)]
| o- disk_3 ............................................ [rbd/disk_3 (2G)]
| o- disk_4 ............................................ [rbd/disk_4 (1G)]
| o- disk_5 ............................................ [rbd/disk_5 (2G)]
o- iscsi-targets .............................................. [Targets: 1]
o- iqn.2003-01.com.redhat.iscsi-gw:ceph-gw ................. [Gateways: 2]
o- disks .................................................... [Disks: 5]
| o- rbd/disk_1 ....................................... [Owner: rh7-gw1]
| o- rbd/disk_5 ....................................... [Owner: rh7-gw2]
o- gateways ...................................... [Up: 2/2, Portals: 2]
| o- rh7-gw1 ..................................... [192.168.122.69 (UP)]
| o- rh7-gw2 .................................... [192.168.122.104 (UP)]
o- host-groups ............................................ [Groups : 1]
| o- group1 ....................................... [Hosts: 1, Disks: 1]
| o- iqn.1994-05.com.redhat:rh7-client ........................ [host]
| o- rbd/disk_5 ............................................... [disk]
o- hosts .................................................... [Hosts: 2]
o- iqn.1994-05.com.redhat:myhost1 ......... [Auth: None, Disks: 1(1G)]
| o- lun 0 .......................... [rbd/disk_1(1G), Owner: rh7-gw2]]
o- iqn.1994-05.com.redhat:rh7-client [LOGGED-IN, Auth: CHAP, Disks: 1(2G)]
o- lun 0 .......................... [rbd/disk_5(2G), Owner: rh7-gw2]]
Disks exported through the gateways use ALUA attributes to provide ActiveOptimised and ActiveNonOptimised access to the rbd images. Each disk is assigned a primary owner at creation/import time - shown above with the owner attribute.
In order to manage rbd images (disks) within the environment there are several commands that enable you to create, resize and delete rbd's from the ceph cluster. When an rbd image is created, it is registered with all gateways. Part of this registration process defines the gateway that will provide the active I/O path to the LUN (disk) for any/all clients. This means that the iscsi-target definition and the gateway hosts must be defined prior to any disks being created (added to the gateways). It's also important to note that for an rbd image to be compatible with the iSCSI environment, it must have specific image features enabled (exclusive_lock, layering). The easiest way to create new disks is using the /disks create command.
The iscsi-target provides the end-point name that clients will know the iSCSI 'cluster' as. The target IQN will be created across all gateways within the configuration. Once the target is defined, the iscsi-target sub-tree is populated with entries for gateways and hosts.
Gateways provide the access points for rbd images over iSCSI, so there should be a minimum of 2 defined to provide fault tolerance.
Host groups provide a more convenient way of managing multiple servers that must share the same disk masking configuration. For example in a RHV/oVirt or Vmware environment, each host needs access to the same LUNs. Host groups allow you to create a logical group which contains the hosts and the disks that each host in the group should have access to. Please note that sharing devices across hosts needs a cluster aware filesystem or equivalent locking to avoid data corruption.
The 'hosts' section defines the iSCSI client definitions (NodeACLs) that provide access to the rbd images. The CLI provides the ability to create and delete clients, define/update chap authentication and add and remove rbd images for the client.
It is the Administrators responsibility to ensure that any disk shared between clients uses a cluster-aware filesystem to prevent data corruption.
>cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw/gateways
>create ceph-gw-1 10.172.19.21
>cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw/hosts/fR
Written by Paul Cuzner (pcuzner@redhat.com)
Report bugs via <https://github.com/ceph/ceph-iscsi-cli/issues>
Ceph iSCSI Gateway Tools | 23 Jul 2017 |