-
Notifications
You must be signed in to change notification settings - Fork 59
/
gwcli.8
243 lines (233 loc) · 16.1 KB
/
gwcli.8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
.\" Manpage for gwcli
.\" Contact [email protected] to correct errors or typos.
.TH gwcli 8 "Ceph iSCSI Gateway Tools" "23 Jul 2017" "Ceph iSCSI Gateway Tools"
.SH NAME
\fBgwcli\fR \- manage iscsi gateway configuration from the command line
.SH DESCRIPTION
\fBgwcli\fR is a configuration shell interface used for viewing, editing and saving the configuration of a ceph/iSCSI gateway environment. It enables the administrator to define rbd devices, map them across gateway nodes and export them to various clients over iSCSI. In addition to managing the iSCSI related elements of the configuration, the shell provides an overview of the ceph cluster, describing the available pools and the capacity they provide. Since rbd images are thin provisioned, the capacity information also indicates the capacity over-commit of the pools, enabling the admin to make more informed choices when allocating new rbd images.
.PP
iSCSI services are implemented by the kernel's LIO target subsystem layer, with iSCSI settings enforced by the rbd-target-gw daemon. The targetcli command may still be used to view lower level detail of the LIO environment, but all changes \fBmust\fR be made using the gwcli.
.PP
The gwcli shell is similar to the targetcli interface, and is also based on 'configshell'. The layout of the UI is a tree format, and is navigated in much the same way as a filesystem.
.SH USAGE
\fBgwcli\fR [-d | --debug]
The -d option provides additional verbosity within the shell
\fBgwcli [cmd]\fR
Invoke gwcli as root to enter the interactive shell, or supply a command to execute without entering the shell. Within the shell, us \fBls\fR to list nodes beneath the current path. Moving around the tree is done using the \fBcd\fR command, or by simply entering the 'path' of the new location/node directly. Use \fBhelp <cmd>\fR for specific help information. The shell provides tab completion for commands and command arguments.
.PP
Configuration state is persisted within a rados object stored in the 'rbd' pool. gwcli orchestrates changes across all iscsi gateways via the rbd-target-api service running on each gateway. Once the change to the local LIO subsystem is complete, the change is committed to the rados configuration object. Although 'targetcli' is available, it can only really provide a view of the local LIO configuration.
.SH QUICKSTART
gwcli interacts with an API service provided by each iSCSI gateway's rbd-target-api daemon. The API service is installed with the cli, and can be configured by updating the api_* related settings in '/etc/ceph/iscsi-gateway.cfg'.
.PP
Typically, the following options are regarded as site specific;
.PP
.PD 0.4
.RS 3
\fBapi_user = <user_name>\fR
.PP
\fBapi_password = <password>\fR
.PP
\fBapi_port = <port_number>\fR
.PP
\fBapi_secure = <true or false>\fR
.RE
.PD 1
.PP
\fBNB.\fR An example iscsi-gateway.cfg file is provided under /usr/share/doc/ceph-iscsi-config*
.PP
Access to the API is normally restricted to the IP's of the gateway nodes, but you may also define other IP addresses that should be granted access to the API by adding the following entry to the configuration file;
.PP
.RS 3
\fBtrusted_ip_list = <ip_address,ip_address...>\fR
.RE
.PP
By default the API service is not running with TLS, so for a more secure environment ensure iscsi-gateway.cfg has "api_secure = true" defined. When using secure mode you will need to create the appropriate certificate and private key files, and place them in /etc/ceph as 'iscsi-gateway.crt' and 'iscsi-gateway.key' on \fBeach\fR gateway node.
.PP
Once these files are inplace across the nodes, the rbd-target-api service can be started. Check that the API service is enabled and in the correct mode by looking at the output of 'systemctl status rbd-target-api'. You should see a message similar to
.PP
.RS 3
\fB* Running on https://0.0.0.0:5000/\fR.
.RE
.PP
The example gwcli output below shows a small two-gateway configuration, supporting 2 iSCSI clients
.PP
.PD 0.4
$ sudo gwcli
/> ls
.PP
.nf
/> ls
o- / ................................................................... [...]
o- clusters .................................................. [Clusters: 1]
| o- ceph ...................................................... [HEALTH_OK]
| o- pools .................................................... [Pools: 3]
| | o- ec ......................... [(2+1), Commit: 0b/40G (0%), Used: 0b]
| | o- iscsi ...................... [(x3), Commit: 0b/20G (0%), Used: 18b]
| | o- rbd ........................ [(x3), Commit: 8G/20G (40%), Used: 5K]
| o- topology .......................................... [OSDs: 3,MONs: 3]
o- disks .................................................... [8G, Disks: 5]
| o- rbd ........................................................ [rbd (8G)]
| o- disk_1 ............................................ [rbd/disk_1 (1G)]
| o- disk_2 ............................................ [rbd/disk_2 (2G)]
| o- disk_3 ............................................ [rbd/disk_3 (2G)]
| o- disk_4 ............................................ [rbd/disk_4 (1G)]
| o- disk_5 ............................................ [rbd/disk_5 (2G)]
o- iscsi-targets .............................................. [Targets: 1]
o- iqn.2003-01.com.redhat.iscsi-gw:ceph-gw ................. [Gateways: 2]
o- disks .................................................... [Disks: 5]
| o- rbd/disk_1 ....................................... [Owner: rh7-gw1]
| o- rbd/disk_5 ....................................... [Owner: rh7-gw2]
o- gateways ...................................... [Up: 2/2, Portals: 2]
| o- rh7-gw1 ..................................... [192.168.122.69 (UP)]
| o- rh7-gw2 .................................... [192.168.122.104 (UP)]
o- host-groups ............................................ [Groups : 1]
| o- group1 ....................................... [Hosts: 1, Disks: 1]
| o- iqn.1994-05.com.redhat:rh7-client ........................ [host]
| o- rbd/disk_5 ............................................... [disk]
o- hosts .................................................... [Hosts: 2]
o- iqn.1994-05.com.redhat:myhost1 ......... [Auth: None, Disks: 1(1G)]
| o- lun 0 .......................... [rbd/disk_1(1G), \fBOwner: rh7-gw2]\fR]
o- iqn.1994-05.com.redhat:rh7-client [LOGGED-IN, Auth: CHAP, Disks: 1(2G)]
o- lun 0 .......................... [rbd/disk_5(2G), \fBOwner: rh7-gw2]\fR]
.fi
.PD 1
.PP
Disks exported through the gateways use ALUA attributes to provide ActiveOptimised and ActiveNonOptimised access to the rbd images. Each disk is assigned a primary owner at creation/import time - shown above with the \fBowner\fR attribute.
.SH DISKS
In order to manage rbd images (disks) within the environment there are several commands that enable you to create, resize and delete rbd's from the ceph cluster. When an rbd image is created, it is registered with all gateways. Part of this registration process defines the gateway that will provide the active I/O path to the LUN (disk) for any/all clients. This means that the iscsi-target definition \fIand\fR the gateway hosts must be defined prior to any disks being created (added to the gateways). It's also important to note that for an rbd image to be compatible with the iSCSI environment, it must have specific image features enabled (exclusive_lock, layering). The easiest way to create new disks is using the \fB/disks create\fR command.
.PP
.TP
\fB/disks/ create pool=<pool> image=<image_name> size=<N>G\fR
Using the create command ensure the image features are applied correctly. You can also choose to create your rbd images by some other means, in which case the 'create' command will effectively 'import' the rbd into the configuration leaving any data already on the device, intact.
.PP
.TP
.PD 0
\fB/disks/<disk_name>/ resize <N>g\fR
.TP
\fB/disks resize <disk_name> <new_size>\fR
Use the resize command to increase the capacity of a specific rbd image.
.PD 1
.PP
.TP
\fB/disks/ delete <disk_name>\fR
The delete command allows you to remove the rbd from the LIO and ceph cluster. Prior to the delete being actioned the current configuration is checked to ensure that the requested rbd image is not masked to any iSCSI client. Once this check is successful, the rbd image will be purged from the LIO environment on each gateway and deleted from the ceph cluster.
.SH ISCSI-TARGETS
The iscsi-target provides the end-point name that clients will know the iSCSI 'cluster' as. The target IQN will be created across all gateways within the configuration. Once the target is defined, the iscsi-target sub-tree is populated with entries for \fBgateways\fR and \fBhosts\fR.
.PP
.TP
\fB/iscsi-targets/ create <valid_IQN>\fR
The IQN provided will be validated and defined to the configuration object. Adding gateway nodes will then pick up the configuration's IQN and apply it to their local LIO instance.
.TP
\fB/iscsi-targets/ clearconfig confirm=true\fR
The clearconfig command provides the ability to return each of the gateways to their undefined state. However, since this is a disruptive command you must remove the clients and disks first, before issuing a clearconfig.
.SH GATEWAYS
Gateways provide the access points for rbd images over iSCSI, so there should be a minimum of 2 defined to provide fault tolerance.
.PP
.TP
\fB/iscsi-targets/<iqn>/ create <node_name> <portal_ip_address>
Gateways are defined by a node name (preferably a shortname, but it must resolve), and an IPv4/IPv6 address that the iSCSI 'service' will be bound to (i.e. the iSCSI portal IP address). When adding a gateway, the candidate machine will be checked to ensure the relevant files and daemons are in place.
.SH HOST-GROUPS
Host groups provide a more convenient way of managing multiple servers that must share the same disk masking configuration. For example in a RHV/oVirt or Vmware environment, each host needs access to the same LUNs. Host groups allow you to create a logical group which contains the hosts and the disks that each host in the group should have access to. Please note that sharing devices across hosts needs a cluster aware filesystem or equivalent locking to avoid data corruption.
.PP
.TP
\fB/iscsi-targets/<iqn>/host-groups/ create | delete <group-name>
Create or delete a given group name. Deleting a group definition does \fBnot\fR remove the hosts or LUN masking, it simply removes the logical grouping used for management purposes.
.PP
.TP
\fB/iscsi-targets/<iqn>/host-groups/<group_name>/ host add | remove <client-iqn>
The host subcommand within a group definition allows you to add and remove hosts from the group. When adding a host, it must not have existing LUN masking in place - this restriction ensure lun id consistency across all hosts within the host group. Removing a host from a group does \fBnot\fR automatically remove it's LUN masking.
.TP
\fB/iscsi-targets/<iqn>/host-groups/<group_name>/ disk add | remove <pool>.<image_name>
The disk subcommand enables you to add and remove disks to/from all members of the host group.
.PP
.RS
\fBNB.\fROnce a client is a member of a host group, it's disks \fBcan only\fR be managed at the group level.
.RE
.SH HOSTS
The 'hosts' section defines the iSCSI client definitions (NodeACLs) that provide access to the rbd images. The CLI provides the ability to create and delete clients, define/update chap authentication and add and remove rbd images for the client.
.PP
.TP
\fB/iscsi-targets/<iqn>/hosts/ create <client_iqn>
The create command will define the client IQN to all gateways within the configuration. At creation time, the client IQN is added to a ACL that allows normal iSCSI session logins for all clients with the IQN. To enable CHAP authentication use the \fBauth\fR command described below.
.TP
\fB/iscsi-targets/<iqn>/hosts/ delete <client_iqn>
The delete command will attempt to remove client IQN from all gateways within the configuration. The client must be logged out, for the delete command to be successful.
.TP
.nf
\fB/iscsi-targets/<iqn>/hosts/ auth nochap\fR
.fi
CHAP authentication can be reset to initiator based ACLs target wide for all setup ACLs using the \fBnochap\fR keyword. If there are multiple clients, CHAP must be enabled for all clients or disabled for all clients. gwcli does not support mixing CHAP clients with IQN ACL clients.
.TP
.nf
\fB/iscsi-targets/<iqn>/hosts/<client_iqn>/ auth chap=<user>/<pswd>\fR
.fi
CHAP authentication can be defined for the client with the \fBchap=\fR parameter. The username and password defined here must then be used within the clients login credentials for this iscsi target. If there are multiple clients, CHAP must be enabled for all clients or disabled for all clients. gwcli does not support mixing CHAP clients with IQN ACL clients.
.TP
.nf
\fB/iscsi-targets/<iqn>/hosts/<client_iqn>/ disk add | remove <disk_name>\fR
.fi
rbd images defined to the iscsi gateway, become LUNs within the LIO environment. These LUNs can be masked to, or masked from specific clients using the \fBdisk\fR command. When a disk is masked to a client, the disk is automatically assigned a LUN id. The disk->LUN id relationship is persisted in the rados configuration object to ensure that the disk always appears on the clients SCSI interface at the same point.
It is the Administrators responsibility to ensure that any disk shared between clients uses a cluster-aware filesystem to prevent data corruption.
.SH EXAMPLES
.PP
.SS CREATING ISCSI GATEWAYS
.TP
\fB>/iscsi-targets create iqn.2003-01.com.redhat.iscsi-gw:ceph-igw\fR
Create a iscsi target name of 'iqn.2003-01.com.redhat.iscsi-gw:ceph-igw', that will be used by each gateway node added to the configuration
.PP
\fB>cd /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw/gateways
.PD 0
.PP
\fB>create ceph-gw-1 10.172.19.21
.TP
\fB>create ceph-gw-2 10.172.19.22
Create 2 gateways, using servers ceph-gw-1 and ceph-gw-2. The iSCSI portals will be bound to the IP addresses provided. During the registration of a gateway a check is performed to ensure the candidate machine has the required IP address available.
.PD 1
.SS ADDING AN RBD
.TP
\fB>/disks/ create pool=rbd image=disk_1 size=50g
Create/import a 50g rbd image and register it with each gateway node
.SS CREATING A CLIENT
.PD 0
\fB>cd /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw/hosts/fR
.PP
.TP
\fB>create iqn.1994-05.com.redhat:rh7-client\fr
Create an iscsi client called 'iqn.1994-05.com.redhat:rh7-client'. The initial client definition will not have CHAP authentication enabled, resulting in red highlighting against this clients summary information in the output of the \fBls\fR command.
.PD 1
.PP
.SS ADDING DISKS TO A CLIENT
.PP
.PD 0
.TP
\fB>/iscsi-target..eph-igw/hosts> cd iqn.1994-05.com.redhat:rh7-client\fR
.PP
.TP
\fB>disk add rbd/disk_1
The first command navigates to the client's entry in the UI at which point the \fBdisk\fR or \fBauth\fR sub-commands may be used. In this example the disk subcommand is used to mask \fIdisk_1\fR in the \fIrbd\fR pool to the iSCSI client. The LUN id associated with this device is automatically assigned and maintained by the system.
.PD 1
.SH OTHER COMMANDS
.TP
\fBexport mode=[ copy ]\fR
with the export command a copy of the current configuration can be exported as a backup (mode=copy). The resulting output is written to stdout.
.TP
\fB/ceph refresh\fR
refreshes the ceph information present in the UI
.TP
\fBinfo\fR
when run at the root of the shell (/), info will show you configuration settings such as http mode, API port, local ceph cluster name and 2ndary API trusted IP addresses.
.TP
\fBgoto [ gateways | hosts | host-groups | 'bookmark']\fR
to ease navigation within the UI, gwcli automatically creates bookmarks for hosts and gateways. This allows you to switch to those sub-trees in the UI by simply using '\fBgoto hosts\fR'. The 'goto' command will also work for any other bookmarks you create.
.PP
.SH FILES
.TP
\fB~/gwcli.log\fR
log file maintained by gwcli, recording all changes made via the shell interface in a timestamped format.
.TP
\fB~/.gwcli/history.txt
log containing a record of all commands executed within the gwcli shell on this system.
.SH AUTHOR
Written by Paul Cuzner ([email protected])
.SH REPORTING BUGS
Report bugs via <https://github.com/ceph/ceph-iscsi-cli/issues>