diff --git a/README.md b/README.md index 557f9c31..e74139e5 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,7 @@ Optional tools and benchmarks can be used if desired: 6. fio - benchmark suite with integrated posix, libaio, and librbd support 7. cosbench - object storage benchmark from Intel + 8. pytest - to run the unit tests. ## USER AND NODE SETUP @@ -144,10 +145,12 @@ lab here: -## CREATING A YAML FILE +## CREATING A TEST PLAN YAML FILE CBT yaml files have a basic structure where you define a cluster and a set of -benchmarks to run against it. For example, the following yaml file creates a +benchmarks to run against it. The high level structure of a test plan is +detailed in the [documentation](docs/TestPlanSchema.md). +For example, the following yaml file creates a single node cluster on a node with hostname "burnupiX". A pool profile is defined for a 1x replication pool using 256 PGs, and that pool is used to run RBD performance tests using fio with the librbd engine. @@ -250,6 +253,15 @@ called mkcephconf.py lets you automatically generate hundreds or thousands of ceph.conf files from defined ranges of different options that can then be used with cbt in this way. +## RECENT FEATURES + +* Support for [workloads](docs/Workloads.md), that is sequence of performance tests, particularly +useful to generate *Response latency curves*. + +* Automatic unit test [generation](docs/AutomaticUnitTestGeneration.md) for the Benchmark classes, intended to help +refactoring to detect regressions. + + ## CONCLUSION There are many additional and powerful ways you can use cbt that are not yet diff --git a/docs/AutomaticUnitTestGeneration.md b/docs/AutomaticUnitTestGeneration.md index 9f8cb0e6..15258752 100644 --- a/docs/AutomaticUnitTestGeneration.md +++ b/docs/AutomaticUnitTestGeneration.md @@ -22,7 +22,7 @@ The following is an example of the execution of the script: ``` An example of the expected normal ouput is shown below. -![cbt_utests_gen](cbt_utest_gen.png) +![cbt_utests_gen](./cbt_utests_gen.png) This would have created (or updated if existing already) the set of unit tests for the supported benchmarks. @@ -35,7 +35,7 @@ The unit tests can be executed from the command line as follows: ``` An example output showing a successful execution: -![cbt_utests_gen](cbt_utest_gen.png) +![cbt_utests_run](./cbt_utests_run.png) Note: the tests skipped above require an environment variable to be defined to identify the target nodes for exercising pdsh. @@ -69,7 +69,7 @@ whether some attributes has been changed, replaced or deleted. This is especiall during code refactoring. -## Workflow recommeded +## Workflow recommeded * Before starting a code refactoring effort, run the unit tests: they should all pass as shown above. diff --git a/docs/TestPlanSchema.md b/docs/TestPlanSchema.md new file mode 100644 index 00000000..16b40076 --- /dev/null +++ b/docs/TestPlanSchema.md @@ -0,0 +1,70 @@ +# Test plan schema + +A valid test plan .yaml consists of the following compulsory sections at the top level (the level is +indicated by the indentation in .yaml: the top level has 0 indentation): + +* `cluster` +* `benchmarks`. + +It may also have the following optional sections at the same level: + +* `monitoring_profile` +* `client_endpoints`. + +![top_level](./toplevel.png) + +## `cluster` + +The cluster section enumerates the components of the Ceph cluster relevant to CBT. There are two +general classes of components: + +* scalars: for example names whose value is a string, a numeric or a boolean; +* collections: components that in turn contain further information, for example profile of pool +replication. + +The following are scalar compulsory entities: +* a head node: this is a string indicating the node that starts the cluster. +* a list of clients, each a string, representing a ssh-reachable host that has a benchmark +executable installed, +* a list of osds nodes, each of which has at least a running OSD process. + +![cluster](./cluster.png) + + +## `benchmarks` + +The benchmarks section consists of a non-empty list of collections, each describing a benchmark +entity. + +* A benchmark entity starts with its *name* (second level indentation), valid names are for example: +`radosbench`, `hsbench`, `kvmrbdfio`, `librbdfio`, etc. + +* The contents of the benchmark entity (third level indentation) consist of a collection of items +(either scalars or collections themselves). Most of these entities represent options for the +command line invocation of the benchmark when executed by the clients. + +![benchmarks](./benchmarks.png) + + +## `monitoring_profiles` + + +The monitoring_profiles section consists of a non-empty list of of collections, each describing a +monitoring tool. + +A monitoring entity starts with its name (at second level indentation). Currently supported are `perf` +, `collectl`, `top`. + +The contents of the monitoring entity consists of : +* a `nodes` (third level indentation) list of processes to monitor (by default the osd nodes), and +* an optional string `args` (third level indentation) to indicate the arguments to the monitoring tool. + + +## `client_endpoints` + +The client_endpoints section consists of a non-empty list of collections, each associated to a +benchmark entity, and typically indicating the driver for the benchmark. The client_endpoints, if +specified on a test plan, must be cross referenced by the benchmark section, and as such normally the +client_endpoints section precedes the benchmarks section in the test plan. + +See the dir `example/` for a number of test plan examples. diff --git a/docs/Workloads.md b/docs/Workloads.md new file mode 100644 index 00000000..81d1f2b0 --- /dev/null +++ b/docs/Workloads.md @@ -0,0 +1,46 @@ +# Workloads + +A workload is the specification of a sequence of tests to be executed in the order given. +Typically this involves a *range* of values for a specific benchmark argument. The most used is +the *queue depth*. Depending of the benchmark, this can be expressed as a function of the number +of jobs (or threads, or processes), such that the increase number of these causes a proportional +increase in the I/O. Specifiying workloads in this way permits to generate *response latency curves* +from the results. + +The workload feature is currently supported for `librbdfio` only. + +![workloads](./workloads.png) + +* A `workloads` section is composed by a non-empty collection. Each item in the workload has a free name, +and contains in turn a collection of valid options with values for the benchmark. +* For each of the `iodepth` and `numjobs` options, a range of integer values is permitted. + +During execution, any of the given values for the benchmark options in the global section are overwritten +by the given values within the current test workload. The global values are restored once the workload test +completes. + +As an example, the following specifies two workloads: + +* the first is named `precondition` and consists of executing a random write over a queue depth of 4, +(that is, the product of numjobs and iodepth), and indicates that monitoring should be disabled during the +execution of the workload, +* the second is named test1, and specifies a random read over the combinatorial of the provided sequences for +the numjobs and iodepth, resp. That is, (1,1), (1,4), (1,8) .. (8,8). + + +```yaml + +workloads: + precondition: + jobname: 'precond1rw' + mode: 'randwrite' + numjobs: [ 1 ] + iodepth: [ 4 ] + monitor: False # whether to run the monitors along the test + test1: + jobname: 'rr' + mode: 'randread' + numjobs: [ 1, 4, 8 ] + iodepth: [ 1, 4, 8 ] + +``` diff --git a/docs/benchmarks.png b/docs/benchmarks.png new file mode 100644 index 00000000..b39686af Binary files /dev/null and b/docs/benchmarks.png differ diff --git a/docs/cluster.png b/docs/cluster.png new file mode 100644 index 00000000..26df07e6 Binary files /dev/null and b/docs/cluster.png differ diff --git a/docs/toplevel.png b/docs/toplevel.png new file mode 100644 index 00000000..cc7e2bb9 Binary files /dev/null and b/docs/toplevel.png differ diff --git a/docs/workloads.png b/docs/workloads.png new file mode 100644 index 00000000..43b50c3d Binary files /dev/null and b/docs/workloads.png differ diff --git a/tools/fio-parse-jsons/fio-parse-jsons.py b/tools/fio-parse-jsons/fio-parse-jsons.py index f21cf9f8..e48faabf 100755 --- a/tools/fio-parse-jsons/fio-parse-jsons.py +++ b/tools/fio-parse-jsons/fio-parse-jsons.py @@ -26,7 +26,7 @@ # crimson4cores_200gb_1img_4k_1procs_randwrite_avg.json # -import glob, os, sys +import os import pprint import json import argparse @@ -39,48 +39,49 @@ # Keys are metric names, vallues are the string path in the .json to seek # For MultiFIO JSON files, the jobname no neccessarily matches any of the predef_dict keys, # so we need instead to use a separate query: -job_type='jobs/jobname=*/job options/rw' +#job_type='jobs/jobname=*/job options/rw' # Better still, use the global options -#job_type='global options/rw' +job_type='global options/rw' # All the following should be within the path # 'jobs/jobname=*/read/iops' predef_dict = { - 'randwrite': { - 'iops' : 'write/iops', - 'total_ios' : 'write/total_ios', - 'clat_ms' : 'write/clat_ns', - 'clat_stdev' : 'write/clat_ns', - 'usr_cpu': 'usr_cpu', - 'sys_cpu': 'sys_cpu' - }, - 'randread': { - 'iops' : 'read/iops', - 'total_ios' : 'read/total_ios', - 'clat_ms' : 'read/clat_ns', - 'clat_stdev' : 'read/clat_ns', - 'usr_cpu': 'usr_cpu', - 'sys_cpu': 'sys_cpu' - }, - 'seqwrite': { - 'bw' : 'write/bw', - 'total_ios' : 'write/total_ios', - 'clat_ms' : 'write/clat_ns', - 'clat_stdev' : 'write/clat_ns', - 'usr_cpu': 'usr_cpu', - 'sys_cpu': 'sys_cpu' - }, - 'seqread': { - 'bw' : 'read/bw', - 'total_ios' : 'read/total_ios', - 'clat_ms' : 'read/clat_ns', - 'clat_stdev' : 'read/clat_ns', - 'usr_cpu': 'usr_cpu', - 'sys_cpu': 'sys_cpu' - } - } + "randwrite": { + "iops": "write/iops", + "total_ios": "write/total_ios", + "clat_ms": "write/clat_ns", + "clat_stdev": "write/clat_ns", + "usr_cpu": "usr_cpu", + "sys_cpu": "sys_cpu", + }, + "randread": { + "iops": "read/iops", + "total_ios": "read/total_ios", + "clat_ms": "read/clat_ns", + "clat_stdev": "read/clat_ns", + "usr_cpu": "usr_cpu", + "sys_cpu": "sys_cpu", + }, + "seqwrite": { + "bw": "write/bw", + "total_ios": "write/total_ios", + "clat_ms": "write/clat_ns", + "clat_stdev": "write/clat_ns", + "usr_cpu": "usr_cpu", + "sys_cpu": "sys_cpu", + }, + "seqread": { + "bw": "read/bw", + "total_ios": "read/total_ios", + "clat_ms": "read/clat_ns", + "clat_stdev": "read/clat_ns", + "usr_cpu": "usr_cpu", + "sys_cpu": "sys_cpu", + }, +} + def filter_json_node(next_branch, jnode_list_in): - """" + """ Traverse the JSON jnode_list_in according to the next_branch: jnode_list_in: [dict] Assumption: json output of non-leaf nodes consists of either @@ -94,14 +95,14 @@ def filter_json_node(next_branch, jnode_list_in): if not next_branch: return next_node_list for n in jnode_list_in: - dotlist = next_branch.split('=') + dotlist = next_branch.split("=") if len(dotlist) > 2: - print( f"unrecognized syntax at {next_branch}") + print(f"unrecognized syntax at {next_branch}") return [] if len(dotlist) == 1: assert isinstance(n, dict) next_node_list.append(n[next_branch]) - else: # must be a sequence, take any element with key matching value + else: # must be a sequence, take any element with key matching value select_key = dotlist[0] select_value = dotlist[1] assert isinstance(n, list) @@ -109,18 +110,19 @@ def filter_json_node(next_branch, jnode_list_in): # n is a list # print 'select with key %s value %s sequence # element %s'%(select_key, select_value, e) - if select_value == '*': + if select_value == "*": next_node_list.append(e) else: v = e[select_key] if v == select_value: next_node_list.append(e) - #print('selecting: %s'%str(e)) + # print('selecting: %s'%str(e)) if len(next_node_list) == 0: print(f"{select_key}={select_value} not found") return [] return next_node_list + def process_fio_item(k, next_node_list): """ Dict of results: @@ -137,27 +139,30 @@ def process_fio_item(k, next_node_list): """ # match k: # Python version on the SV1 node does not support 'match' # case 'iops' | 'usr_cpu' | 'sys_cpu': - if re.search('iops|usr_cpu|sys_cpu|iodepth|total_ios', k): + if re.search("iops|usr_cpu|sys_cpu|iodepth|total_ios", k): return next_node_list[0] - if k == 'bw': - return next_node_list[0]/1000 - if k == 'latency_ms': - # case 'latency_ms': - unsorted_dict=next_node_list[0] - sorted_dict=dict(sorted(unsorted_dict.items(), key=lambda x:x[1], reverse=True)) - firstk=list(sorted_dict.keys())[0] + if k == "bw": + return next_node_list[0] / 1000 + if k == "latency_ms": + # case 'latency_ms': + unsorted_dict = next_node_list[0] + sorted_dict = dict( + sorted(unsorted_dict.items(), key=lambda x: x[1], reverse=True) + ) + firstk = list(sorted_dict.keys())[0] return firstk - if k == 'clat_ms': - # case 'clat_ns': - unsorted_dict=next_node_list[0] - clat_ms=unsorted_dict['mean']/1e6 + if k == "clat_ms": + # case 'clat_ns': + unsorted_dict = next_node_list[0] + clat_ms = unsorted_dict["mean"] / 1e6 return clat_ms - if k == 'clat_stdev': - # case 'clat_ns': - unsorted_dict=next_node_list[0] - clat_stdev=unsorted_dict['stddev']/1e6 + if k == "clat_stdev": + # case 'clat_ns': + unsorted_dict = next_node_list[0] + clat_stdev = unsorted_dict["stddev"] / 1e6 return clat_stdev + def combined_mean(a, b): """ Calculates the combined mean of two groups: @@ -165,48 +170,51 @@ def combined_mean(a, b): FIO already provides the (mean,stdev) of completion latency per sample Expects two tuples: (mx_1, n_1) and (mx_2,n_2), and returns a tuple. """ - mx_1,n_1 = a - mx_2,n_2 = b + mx_1, n_1 = a + mx_2, n_2 = b n_c = n_1 + n_2 - return ( (n_1 * mx_1 + n_2 * mx_2)/n_c, n_c) + return ((n_1 * mx_1 + n_2 * mx_2) / n_c, n_c) + -def combined_std_dev(a,b): +def combined_std_dev(a, b): """ Calculats the combined std dev, normally for the completion latency Expects a,b to be tuples (s_i,x_i) std dev and mean, respectively, and returns a tuple. """ - y_1,n_1 = a - y_2,n_2 = b - s_1,mx_1 = y_1 - s_2,mx_2 = y_2 - mx_c,_nc = combined_mean( (mx_1,n_1), (mx_2,n_2) ) + y_1, n_1 = a + y_2, n_2 = b + s_1, mx_1 = y_1 + s_2, mx_2 = y_2 + mx_c, _nc = combined_mean((mx_1, n_1), (mx_2, n_2)) v_1 = s_1 * s_1 v_2 = s_2 * s_2 q_1 = (n_1 - 1.0) * v_1 + n_1 * (mx_1 * mx_1) q_2 = (n_2 - 1.0) * v_2 + n_2 * (mx_2 * mx_2) q_c = q_1 + q_2 n_c = n_1 + n_2 - return ((math.sqrt( (q_c - n_c * mx_c * mx_c )/(n_c - 1.0) ), mx_c), n_c) + return ((math.sqrt((q_c - n_c * mx_c * mx_c) / (n_c - 1.0)), mx_c), n_c) + def apply_reductor(result_dict, metric): """ Applies the particular reduction to the list of values. Returns a value (scalar numeric) """ - if re.search('iops|usr_cpu|sys_cpu|bw|total_ios', metric): - return functools.reduce( add, result_dict[metric]) - if metric == 'clat_ms': - z = zip(result_dict['clat_ms'], result_dict['total_ios']) - mx,_ = functools.reduce( lambda x,y : combined_mean(x,y), z) + if re.search("iops|usr_cpu|sys_cpu|bw|total_ios", metric): + return functools.reduce(add, result_dict[metric]) + if metric == "clat_ms": + z = zip(result_dict["clat_ms"], result_dict["total_ios"]) + mx, _ = functools.reduce(lambda x, y: combined_mean(x, y), z) return mx - if metric == 'clat_stdev': - z = zip(result_dict['clat_stdev'], result_dict['clat_ms']) - zz = zip(z, result_dict['total_ios']) - zc,_ = functools.reduce( lambda x,y : combined_std_dev(x,y), zz) - sc,_ = zc + if metric == "clat_stdev": + z = zip(result_dict["clat_stdev"], result_dict["clat_ms"]) + zz = zip(z, result_dict["total_ios"]) + zc, _ = functools.reduce(lambda x, y: combined_std_dev(x, y), zz) + sc, _ = zc return sc + def reduce_result_list(result_dict, jobname): """ Applies a reduction to each of the lists of the result_dict: @@ -220,17 +228,18 @@ def reduce_result_list(result_dict, jobname): _res[metric] = apply_reductor(result_dict, metric) return _res + def process_fio_json_file(json_file, json_tree_path): """ Collect metrics from an individual JSON file, which might contain several entries, one per job """ - with open(json_file, 'r') as json_data: + with open(json_file, "r") as json_data: result_dict = {} # check for empty file f_info = os.fstat(json_data.fileno()) if f_info.st_size == 0: - print( f'JSON input file {json_file} is empty') + print(f"JSON input file {json_file} is empty") return result_dict # parse the JSON object node = json.load(json_data) @@ -238,22 +247,23 @@ def process_fio_json_file(json_file, json_tree_path): # different FIO processes result_dict['timestamp'] = str(node['timestamp']) result_dict['iodepth'] = node['global options']['iodepth'] - #result_dict['iodepth'] = node['global options']['rw'] + result_dict['jobname'] = node['global options']['rw'] + result_dict['iodepth'] = node['global options']['iodepth'] # Use the jobname to index the predef_dict for the json query - jobs_list = node['jobs'] + jobs_list = node["jobs"] print(f"Num jobs: {len(jobs_list)}") job_result = {} - for _i,job in enumerate(jobs_list): - jobname = str(job['jobname']) + for _i, job in enumerate(jobs_list): + jobname = str(job["jobname"]) if jobname in predef_dict: # this gives the paths to query for the metrics query_dict = predef_dict[jobname] else: - jobname = job['job options']['rw'] + jobname = job["job options"]["rw"] query_dict = predef_dict[jobname] - result_dict['jobname'] = jobname + #result_dict["jobname"] = jobname for k in query_dict.keys(): - json_tree_path = query_dict[k].split('/') + json_tree_path = query_dict[k].split("/") next_node_list = [job] for next_branch in json_tree_path: @@ -263,10 +273,11 @@ def process_fio_json_file(json_file, json_tree_path): job_result[k] = [] job_result[k].append(item) - reduced = reduce_result_list(job_result, result_dict['jobname']) - merged = { **result_dict, **reduced } + reduced = reduce_result_list(job_result, result_dict["jobname"]) + merged = {**result_dict, **reduced} return merged + def traverse_files(sdir, config, json_tree_path): """ Traverses the JSON files given in the config @@ -283,29 +294,32 @@ def traverse_files(sdir, config, json_tree_path): pp = pprint.PrettyPrinter(width=41, compact=True) dict_new = {} for fname in json_files: - node_list = process_fio_json_file(fname,json_tree_path) + node_list = process_fio_json_file(fname, json_tree_path) dict_new[fname] = node_list print(f"== {fname} ==") pp.pprint(node_list) return dict_new + def gen_plot(config, data, list_subtables, title): """ Generate a gnuplot script and .dat files -- assumes OSD CPU util only """ plot_dict = { - # Use the dict key as the suffix for the output file .png, - # the .dat file is the same for the different charts - 'iops_vs_lat_vs_cpu_sys': { - 'ylabel': "Latency (ms)", - 'ycolumn': '4', - 'y2label': "OSD CPU system", - 'y2column': '9'}, - 'iops_vs_lat_vs_cpu_usr': { - 'ylabel': "Latency (ms)", - 'ycolumn': '4', - 'y2label': "OSD CPU user", - 'y2column': '8'} + # Use the dict key as the suffix for the output file .png, + # the .dat file is the same for the different charts + "iops_vs_lat_vs_cpu_sys": { + "ylabel": "Latency (ms)", + "ycolumn": "4", + "y2label": "OSD CPU system", + "y2column": "9", + }, + "iops_vs_lat_vs_cpu_usr": { + "ylabel": "Latency (ms)", + "ycolumn": "4", + "y2label": "OSD CPU user", + "y2column": "8", + }, } header = """ set terminal pngcairo size 650,420 enhanced font 'Verdana,10' @@ -320,19 +334,19 @@ def gen_plot(config, data, list_subtables, title): """ template = "" - out_plot = config.replace("_list",".plot") - out_data = config.replace("_list",".dat") - # Gnuplot quirk: '_' is interpreted as a sub-index: - _title = title.replace("_","-") + out_plot = config.replace("_list", ".plot") + out_data = config.replace("_list", ".dat") + # Gnuplot quirk: '_' is interpreted as a sub-index: + _title = title.replace("_", "-") - with open(out_plot, 'w') as f: + with open(out_plot, "w") as f: f.write(header) for pk in plot_dict.keys(): out_chart = config.replace("list", pk + ".png") - ylabel = plot_dict[pk]['ylabel'] - ycol = plot_dict[pk]['ycolumn'] - y2label = plot_dict[pk]['y2label'] - y2col = plot_dict[pk]['y2column'] + ylabel = plot_dict[pk]["ylabel"] + ycol = plot_dict[pk]["ycolumn"] + y2label = plot_dict[pk]["y2label"] + y2col = plot_dict[pk]["y2column"] template += f""" set ylabel "{ylabel}" set xlabel "IOPS" @@ -347,21 +361,24 @@ def gen_plot(config, data, list_subtables, title): """ # To plot CPU util in the same response curve, we need the extra axis # This list_subtables indicates how many sub-tables the .datfile will have - # The stdev is the error column:5 + # The stdev is the error column:5 if len(list_subtables) > 0: head = f"plot '{out_data}' index 0 using 2:{ycol}:5 t '{list_subtables[0]} q-depth' w yerr axes x1y1" head += f",\\\n '' index 0 using 2:{ycol}:5 notitle w lp axes x1y1" head += f",\\\n '' index 0 using 2:{y2col} w lp axes x1y2 t 'CPU%'" - tail = ",\\\n".join([ f" '' index {i} using 2:{ycol} t '{list_subtables[i]} q-depth' w lp axes x1y1" - for i in range(1,len(list_subtables))]) + tail = ",\\\n".join([ + f" '' index {i} using 2:{ycol} t '{list_subtables[i]} q-depth' w lp axes x1y1" + for i in range(1, len(list_subtables)) + ]) template += ",\\\n".join([head, tail]) f.write(template) f.close() - with open(out_data, 'w') as f: + with open(out_data, "w") as f: f.write(data) f.close() + def initial_fio_table(dict_files, multi): """ Construct a table from the input mesurements FIO json @@ -372,16 +389,16 @@ def initial_fio_table(dict_files, multi): # Traverse each json sample for name in dict_files.keys(): item = dict_files[name] - jobname = item['jobname'] + jobname = item["jobname"] subdict = predef_dict[jobname] - _keys = [ 'iodepth' ] + [ *subdict.keys() ] + _keys = ["iodepth"] + [*subdict.keys()] # Each k is a FIO metric (iops, lat, etc) for k in _keys: - if not k in table: + if k not in table: table[k] = [] table[k].append(item[k]) if multi: - if not k in table: + if k not in table: avg[k] = 0.0 avg[k] += item[k] # Probably might use some other avg rather than arithmetic avg @@ -392,28 +409,29 @@ def initial_fio_table(dict_files, multi): avg[k] /= len(table[k]) return table, avg + def aggregate_cpu_avg(avg, table, avg_cpu): """ Depending of whether this set of results are from a Multi FIO or a typical response curve, aggregate the OSD CPU avg measurements into the main table """ - # Note: if num_files > len(avg_cpu): this is a MultiFIO + # Note: if num_files > len(avg_cpu): this is a MultiFIO if len(avg_cpu): print(f" avg_cpu list has: {len(avg_cpu)} items") - # The number of CPU items should be the same as the number of dict_files.keys() + # The number of CPU items should be the same as the number of dict_files.keys() for cpu_item in avg_cpu: - for k in cpu_item.keys(): # 'sys', 'us' normally from the OSD + for k in cpu_item.keys(): # 'sys', 'us' normally from the OSD cpu_avg_k = 0 samples = cpu_item[k] for cpu in samples.keys(): cpu_avg_k += samples[cpu] cpu_avg_k /= len(samples.keys()) - # Aggregate the CPU values in the avg table - if not k in avg: + # Aggregate the CPU values in the avg table + if k not in avg: avg[k] = 0 avg[k] += cpu_avg_k - # Aggregate the CPU values in the FIO table - if not k in table: + # Aggregate the CPU values in the FIO table + if k not in table: table[k] = [] table[k].append(cpu_avg_k) @@ -421,6 +439,7 @@ def aggregate_cpu_avg(avg, table, avg_cpu): print("Table (after aggregating OSD CPU avg data):") pp.pprint(table) + def gen_table(dict_files, config, title, avg_cpu, multi=False): """ Construct a table from the predefined keys, sorted according to the @@ -453,17 +472,21 @@ def gen_table(dict_files, config, title, avg_cpu, multi=False): # For the gnuplot .dat, each subtable ranges over num_jobs(threads) # whereas each row within a table ranges over iodepth gplot_hdr = "# " - gplot_hdr += ' '.join(table.keys()) + gplot_hdr += " ".join(table.keys()) gplot_hdr += "\n" gplot = "" - wiki = r"""{| class="wikitable" + wiki = ( + r"""{| class="wikitable" |- -! colspan="7" | """ + config.replace("_list","") + """ +! colspan="7" | """ + + config.replace("_list", "") + + """ ! colspan="2" | OSD CPU% |- ! """ - wiki += ' !! '.join(table.keys()) + ) + wiki += " !! ".join(table.keys()) wiki += "\n|-\n" for k in table.keys(): @@ -473,13 +496,13 @@ def gen_table(dict_files, config, title, avg_cpu, multi=False): # our naming convention to identify this: # fio_crimson_1osd_default_8img_fio_unrest_2job_16io_4k_randread_p5.json for name in dict_files.keys(): - m = re.match(r".*(?P\d+)job_(?P\d+)io_",name) + m = re.match(r".*(?P\d+)job_(?P\d+)io_", name) if m: # Note: 'job' (num threads) is constant within each table, # each row corresponds to increasing the iodepth, that is, each # sample run - job = int(m.group('job')) # m.group(1) - _io = int(m.group('io')) # m.group(2) + job = int(m.group("job")) # m.group(1) + _io = int(m.group("io")) # m.group(2) if job not in list_subtables: # Add a gnuplot table break (ie new block) if len(list_subtables) > 0: @@ -490,32 +513,33 @@ def gen_table(dict_files, config, title, avg_cpu, multi=False): for k in table.keys(): item = next(table_iters[k]) - if k == 'iodepth': #This metric is the first column + if k == "iodepth": # This metric is the first column gplot += f" {item} " - wiki += f' | {item} ' + wiki += f" | {item} " else: gplot += f" {item:.2f} " - wiki += f' || {item:.2f} ' + wiki += f" || {item:.2f} " gplot += "\n" wiki += "\n|-\n" if multi: - wiki += '! Avg:' + wiki += "! Avg:" for k in avg.keys(): - wiki += f' || {avg[k]:.2f} ' + wiki += f" || {avg[k]:.2f} " wiki += "\n|-\n" - if 'iops' in avg.keys(): - total = avg['iops'] * num_files + if "iops" in avg.keys(): + total = avg["iops"] * num_files else: - total = avg['bw'] * num_files - wiki += f'! Total: || {total:.2f} ' + total = avg["bw"] * num_files + wiki += f"! Total: || {total:.2f} " wiki += "\n|-\n" - # format_numeric = lambda num: f"{num:e}" if isinstance(num, int) else f"{num:,.2f}" + # format_numeric = lambda num: f"{num:e}" if isinstance(num, int) else f"{num:,.2f}" wiki += "|}\n" print(f" Wiki table: {title}") print(wiki) gen_plot(config, gplot, list_subtables, title) - print('Done') + print("Done") + def main(directory, config, json_query): """ @@ -523,13 +547,14 @@ def main(directory, config, json_query): evolved """ if not bool(json_query): - json_query='jobs/jobname=*' - json_tree_path = json_query.split('/') + json_query = "jobs/jobname=*" + json_tree_path = json_query.split("/") dicto_files = traverse_files(directory, config, json_tree_path) - print('Note: clat_ns has been converted to milliseconds') - print('Note: bw has been converted to MiBs') + print("Note: clat_ns has been converted to milliseconds") + print("Note: bw has been converted to MiBs") return dicto_files + def load_avg_cpu_json(json_fname): """ Load a .json file containing the CPU avg samples -- normally produced by the script @@ -541,7 +566,7 @@ def load_avg_cpu_json(json_fname): # check for empty file f_info = os.fstat(json_data.fileno()) if f_info.st_size == 0: - print(f'JSON input file {json_fname} is empty') + print(f"JSON input file {json_fname} is empty") return cpu_avg_list # parse the JSON: list of dicts with keys 'sys' and 'us' cpu_avg_list = json.load(json_data) @@ -549,37 +574,61 @@ def load_avg_cpu_json(json_fname): except IOError as e: raise argparse.ArgumentTypeError(str(e)) + def parse_args(): """ As it says on the tin """ - parser = argparse.ArgumentParser(description='Parse set of output json FIO results.') + parser = argparse.ArgumentParser( + description="Parse set of output json FIO results." + ) parser.add_argument( - "-c", "--config", type=str, - required=True, - help="Name of the file with the list of JSON files names to examine", default="") + "-c", + "--config", + type=str, + required=True, + help="Name of the file with the list of JSON files names to examine", + default="", + ) parser.add_argument( - "-t", "--title", type=str, - required=True, - help="Title for the response curve gnuplot chart", default="") + "-t", + "--title", + type=str, + required=True, + help="Title for the response curve gnuplot chart", + default="", + ) parser.add_argument( - "-a", "--average", type=str, - help="Name of the JSON file with the CPU avg", default="") + "-a", + "--average", + type=str, + help="Name of the JSON file with the CPU avg", + default="", + ) parser.add_argument( - "-d", "--directory", type=str, - help="result directory to evaluate", default="./") + "-d", "--directory", type=str, help="result directory to evaluate", default="./" + ) parser.add_argument( - "-q", "--query", type=str, - required=False, - help="JSON query", default="jobs/jobname=*") + "-q", + "--query", + type=str, + required=False, + help="JSON query", + default="jobs/jobname=*", + ) parser.add_argument( - '-m', '--multi', action='store_true', - required=False, - help="Indicate multiple FIO instance as opposed to response curves", default=False) + "-m", + "--multi", + action="store_true", + required=False, + help="Indicate multiple FIO instance as opposed to response curves", + default=False, + ) args = parser.parse_args() return args -if __name__=='__main__': + +if __name__ == "__main__": args = parse_args() dict_files = main(args.directory, args.config, args.query) avg_cpu = load_avg_cpu_json(args.average)