-
Notifications
You must be signed in to change notification settings - Fork 166
/
NEWS
299 lines (250 loc) · 13 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Version 4.1.0+dev
--------------------------------------------------------------------------------
New major features:
New minor features:
Bugfixes:
Version 4.0.0
--------------------------------------------------------------------------------
New major features:
- Add PMDK backend (Adrian Jackson)
- Add CHFS backend (Osamu Tatebe)
- Add AIO backend (Julian Kunkel)
- New S3 backend (Julian Kunkel)
- New md-workbench benchmark
- New BFS backend (Osamu Tatebe)
- Add GPUDirect Storage support (Julian Kunkel)
New minor features:
- Updated DAOS backend to match latest DAOS APIs (Mohamad Chaarawi)
- Updated HDF5 backend (Jordan Henderson)
- Improved Spectrum Scale hint support (Pidad D'Souza)
- Add CSV output (Sebastian Oeste)
- Improved Gfarm support (Osamu Tatebe)
- Updated HDFS backend
- MDTest now has a directory renaming test
- Add prologue/epilogue support (GUAN Xin)
- Add stronger randomized data using golden primes (Glenn K. Lockwood)
- Add dummy backend (Julian Kunkel)
- Improved CephFS support (Pablo Llopis)
- Improved PnetCDF support (Wei-keng Liao)
Bugfixes:
- Expansive fixes to mdtest correctness (Julian Kunkel)
Version 3.3.0
--------------------------------------------------------------------------------
New major features:
- Add CephFS AIORI (Mark Nelson)
- Add Gfarm AIORI (Osamu Tatebe)
- Add DAOS AIORI (Mohamad Chaarawi)
- Add DAOS DFS AIORI (Mohamad Chaarawi)
- -B option has been replaced with --posix.odirect
New minor features:
- Display outlier host names (Jean-Yves Vet)
- Enable global default dir layout for subdirs in Lustre (Petros Koutoupis)
- Removed pound signs (#) from mdtest output file names (Julian Kunkel)
- Print I/O hints from NCMPI (Wei-keng Liao)
- Add mknod support to mdtest (Gu Zheng)
- Refactor AIORI-specific options (Julian Kunkel)
- Enable IME native backend for mdtest (Jean-Yves Vet)
- Enable mkdir/rmdir to IME AIORI (Jean-Yves Vet)
- Add HDF5 collective metadata option (Rob Latham)
- Add support for sync to AIORIs (Julian Kunkel)
General user improvements and bug fixes:
- Allocate aligned buffers to support DirectIO for BeeGFS (Sven Breuner)
- Added IOPS and latency results to json output (Robert LeBlanc)
- Fixed case where numTasks is not evenly divisible by tasksPerNode (J. Schwartz)
- Fix several memory leaks and buffer alignment problems (J. Schwartz, Axel Huebl, Sylvain Didelot)
- Add mdtest data verification (Julian Kunkel)
- Clean up functionality of stonewall (Julian Kunkel)
- Fix checks for lustre_user.h (Andreas Dilger)
- Make write verification work without read test (Jean-Yves Vet)
- Documentation updates (Vaclav Hapla, Glenn Lockwood)
- Add more debugging support (J. Schwartz)
General developer improvements:
- Fix type casting errors (Vaclav Hapla)
- Add basic test infrastructure (Julian Kunkel, Glenn Lockwood)
- Conform to strict C99 (Glenn Lockwood)
Known issues:
- S3 and HDFS backends may not compile with new versions of respective libraries
Version 3.2.1
--------------------------------------------------------------------------------
- Fixed a memory protection bug in mdtest (Julian Kunkel)
- Fixed correctness bugs in mdtest leaf-only mode (#147) (rmn1)
- Fixed bug where mdtest attempted to stat uncreated files (Julian Kunkel)
Version 3.2.0
--------------------------------------------------------------------------------
New major features:
- mdtest now included as a frontend for the IOR aiori backend (Nathan Hjelm,
LANL)
- Added mmap AIORI (Li Dongyang, DDN)
- Added RADOS AIORI (Shane Snyder, ANL)
- Added IME AIORI (Jean-Yves Vet, DDN)
- Added stonewalling for mdtest (Julian Kunkel, U Reading)
New minor features:
- Dropped support for PLFS AIORI (John Bent)
- Added stoneWallingWearOut functionality to allow stonewalling to ensure that
each MPI rank writes the same amount of data and captures the effects of slow
processes (Julian Kunkel, DKRZ)
- Added support for JSON output (Enno Zickler, U Hamburg; Julian Kunkel, U
Reading)
- Added dummy AIORI (Julian Kunkel, U Reading)
- Added support for HDF5 collective metadata operations (Rob Latham, ANL)
General user improvements:
- BeeGFS parameter support (Oliver Steffen, ThinkParQ)
- Semantics of `-R` now compares to expected signature (`-G`) (Julian Kunkel,
DKRZ)
- Improved macOS support for ncmpi (Vinson Leung)
- Added more complete documentation (Enno Zickler, U Hamburg)
- Assorted bugfixes and code refactoring (Adam Moody, LLNL; Julian Kunkel, U
Reading; Enno Zickler, U Hamburg; Nathan Hjelm, LANL; Rob Latham, ANL;
Jean-Yves Vet, DDN)
- More robust support for non-POSIX-backed AIORIs (Shane Snyder, ANL)
General developer improvements:
- Improvements to build process (Nathan Hjelm, LANL; Ian Kirker, UCL)
- Added continuous integration support (Enno Zickler, U Hamburg)
- Added better support for automated testing (Julian Kunkel, U Reading)
- Rewritten option handling to improve support for backend-specific options
(Julian Kunkel, U Reading)
Known issues:
- `api=RADOS` cannot work correctly if specified in a config file (via `-f`)
because `-u`/`-c`/`-p` cannot be specified (issue #98)
- `writeCheck` cannot be enabled for write-only tests using some AIORIs such as
MPI-IO (pull request #89)
Version 3.0.2
--------------------------------------------------------------------------------
- IOR and mdtest now share a common codebase. This will make it easier
run performance benchmarks on new hardware.
- Note: this version was never properly released
Version 3.0.0
--------------------------------------------------------------------------------
- Reorganization of the build system. Now uses autoconf/automake. N.B. Windows
support is not included. Patches welcome.
- Much code refactoring.
- Final summary table is printed after all tests have finished.
- Error messages significantly improved.
- Drop all "undocumented changes". If they are worth having, they need to be
implemented well and documented.
Version 2.10.3
--------------------------------------------------------------------------------
- bug 2962326 "Segmentation Fault When Summarizing Results" fixed
- bug 2786285 "-Wrong number of parameters to function H5Dcreate" fixed
(NOTE: to compile for HDF5 1.6 libs use "-D H5_USE_16_API")
- bug 1992514 "delay (-d) doesn't work" fixed
Contributed by [email protected]
- Ported to Windows. Required changes related to 'long' types, which on Windows
are always 32-bits, even on 64-bit systems. Missing system headers and
functions account for most of the remaining changes.
New files for Windows:
- IOR/ior.vcproj - Visual C project file
- IOR/src/C/win/getopt.{h,c} - GNU getopt() support
See updates in the USER_GUIDE for build instructions on Windows.
- Fixed bug in incrementing transferCount
- Fixed bugs in SummarizeResults with mismatched format specifiers
- Fixed inconsistencies between option names, -h output, and the USER_GUIDE.
Version 2.10.2
--------------------------------------------------------------------------------
- Extend existing random I/O capabilities and enhance performance
output statistics. (Hodson, 8/18/2008)
Version 2.10.1
--------------------------------------------------------------------------------
- Added '-J' setAlignment option for HDF5 alignment in bytes; default value
is 1, which does not set alignment
- Changed how HDF5 and PnetCDF calculate performance -- formerly each used
the size of the stat()'ed file; changed it to be number of data bytes
transferred. these library-generated files can have large headers and
filler as well as sparse file content
- Found potential overflow error in cast -- using IOR_offset_t, not int now
- Replaced HAVE_HDF5_NO_FILL macro to instead directly check if H5_VERS_MAJOR
H5_VERS_MINOR are defined; if so, then must be HDF5-1.6.x or higher for
no-fill usage.
- Corrected IOR_GetFileSize() function to point to HDF5 and NCMPI versions of
IOR_GetFileSize() calls
- Changed the netcdf dataset from 1D array to 4D array, where the 4 dimensions
are: [segmentCount][numTasks][numTransfers][transferSize]
This patch from Wei-keng Liao allows for file sizes > 4GB (provided no
single dimension is > 4GB).
- Finalized random-capability release
- Changed statvfs() to be for __sun (SunOS) only
- Retired Python GUI
Version 2.10.0.1
--------------------------------------------------------------------------------
- Cleaned up WriteOrRead(), reducing much to smaller functions.
- Added random capability for transfer offset.
- Modified strtok(NULL, " \t\r\n") in ExtractHints() so no trailing characters
- Added capability to set hints in NCMPI
Version 2.9.6.1
--------------------------------------------------------------------------------
- For 'pvfs2:' filename prefix, now skips DisplayFreeSpace(); formerly this
caused a problem with statvfs()
- Changed gethostname() to MPI_Get_processor_name() so for certain cases
when gethostname() would only return the frontend node name
- Added SunOS compiler settings for makefile
- Updated O_DIRECT usage for SunOS compliance
- Changed statfs() to instead use statvfs() for SunOS compliance
- Renamed compiler directive _USE_LUSTRE to _MANUALLY_SET_LUSTRE_STRIPING
Version 2.9.5
--------------------------------------------------------------------------------
- Wall clock deviation time relabeled to be "Start time skew across all tasks".
- Added notification for "Using reorderTasks '-C' (expecting block, not cyclic,
task assignment)"
- Corrected bug with read performance with stonewalling (was using full size,
stat'ed file instead of bytes transferred).
Version 2.9.4
--------------------------------------------------------------------------------
- Now using IOR_offset_t instead of int for tmpOffset in IOR.c:WriteOrRead().
Formerly, this would cause error in file(s) > 2GB for ReadCheck. The
more-commonly-used WriteCheck option was not affected by this.
Version 2.9.3
--------------------------------------------------------------------------------
- Changed FILE_DELIMITER from ':' to '@'.
- Any time skew between nodes is automatically adjusted so that all nodes
are calibrated to the root node's time.
- Wall clock deviation time is still reported, but have removed the warning
message that was generated for this. (Didn't add much new information,
just annoyed folks.)
- The '-j' outlierThreshold option now is set to 0 (off) as default. To use
this, set to a positive integer N, and any task who's time (for open,
access, etc.) is not within N seconds of the mean of all the tasks will show
up as an outlier.
Version 2.9.2
--------------------------------------------------------------------------------
- Simple cleanup of error messages, format, etc.
- Changed error reporting so that with VERBOSITY=2 (-vv) any error encountered
is displayed. Previously, this was with VERBOSITY=3, along with full test
parameters, environment, and all access times of operations.
- Added deadlineForStonewalling option (-D). This option is used for measuring
the amount of data moved in a fixed time. After the barrier, each task
starts its own timer, begins moving data, and the stops moving data at a
prearranged time. Instead of measuring the amount of time to move a fixed
amount of data, this option measures the amount of data moved in a fixed
amount of time. The objective is to prevent tasks slow to complete from
skewing the performance.
Version 2.9.1
--------------------------------------------------------------------------------
- Updated test script to run through file1:file2 cases.
- Program exit code is now total numbers of errors (both writecheck and
readcheck for all iterations), unless quitOnError (-q) is set.
- For various failure situations, replace abort with warning, including:
- failed uname() for platform name now gives warning
- failed unlink() of data file now gives warning
- failed fsync() of data file now gives warning
- failed open() of nonexistent script file now gives warning
- Changed labelling for error checking output to be (hopefully) more
clear in details on errors for diagnostics.
- Another fix for -o file1:file2 option.
- Corrected bug in GetTestFileName() -- now possible to handle -o file1:file2
cases for file-per-proc.
Version 2.9.0
--------------------------------------------------------------------------------
- Improved checkRead option to reread data from different node (to avoid cache)
and then compare both reads.
- Added outlierThreshold (-j) option to give warning if any task is more than
this number of seconds from the mean of all participating tasks. If so, the
task is identified, its time (start, elapsed create, elapsed transfer,
elapsed close, or end) is reported, as is the mean and standard deviation for
all tasks. The default for this is 5, i.e. any task not within 5 seconds of
the mean for those times is displayed. This value can be set with
outlierThreshold=<value> or -j <value>.
- Correct for clock skew between nodes - if skew greater than wallclock
deviation threshold (WC_DEV_THRESHOLD) in seconds, then broadcast root
node's timestamp and adjust by the difference. WC_DEV_THRESHOLD currently
set to 5.
- Added a Users Guide.