lizardfs-benchmarks

4 nodes

Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 19 write iops (10% HDD)
1 file, 1 thread, rnd 16k writes, simple: 268 write iops (37% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 264 write iops (40% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 335 write iops (95% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 348 write iops (82% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 280 write iops (62% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 312 write iops (61% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 25 write iops (16% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 2594 write iops (396% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 1121 write iops (178% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 842 write iops (180% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 4824 write iops (898% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 4992 write iops (996% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 269 write iops (173% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 264 write iops (109% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 1078 write iops (213% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 1008 write iops (198% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 3 read iops (0% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 370 read iops (216% HDD)
16 files, 1 thread each, seq 1M reads, simple: 104 read iops (2% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 67170 read iops (14291% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 72861 read iops (14689% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 280 write iops (45% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 2827 write iops (564% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 365 read iops (278% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 80553 read iops (16306% HDD)
Tests complete on karatasi @ 2019-09-22 12:11:31.
Files remain. To clean up, add argument "cleanup".
kenneth@karatasi:/mnt/liz-client/backup> ./storage-tuner-benchmark here . 
Running tests in "./stb-testdir" on karatasi @ 2019-09-23 22:24:04 ...
storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 9 write iops (4% HDD)
1 file, 1 thread, rnd 16k writes, simple: 272 write iops (38% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 276 write iops (41% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 412 write iops (117% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 389 write iops (91% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 308 write iops (68% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 325 write iops (64% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 19 write iops (12% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 1919 write iops (293% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 1367 write iops (218% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 1181 write iops (253% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 5032 write iops (937% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 4982 write iops (994% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 177 write iops (114% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 168 write iops (69% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 994 write iops (196% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 1009 write iops (198% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 5 read iops (0% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 420 read iops (245% HDD)
16 files, 1 thread each, seq 1M reads, simple: 6 read iops (0% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 55074 read iops (11717% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 58892 read iops (11873% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 212 write iops (34% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 2972 write iops (593% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 429 read iops (327% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 64134 read iops (12982% HDD)
Tests complete on karatasi @ 2019-09-23 22:27:45.
Files remain. To clean up, add argument "cleanup".

storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 20 write iops (11% HDD)
1 file, 1 thread, rnd 16k writes, simple: 270 write iops (37% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 245 write iops (37% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 356 write iops (101% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 332 write iops (78% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 313 write iops (70% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 298 write iops (59% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 28 write iops (18% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 2603 write iops (398% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 873 write iops (139% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 974 write iops (209% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 3089 write iops (575% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 4961 write iops (990% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 265 write iops (170% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 274 write iops (113% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 968 write iops (191% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 846 write iops (166% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 4 read iops (0% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 356 read iops (208% HDD)
16 files, 1 thread each, seq 1M reads, simple: 165 read iops (4% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 58710 read iops (12491% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 62699 read iops (12640% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 260 write iops (42% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 3222 write iops (643% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 363 read iops (277% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 72852 read iops (14747% HDD)
Tests complete on karatasi @ 2019-09-25 20:32:59.
Files remain. To clean up, add argument "cleanup".

5 nodes

storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 57 write iops (31% HDD)
1 file, 1 thread, rnd 16k writes, simple: 15 write iops (2% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 7 write iops (1% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 17 write iops (4% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 19 write iops (4% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 16 write iops (3% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 16 write iops (3% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 35 write iops (23% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 111 write iops (16% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 322 write iops (51% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 311 write iops (66% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 1016 write iops (189% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 1551 write iops (309% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 14 write iops (9% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 14 write iops (5% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 271 write iops (53% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 156 write iops (30% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 657 read iops (17% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 1347 read iops (787% HDD)
16 files, 1 thread each, seq 1M reads, simple: 175 read iops (4% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 67676 read iops (14399% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 75265 read iops (15174% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 19 write iops (3% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 358 write iops (71% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 1486 read iops (1134% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 78257 read iops (15841% HDD)
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 72 write iops (39% HDD)
1 file, 1 thread, rnd 16k writes, simple: 20 write iops (2% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 17 write iops (2% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 18 write iops (5% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 12 write iops (2% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 22 write iops (4% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 17 write iops (3% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 35 write iops (23% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 382 write iops (58% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 366 write iops (58% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 343 write iops (73% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 1301 write iops (242% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 1126 write iops (224% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 14 write iops (9% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 11 write iops (4% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 376 write iops (74% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 198 write iops (38% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 228 read iops (6% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 807 read iops (471% HDD)
16 files, 1 thread each, seq 1M reads, simple: 173 read iops (4% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 63355 read iops (13479% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 70160 read iops (14145% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 22 write iops (3% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 450 write iops (89% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 782 read iops (596% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 73098 read iops (14797% HDD)
Tests complete on karatasi @ 2019-09-23 23:05:57.
Files remain. To clean up, add argument "cleanup".
Creating test directory "stb-testdir"
Running tests in "./stb-testdir" on mwanafunzi @ 2019-09-25 21:54:20 ...
storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 85 write iops (46% HDD)
1 file, 1 thread, rnd 16k writes, simple: 24 write iops (3% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 30 write iops (4% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 27 write iops (7% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 25 write iops (5% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 32 write iops (7% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 25 write iops (4% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 34 write iops (22% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 947 write iops (144% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 902 write iops (143% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 949 write iops (203% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 2767 write iops (515% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 2729 write iops (544% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 25 write iops (16% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 28 write iops (11% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 893 write iops (176% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 650 write iops (127% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 105 read iops (2% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 468 read iops (273% HDD)
16 files, 1 thread each, seq 1M reads, simple: 7409 read iops (183% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 151158 read iops (32161% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 440291 read iops (88768% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 25 write iops (4% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 818 write iops (163% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 508 read iops (387% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 508793 read iops (102994% HDD)
Tests complete on mwanafunzi @ 2019-09-25 21:57:13.
Files remain. To clean up, add argument "cleanup".

Laptop was running at 100 mb instead of 1Gb. Took that node offline and ran with 4 including imara2

Running tests in "./stb-testdir" on mwanafunzi @ 2019-09-25 21:58:11 ...
storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 79 write iops (43% HDD)
1 file, 1 thread, rnd 16k writes, simple: 25 write iops (3% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 25 write iops (3% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 28 write iops (7% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 27 write iops (6% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 28 write iops (6% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 26 write iops (5% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 87 write iops (58% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 1267 write iops (193% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 1267 write iops (202% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 1276 write iops (273% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 5135 write iops (956% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 5267 write iops (1051% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 25 write iops (16% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 22 write iops (9% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 1180 write iops (233% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 850 write iops (166% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 230 read iops (6% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 480 read iops (280% HDD)
16 files, 1 thread each, seq 1M reads, simple: 10396 read iops (257% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 152363 read iops (32417% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 443535 read iops (89422% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 23 write iops (3% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 1262 write iops (251% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 448 read iops (341% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 480207 read iops (97207% HDD)
Tests complete on mwanafunzi @ 2019-09-25 22:00:54.
Files remain. To clean up, add argument "cleanup".
Creating test directory "stb-testdir"
Running tests in "./stb-testdir" on mwanafunzi @ 2019-09-25 22:01:40 ...
storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 103 write iops (56% HDD)
1 file, 1 thread, rnd 16k writes, simple: 33 write iops (4% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 34 write iops (5% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 37 write iops (10% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 37 write iops (8% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 32 write iops (7% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 38 write iops (7% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 86 write iops (57% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 641 write iops (98% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 603 write iops (96% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 566 write iops (121% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 3117 write iops (580% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 2441 write iops (487% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 33 write iops (21% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 30 write iops (12% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 600 write iops (118% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 154 write iops (30% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 122 read iops (3% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 529 read iops (309% HDD)
16 files, 1 thread each, seq 1M reads, simple: 9609 read iops (237% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 110624 read iops (23537% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 544756 read iops (109829% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 36 write iops (5% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 662 write iops (132% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 311 read iops (237% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 525425 read iops (106361% HDD)
Tests complete on mwanafunzi @ 2019-09-25 22:04:27.
Files remain. To clean up, add argument "cleanup".
Creating test directory "stb-testdir"
Running tests in "./stb-testdir" on mwanafunzi @ 2019-09-25 22:12:45 ...
storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 108 write iops (59% HDD)
1 file, 1 thread, rnd 16k writes, simple: 47 write iops (6% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 40 write iops (6% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 37 write iops (10% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 52 write iops (12% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 47 write iops (10% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 51 write iops (10% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 95 write iops (63% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 1187 write iops (181% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 1164 write iops (185% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 676 write iops (145% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 3062 write iops (570% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 2219 write iops (442% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 45 write iops (29% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 40 write iops (16% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 409 write iops (80% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 312 write iops (61% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 91 read iops (2% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 555 read iops (324% HDD)
16 files, 1 thread each, seq 1M reads, simple: 7014 read iops (173% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 2688 read iops (571% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 5759 read iops (1161% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 41 write iops (6% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 958 write iops (191% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 252 read iops (192% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 221980 read iops (44935% HDD)
Tests complete on mwanafunzi @ 2019-09-25 22:15:27.
Files remain. To clean up, add argument "cleanup".
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 103 write iops (56% HDD)
1 file, 1 thread, rnd 16k writes, simple: 30 write iops (4% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 30 write iops (4% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 37 write iops (10% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 34 write iops (8% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 30 write iops (6% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 30 write iops (5% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 91 write iops (60% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 636 write iops (97% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 660 write iops (105% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 581 write iops (124% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 3014 write iops (561% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 2336 write iops (466% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 29 write iops (18% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 32 write iops (13% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 609 write iops (120% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 379 write iops (74% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 1198 read iops (32% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 594 read iops (347% HDD)
16 files, 1 thread each, seq 1M reads, simple: 7023 read iops (173% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 230192 read iops (48977% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 588258 read iops (118600% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 34 write iops (5% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 610 write iops (121% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 600 read iops (458% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 605657 read iops (122602% HDD)

Comparison benchmarks

Using this script

Single NVME

Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 788 write iops (435% HDD)
1 file, 1 thread, rnd 16k writes, simple: 2222 write iops (312% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 2216 write iops (336% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 5911 write iops (1684% HDD)
1 file, 16 threads, rnd 8k writes, posixaio:   5890 write iops (1389% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 5446 write iops (1218% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 5450 write iops (1081% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 1361 write iops (907% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 6853 write iops (1047% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 6926 write iops (1104% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 6001 write iops (1287% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 7825 write iops (1457% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 7798 write iops (1556% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 1935 write iops (1248% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 2020 write iops (838% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 5718 write iops (1132% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 5538 write iops (1088% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 1022 read iops (27% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 4845 read iops (2833% HDD)
16 files, 1 thread each, seq 1M reads, simple: 1218 read iops (30% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 57716 read iops (12280% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 63905 read iops (12884% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 2190 write iops (357% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 6817 write iops (1360% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 5994 read iops (4575% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 61152 read iops (12378% HDD)
Tests complete on linux-k9r1 @ 2019-10-02 23:20:45.
Files remain. To clean up, add argument "cleanup".
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 820 write iops (453% HDD)
1 file, 1 thread, rnd 16k writes, simple: 2258 write iops (317% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 2243 write iops (340% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 6301 write iops (1795% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 6203 write iops (1462% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 5711 write iops (1277% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 5651 write iops (1121% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 1358 write iops (905% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 6920 write iops (1058% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 6941 write iops (1107% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 6068 write iops (1302% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 7612 write iops (1417% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 7568 write iops (1510% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 1942 write iops (1252% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 2015 write iops (836% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 5842 write iops (1156% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 5675 write iops (1114% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 1032 read iops (27% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 4760 read iops (2783% HDD)
16 files, 1 thread each, seq 1M reads, simple: 1211 read iops (29% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 58127 read iops (12367% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 64106 read iops (12924% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 2259 write iops (368% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 6435 write iops (1284% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 6169 read iops (4709% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 62023 read iops (12555% HDD)

SATA SSD

Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 134 write iops (74% HDD)
1 file, 1 thread, rnd 16k writes, simple: 207 write iops (29% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 230 write iops (34% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 347 write iops (98% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 367 write iops (86% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 270 write iops (60% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 273 write iops (54% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 57 write iops (38% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 726 write iops (111% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 958 write iops (152% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 568 write iops (121% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 759 write iops (141% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 774 write iops (154% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 102 write iops (65% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 108 write iops (44% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 546 write iops (108% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 665 write iops (130% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 3826 read iops (102% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 3725 read iops (2178% HDD)
16 files, 1 thread each, seq 1M reads, simple: 5693 read iops (140% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 333893 read iops (71041% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 672383 read iops (135561% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 192 write iops (31% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 895 write iops (178% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 3638 read iops (2777% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 941116 read iops (190509% HDD)
Tests complete on mwanafunzi @ 2019-10-03 17:49:55.
Files remain. To clean up, add argument "cleanup".

BTRFS RAID 1 HDD

Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 89 write iops (49% HDD)
1 file, 1 thread, rnd 16k writes, simple: 19 write iops (2% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 17 write iops (2% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 287 write iops (81% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 285 write iops (67% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 254 write iops (56% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 257 write iops (50% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 64 write iops (42% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 166 write iops (25% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 167 write iops (26% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 178 write iops (38% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 1043 write iops (194% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 946 write iops (188% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 16 write iops (10% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 19 write iops (7% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 158 write iops (31% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 168 write iops (33% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 19 read iops (0% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 123 read iops (71% HDD)
16 files, 1 thread each, seq 1M reads, simple: 4 read iops (0% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 474 read iops (100% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 466 read iops (93% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 22 write iops (3% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 176 write iops (35% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 115 read iops (87% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 479 read iops (96% HDD)
Tests complete on mwanafunzi @ 2019-10-03 17:47:04.
Files remain. To clean up, add argument "cleanup".

XFS HDD

Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 152 write iops (83% HDD)
1 file, 1 thread, rnd 16k writes, simple: 118 write iops (16% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 117 write iops (17% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 160 write iops (45% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 159 write iops (37% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 155 write iops (34% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 155 write iops (30% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 117 write iops (78% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 174 write iops (26% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 171 write iops (27% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 111 write iops (23% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 212 write iops (39% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 205 write iops (40% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 30 write iops (19% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 30 write iops (12% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 104 write iops (20% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 108 write iops (21% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 165 read iops (4% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 136 read iops (79% HDD)
16 files, 1 thread each, seq 1M reads, simple: 132 read iops (3% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 284 read iops (60% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 316 read iops (63% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 121 write iops (19% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 192 write iops (38% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 139 read iops (106% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 300 read iops (60% HDD)

Moosefs comparison

absolutely absurdly faster performance compared to lizardfs.

Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 101 write iops (55% HDD)
1 file, 1 thread, rnd 16k writes, simple: 393 write iops (55% HDD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 386 write iops (58% HDD)
1 file, 16 threads, rnd 4k writes, posixaio: 5422 write iops (1544% HDD)
1 file, 16 threads, rnd 8k writes, posixaio: 4401 write iops (1037% HDD)
1 file, 16 threads, rnd 16k writes, posixaio: 3239 write iops (724% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 3238 write iops (642% HDD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 110 write iops (73% HDD)
16 files, 1 thread each, rnd 16k writes, simple: 5096 write iops (779% HDD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 5127 write iops (817% HDD)
16 files, 1 thread each, rnd 16k writes, posixaio: 5204 write iops (1116% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio: 7020 write iops (1307% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 7015 write iops (1400% HDD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 372 write iops (240% HDD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 355 write iops (147% HDD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 5183 write iops (1026% HDD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 5199 write iops (1021% HDD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 1515 read iops (40% HDD)
1 file, 16 threads, rnd 16k reads, posixaio: 1292 read iops (755% HDD)
16 files, 1 thread each, seq 1M reads, simple: 10113 read iops (250% HDD)
16 files, 1 thread each, rnd 16k reads, posixaio: 393889 read iops (83806% HDD)
16 files, 16 threads each, rnd 16k reads, posixaio: 872489 read iops (175905% HDD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 401 write iops (65% HDD)
16 files, 16 threads each, rnd 16k writes, native aio: 5115 write iops (1020% HDD)
1 file, 16 threads, rnd 16k reads, native aio: 1412 read iops (1077% HDD)
16 files, 16 threads each, rnd 16k reads, native aio: 900873 read iops (182362% HDD)
Tests complete on mwanafunzi @ 2020-10-01 12:04:34.
Files remain. To clean up, add argument "cleanup".