use as many goroutines for ForgetInode op #79

kungf · 2020-03-17T03:46:12Z

in #30 forgetinode was changed into inline ServerOps, this may solove
memory oom, but the performance of rm op will be very slow, and it will
also hang other ops, so i think add a goroutine pool to limit the max
num of forgetinode goroutines, and it will not affect the performance

kungf · 2020-03-17T06:40:04Z

hi @stapelberg do you have some advise?

stapelberg

Have you done benchmarks for this? Can you share the results?

I’m asking because spawning goroutines is actually not necessarily beneficial. In fact, when I did my last measurements, it was more performant to never spawn a goroutine.

stapelberg · 2020-03-17T07:45:30Z

fuseutil/file_system.go

@@ -21,6 +21,7 @@ import (

 	"github.com/jacobsa/fuse"
 	"github.com/jacobsa/fuse/fuseops"
+	"github.com/panjf2000/ants/v2"


Not keen on adding a new dependency just for this change. If we decide to go forward with this change, we should implement it without extra dependencies. Check out sourcegraph/gophercon-2018-liveblog#35 for inspiration

stapelberg · 2020-03-17T07:46:20Z

fuseutil/file_system.go

@@ -93,6 +94,12 @@ type fileSystemServer struct {
 	opsInFlight sync.WaitGroup
 }

+type opCtx struct {
+	c   *fuse.Connection
+	ctx context.Context


Embedding context.Context in structs is an anti-pattern: https://groups.google.com/forum/#!topic/golang-nuts/xRbzq8yzKWI

kahing · 2020-03-17T07:49:17Z

I don't think 100000 is the right number. Either we allocate all of them at startup which wastes 200MB for most cases, or we don't allocate them at startup in which case you have OOM during a storm of ForgetInode.

Using a very small pool maybe ok, note that implementation may have to lock anyway so having that many forget goroutines running isn't useful

kungf · 2020-03-18T02:23:04Z

I don't think 100000 is the right number. Either we allocate all of them at startup which wastes 200MB for most cases, or we don't allocate them at startup in which case you have OOM during a storm of ForgetInode.

hi, @kahing the 10w is the capacity, will not allocate until used, and if the goroutine idle too long time, it will be recycled!

kungf · 2020-03-18T09:24:42Z

Have you done benchmarks for this? Can you share the results?

you can see from the follow result, 4 tasks,1000files, the "File removal" performance up from 22 to 5175!
Before modify

./mdtest "-n" "1000" "-u" "-z" "2" "-i" "3" "-F" "-d" "/home/wangyang5/polefs/mnt"
Path: /home/wangyang5/polefs
FS: 50.0 GiB   Used FS: 37.9%   Inodes: 25.0 Mi   Used Inodes: 0.8%

4 tasks, 3996 files

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       4530.794       4530.784       4530.788          0.004
   File stat         :     972973.342     972860.389     972935.691         53.246
   File read         :      17802.887      17802.792      17802.849          0.041
   File removal      :         22.489         22.489         22.489          0.000
   Tree creation     :        597.252        424.639        532.511         76.785
   Tree removal      :         58.258         16.345         35.345         17.333

-- finished at 03/18/2020 17:12:34 --

After modify

Command line used: ./mdtest "-n" "1000" "-u" "-z" "2" "-i" "3" "-F" "-d" "/home/wangyang5/polefs/mnt"
Path: /home/wangyang5/polefs
FS: 50.0 GiB   Used FS: 37.9%   Inodes: 25.0 Mi   Used Inodes: 0.8%

4 tasks, 3996 files

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       4469.286       4469.280       4469.283          0.002
   File stat         :    1013572.737    1012898.941    1013286.855        284.380
   File read         :      17506.221      17506.093      17506.136          0.060
   File removal      :       5175.368       5175.358       5175.363          0.004
   Tree creation     :        579.831        540.689        565.124         17.397
   Tree removal      :        447.424        427.932        435.875          8.356

-- finished at 03/18/2020 17:01:36 --

in jacobsa#30 forgetinode was changed into inline ServerOps, this may solove memory oom, but the performance of rm op will be very slow, and it will also hang other ops, so i think limit the max num of forgetinode goroutines, this can avoid oom but not affect performance

stapelberg · 2020-03-18T16:50:59Z

Can you share the results of that same benchmark, but with a goroutine pool of only 2 goroutines please?

kungf · 2020-03-19T01:59:31Z

2 goroutines

./mdtest "-n" "1000" "-u" "-z" "2" "-i" "3" "-F" "-d" "/home/wangyang5/polefs/mnt"
Path: /home/wangyang5/polefs
FS: 50.0 GiB   Used FS: 37.8%   Inodes: 25.0 Mi   Used Inodes: 0.8%

4 tasks, 3996 files

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       4578.542       4578.532       4578.538          0.004
   File stat         :     971056.708     970550.627     970813.018        207.029
   File read         :      18393.587      18393.405      18393.500          0.074
   File removal      :         46.243         46.243         46.243          0.000
   Tree creation     :        653.828        461.589        564.816         79.122
   Tree removal      :         71.337         27.775         45.803         18.560

100 goroutines

   ./mdtest "-n" "1000" "-u" "-z" "2" "-i" "3" "-F" "-d" "/home/wangyang5/polefs/mnt"
Path: /home/wangyang5/polefs
FS: 50.0 GiB   Used FS: 37.9%   Inodes: 25.0 Mi   Used Inodes: 0.8%

4 tasks, 3996 files

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       4580.740       4580.723       4580.733          0.008
   File stat         :     990979.648     990628.216     990842.974        153.729
   File read         :      18073.165      18073.068      18073.120          0.040
   File removal      :       2281.023       2281.021       2281.022          0.001
   Tree creation     :        624.400        446.060        558.440         79.864
   Tree removal      :        457.793        438.781        445.753          8.549

riking · 2020-07-05T06:40:22Z

What we're seeing here is that the kernel will parallelize dispatch of operations that have no logical dependencies on each other. The test is looking for throughput of removals, while @stapelberg's optimization was looking for latency of close(2).

A better solution may be to keep a goroutine pool for all operations instead of spinning them up as requests come in, and then spinning up fresh goroutines only under high load if all pool goroutines are busy.

stapelberg reviewed Mar 17, 2020

View reviewed changes

kungf force-pushed the multi_forgetinode branch from 7dd5564 to 8fc09b8 Compare March 18, 2020 09:17

kungf changed the title ~~use goroutine pool to concurrent forgetinode~~ improve file delete performance Mar 18, 2020

kungf force-pushed the multi_forgetinode branch from 8fc09b8 to 4b07f0d Compare March 18, 2020 09:35

kungf changed the title ~~improve file delete performance~~ use as many goroutines for ForgetInode op Mar 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use as many goroutines for ForgetInode op #79

use as many goroutines for ForgetInode op #79

kungf commented Mar 17, 2020

kungf commented Mar 17, 2020

stapelberg left a comment

stapelberg Mar 17, 2020

stapelberg Mar 17, 2020

kahing commented Mar 17, 2020

kungf commented Mar 18, 2020

kungf commented Mar 18, 2020

stapelberg commented Mar 18, 2020

kungf commented Mar 19, 2020

riking commented Jul 5, 2020 •

edited

Loading

use as many goroutines for ForgetInode op #79

Are you sure you want to change the base?

use as many goroutines for ForgetInode op #79

Conversation

kungf commented Mar 17, 2020

kungf commented Mar 17, 2020

stapelberg left a comment

Choose a reason for hiding this comment

stapelberg Mar 17, 2020

Choose a reason for hiding this comment

stapelberg Mar 17, 2020

Choose a reason for hiding this comment

kahing commented Mar 17, 2020

kungf commented Mar 18, 2020

kungf commented Mar 18, 2020

stapelberg commented Mar 18, 2020

kungf commented Mar 19, 2020

2 goroutines

100 goroutines

riking commented Jul 5, 2020 • edited Loading

riking commented Jul 5, 2020 •

edited

Loading