MacOS unable to complete plot, loops at phase 1:3 leaving behind temp files until drive full

[chiknbake]

I have an issue where the plotman job stops durning or near the start of phase 1:3 and then restarts a new job leaving behind the temp files until drive drive fills up. This happens on every attempted job, with both chia and madam plotter, and with default and modified plotman configs. What is also weird is that the plotman and the failed plot logs do not show any errors. Here is an example of both logs below. Any ideas on how I can troubleshot this? I've already tried restarts, older plotman versions, another t drive, and splitting up plotting to tmp and t2.

System details:
Version: Plotman 0.5.3+dev
System: macOS Big Sur 11.6 + Docker Desktop
Machine: MacBook Pro 13,3, 512gb NVME

Plot Log:

Bucket 125 uniform sort. Ram: 5.199GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
Bucket 126 uniform sort. Ram: 5.199GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
Bucket 127 uniform sort. Ram: 5.199GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
Total matches: 4294920212
Forward propagation table time: 2379.922 seconds. CPU (119.640%) Tue Nov 2 05:15:59 2021
Computing table 3
Bucket 0 uniform sort. Ram: 5.199GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
Bucket 1 uniform sort. Ram: 5.199GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
END OF LOG

Plotman Log:

...sleeping 20 s: max jobs (1) - (2693s/1800s)
...sleeping 20 s: max jobs (1) - (2713s/1800s)
Starting plot job: chia plots create -k 32 -r 3 -u 128 -b 5389 -t /plotting -d /plots -f 84a604f106f96286ca00cc345baa34bc6d4eff58a82b0a68429bed44f3dc7711314b51155b26b1656f6de1120f4ba7bb -c xch19t0dr0gmv5jag2r7ykn4n8t8ex67j76sgfyfq84acsjygenmdzesdhjru0 ; logging to /root/.chia/plotman/logs/2021-11-02T04_30_15.548287-07_00.plot.log
...sleeping 20 s: stagger (19s/1800s)
...sleeping 20 s: stagger (39s/1800s)

Full logs attached:
plotman.log
2021-11-02T05_16_29.160172-07_00.plot.log

[altendky]

At that point plotman has no particular interaction with the plotting process beyond reading the log file and checking process stats from the OS. Have you tried manually running a plot to see what it does?

[chiknbake]

Yes, manually running the plot had the same behavior. Brand new to this so sorry if this is obvious, it sounds like plotman is the supervisor of the plotter so if that's the case are there ways to log the actual plotter?

[altendky]

I call it a plot scheduler, but yeah, supervisor works too. The plotters just log to stdout, your console. You can the --plotter > a_file to overwrite a_file with the output of the plotter or the --plotter | tee a_file to see it on the console and overwrite a_file. I can try to help you here, but if you want more interactive support feel free to hop onto Chia's keybase (https://keybase.io/team/chia_network.public). The basic guesses would be running out of disk or running out of memory.

[chiknbake]

Thank you altendky, this whole time I've been focusing on disk but it was actually the default Docker Desktop preferences which has a resource limit on memory (~3gb) which was trigger all the loops/restarts. Very weird since in my limited linux docker experience there is no default resource limit so I wasn't expecting anything different on the Mac side (aside from the docker desktop GUI). Thanks again.