Azcopy on Mac: too many open files

[pkcecw]

Which version of the AzCopy was used? ---> AzcopyVersion 10.24.0

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux) ---> Mac

What command did you run? ---> azcopy cp https://xxxpip01.blob.core.windows.net/xxx/PIDxxx/" "/Users/vhcadministrator/Desktop/xxx" --overwrite=ifSourceNewer --recursive=true

What problem was encountered? ---> too many open files

How can we reproduce the problem in the simplest way? running the same command again

Have you found a mitigation/solution? No

Here is the session output:

MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1392
virtual memory (kbytes, -v) unlimited
MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -n
256
MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -n 65356
MID-CLINIC-4-IMAC:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 65356
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1392
virtual memory (kbytes, -v) unlimited

XXX:azcopy_darwin_amd64_10.25.1 vhcadministrator$ ./azcopy cp "https://xxxpip01.blob.core.windows.net/xxx/PIDXXX/?xxx" "/Users/vhcadministrator/Desktop/xxx" --overwrite=ifSourceNewer --recursive=true
INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job c363ca9f-3417-164f-6f71-99b7b284486e has started
Log file is located at: /Users/vhcadministrator/.azcopy/c363ca9f-3417-164f-6f71-99b7b284486e.log

5.1 %, 11973 Done, 67109 Failed, 198388 Pending, 2530 Skipped, 280000 Total (scanning...), 2-sec Throughput (Mb/s): 99.564 panic: open /Users/vhcadministrator/.azcopy/plans/c363ca9f-3417-164f-6f71-99b7b284486e--00028.steV18: too many open files

goroutine 1 [running]:
github.com/Azure/azure-storage-azcopy/v10/common.PanicIfErr(...)
/Users/runner/work/1/s/common/lifecyleMgr.go:724
github.com/Azure/azure-storage-azcopy/v10/ste.JobPartPlanFileName.Map({0xc001f27cc0, 0x32})
/Users/runner/work/1/s/ste/JobPartPlanFileName.go:66 +0x1a6
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).AddJobPart2(0xc000b3c708, 0xc00d92b1d0)
/Users/runner/work/1/s/ste/mgr-JobMgr.go:453 +0x25a
github.com/Azure/azure-storage-azcopy/v10/jobsAdmin.ExecuteNewCopyJobPartOrder({0x0, {0xc363ca9f, 0x3417, 0x164f, {0x6f, 0x71, 0x99, 0xb7, 0xb2, 0x84, ...}}, ...})
/Users/runner/work/1/s/jobsAdmin/init.go:194 +0x2ab
github.com/Azure/azure-storage-azcopy/v10/cmd.inprocSend({0x10a01fa7c, 0x10}, {0x10a2598c0, 0xc00016d888}, {0x10a259980, 0xc016eec018})
/Users/runner/work/1/s/cmd/rpc.go:39 +0xf8
github.com/Azure/azure-storage-azcopy/v10/cmd.init.func4({0x10a01fa7c?, 0xc00076c730?}, {0x10a2598c0?, 0xc00016d888?}, {0x10a259980?, 0xc016eec018?})
/Users/runner/work/1/s/cmd/rpc.go:31 +0x2c
github.com/Azure/azure-storage-azcopy/v10/cmd.addTransfer(, {{0xc00e871cb0, 0x86}, {0xc00e86ca00, 0x91}, 0x0, {0x0, 0xeddbb16a0, 0xc001af63f0}, 0x87c2, ...}, ...)
/Users/runner/work/1/s/cmd/copyEnumeratorHelper.go:24 +0xe3
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).initEnumerator.func5({{0xc00ea21d0f, 0x31}, 0x0, {0x0, 0xeddbb16a0, 0xc001af63f0}, {0x0, 0x0, 0x0}, 0x87c2, ...})
/Users/runner/work/1/s/cmd/copyEnumeratorInit.go:321 +0x585
github.com/Azure/azure-storage-azcopy/v10/cmd.processIfPassedFilters({, _, _}, {{0xc00ea21d0f, 0x31}, 0x0, {0x0, 0xeddbb16a0, 0xc001af63f0}, {0x0, ...}, ...}, ...)
/Users/runner/work/1/s/cmd/zc_enumerator.go:841 +0x97
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).parallelList(0xc00022cbd0, 0xc0000a80a0, {0xc000010c2f, 0x38}, {0xc000010c68, 0xb}, {0x0, 0x0}, 0x0, 0xc000037a40, ...)
/Users/runner/work/1/s/cmd/zc_traverser_blob.go:439 +0x338
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).Traverse(0xc00022cbd0, 0x0, 0xc000037a40, {0x10ac2f140, 0x0, 0x0})
/Users/runner/work/1/s/cmd/zc_traverser_blob.go:309 +0xa45
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CopyEnumerator).enumerate(0xc000149ec0)
/Users/runner/work/1/s/cmd/zc_enumerator.go:789 +0x3a
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).processCopyJobPartOrders(0xc0002ae008)
/Users/runner/work/1/s/cmd/copy.go:1623 +0xf06
github.com/Azure/azure-storage-azcopy/v10/cmd.(*CookedCopyCmdArgs).process(0xc0004a8870?)
/Users/runner/work/1/s/cmd/copy.go:1273 +0x65
github.com/Azure/azure-storage-azcopy/v10/cmd.init.2.func2(0xc0000e9b88?, {0xc000149680?, 0x2?, 0x10a01443a?})
/Users/runner/work/1/s/cmd/copy.go:2028 +0x245
github.com/spf13/cobra.(*Command).execute(0xc0000e9b88, {0xc000149600, 0x4, 0x4})
/Users/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x684
github.com/spf13/cobra.(*Command).ExecuteC(0x10abbf940)
/Users/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x38d
github.com/spf13/cobra.(*Command).Execute(...)
/Users/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/Azure/azure-storage-azcopy/v10/cmd.Execute({0xc00003a340?, 0x10aba5d30?}, {0xc00003e7b0?, 0xc0001450c8?}, 0xc0001450c0?, {0xc363ca9f, 0x3417, 0x164f, {0x6f, 0x71, ...}})
/Users/runner/work/1/s/cmd/root.go:220 +0xec
main.main()
/Users/runner/work/1/s/main.go:79 +0x47b

[pkcecw]

Can someone review the above error help me on this please?