See constant suggester errors on Docker instance of opengrok

[codeplay87]

Running in k8s seeing the output flooded with these messages. Not sure what this is or how to diagnose.

SEVERE: Could not create ChronicleMap for field defs in directory /opengrok/data/suggester/_{repo}_ , most popular completion disabled, if you are using JDK9+ make sure to specify: --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-exports java.base/sun.nio.ch=ALL-UNNAMED
java.io.IOException: Permission denied
	at java.base/sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
	at java.base/sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:68)
	at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:109)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
	at java.base/sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:850)
	at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:836)
	at net.openhft.chronicle.hash.impl.util.FileIOUtils.writeFully(FileIOUtils.java:44)
	at net.openhft.chronicle.map.ChronicleMapBuilder.writeNotComplete(ChronicleMapBuilder.java:369)
	at net.openhft.chronicle.map.ChronicleMapBuilder.openWithExistingFile(ChronicleMapBuilder.java:1921)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createWithFile(ChronicleMapBuilder.java:1713)
	at net.openhft.chronicle.map.ChronicleMapBuilder.recoverPersistedTo(ChronicleMapBuilder.java:1629)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1612)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1604)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1598)
	at org.opengrok.suggest.popular.impl.chronicle.ChronicleMapAdapter.<init>(ChronicleMapAdapter.java:55)
	at org.opengrok.suggest.SuggesterProjectData.initSearchCountMap(SuggesterProjectData.java:314)
	at org.opengrok.suggest.SuggesterProjectData.init(SuggesterProjectData.java:161)
	at org.opengrok.suggest.Suggester.lambda$getInitRunnable$1(Suggester.java:233)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Jun 07, 2023 4:21:29 PM org.opengrok.suggest.SuggesterProjectData initSearchCountMap
SEVERE: Could not create ChronicleMap for field defs in directory /opengrok/data/suggester/_{repo}_ , most popular completion disabled, if you are using JDK9+ make sure to specify: --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-exports java.base/sun.nio.ch=ALL-UNNAMED
java.io.IOException: Not enough space to fallocate 0 MiB to / unallocated was 0 MiB
	at net.openhft.chronicle.hash.impl.VanillaChronicleHash.fallocate(VanillaChronicleHash.java:1078)
	at net.openhft.chronicle.hash.impl.VanillaChronicleHash.map(VanillaChronicleHash.java:1058)
	at net.openhft.chronicle.hash.impl.VanillaChronicleHash.createMappedStoreAndSegments(VanillaChronicleHash.java:511)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createWithNewFile(ChronicleMapBuilder.java:1850)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createWithFile(ChronicleMapBuilder.java:1748)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createPersistedTo(ChronicleMapBuilder.java:1593)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1614)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1604)
	at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1598)
	at org.opengrok.suggest.popular.impl.chronicle.ChronicleMapAdapter.<init>(ChronicleMapAdapter.java:55)
	at org.opengrok.suggest.SuggesterProjectData.initSearchCountMap(SuggesterProjectData.java:314)
	at org.opengrok.suggest.SuggesterProjectData.init(SuggesterProjectData.java:161)
	at org.opengrok.suggest.Suggester.lambda$getInitRunnable$1(Suggester.java:233)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

[vladak]

I am not sure myself. Must be something specific to how the container is deployed in k8s because I've never seen it myself with plain Docker. First thing I'd do is to log, er, docker exec -it, into the container in question and make sure /opengrok/data/suggester/ can be written to, has some space available etc.

Assuming this is the official OpenGrok Docker container. The container has tools such as touch, fallocate, etc.

[codeplay87]

So the shared Azure Persistant Volume I have is only using 7% of capacity for the opengrok/{src|data|etc} folders so plenty of space. The ownership and group is root and can be written to. Do the processes of mirror / indexer run as non root users?

[codeplay87]

Apologies still learning around linux / docker. What would be the best way to trace the system calls? Currently I am just looking at the docker output / logs.

[vladak]

Same here. strace -f -p <tomcat_pid> would be good start, I think. You will need to use it on the Tomcat process and look for failed system calls related to those exceptions in the log.

[codeplay87]

I have trace information but not sure what I am looking for and not sure best way to provide some of the information to you.

[vladak]

Basically you are looking for failed syscalls. strace can filter these with -e status option.

[vladak]

Also, the --absolute-timestamps might be handy to correlate the failed syscalls with the exceptions.

[codeplay87]

ETIMEDOUT (Connection timed out)
[pid 114261] 15:55:01 readlink("/opengrok", 0x7eff277fb510, 1023) = -1 EINVAL (Invalid argument)
[pid 114261] 15:55:01 readlink("/opengrok/data", 0x7eff277fb510, 1023) = -1 EINVAL (Invalid argument)
[pid 114261] 15:55:01 readlink("/opengrok/data/suggester", 0x7eff277fb510, 1023) = -1 EINVAL (Invalid argument)
[pid 114261] 15:55:01 readlink("/opengrok/data/suggester/frontend-shared-assets", 0x7eff277fb510, 1023) = -1 EINVAL (Invalid argument)
[pid 114261] 15:55:01 readlink("/opengrok/data/suggester/frontend-shared-assets/defs_search_count.db", 0x7eff277fb510, 1023) = -1 EINVAL
 (Invalid argument)
[pid 114261] 15:55:01 pwrite64(460, "$\6\0\200", 4, 8) = -1 EACCES (Permission denied)

Details of the above file.

-rwxrwxrwx 1 root root   12288 Jun  8 15:54 defs_search_count.db

[codeplay87]

So its on AKS with the file system being a shared persisted volume of type azurefile-csi-premium.

[vladak]

So its on AKS with the file system being a shared persisted volume of type azurefile-csi-premium.

How does it appear in the container ? i.e. what kind of file-system, what mount options.

[codeplay87]

ran the following command df -Th /opengrok/data which resulted in this. This is the same for etc, src, tmp

Filesystem                                Type  Size  Used Avail Use% Mounted on
//{guid}.file.core.windows.net/pvc-{guid} cifs  100G   37M  100G   1% /opengrok/data

[codeplay87]

Running cat mount | grep data

//{guid}.file.core.windows.net/pvc-{pvc-guid} /opengrok/data cifs rw,relatime,vers=3.1.1,cache=strict,username={guid},uid=0,noforceuid,gid=0,noforcegid,addr={addr},file_mode=0777,dir_mode=0777,soft,persistenthandles,nounix,mapposix,mfsymlinks,rsize=10485
76,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=30 0 0

[codeplay87]

@vladak we have also tried changing the disk type to be azure file disk which then mounts as ext4

[stevenewey]

@codeplay87 I've hit the same issue in trying to run Opengrok in Amazon EKS this week. I believe I've solved it by adding an emptyDir mount for /tmp in the opengrok container. In my case that's coming from the local instance NVMe which has a couple of hundred Gb of free space. Hope that's helpful!

[codeplay87]

Thanks for this, we currently have a mount for the /tmp directory on the opengrok container to a PV with 100gb+ space as we were keen to ensure that locks on the projects were kept and used across multiple instances of opengrok containers. I.e. so that two instances didn't try and mirror and index at the same time on the project. Do you have multiple instances of the opengrok container?