Replies: 2 comments 7 replies
-
|
@gjosh91 , I don't think the cluster certificate error is relevant either but just to be sure, are you running a dual management server setup? If so please stop one of the management servers and try again. In your dashboard, do you see not only both systemvms running but also both agent-states as connected? Nothing I see here seems to be relevant to a upload from local storage failing. Can you check if your secondary storage is properly seeded, i.e. there is a templates directory, and if you upload from url that a template gets created? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @DaanHoogland , sorry for the late reply. My agent service is running but in dashboard is not showing as connected.
Please guide how to make my agent state connected ? Also regarding the template directory : My template directory is getting created on the nfs machine and I can see the system vm template there in the directory structure /mnt/secondary/template/tmpl/1/3 
Also please notice that although I can see the secondary storage , in the dashboard the secondary storage allocated space is being reflected as 0 Gb : 
When trying to create instance using the default system VM template . I get this : 
Just to add to the above query : My VM gets created successfully when doing direct download but my instance does not get created . Getting this error : 
Error in logs : 2024-07-29 17:11:39,393 ERROR [c.c.v.VmWorkJobHandlerProxy] Error |
Beta Was this translation helpful? Give feedback.
-
|
Hi @DaanHoogland Sorry for pestering you. Hope you are doing fine. |
Beta Was this translation helpful? Give feedback.
-
|
Het @gjosh91 , sorry for not answering, in visiting the Olympics. I'll have look later. In the meanwhile, check if you can mount the storages on your hosts and ssvm. |
Beta Was this translation helpful? Give feedback.
-
|
@DaanHoogland Hope you are doing great. I was away from this topic for a while. I have gone through all the possibles steps I could have . |
Beta Was this translation helpful? Give feedback.
-
|
@gjosh91 , I would go through the page that Jithin pointed to first. You are right that it might not be completely up to date but most of it is valid. Please suggest any changes if you find errors. |
Beta Was this translation helpful? Give feedback.
-
|
@gjosh91 can you try the troubleshooting steps here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting |
Beta Was this translation helpful? Give feedback.
-
|
@rajujith I tried the troubleshooting steps but there are two issues : |
Beta Was this translation helpful? Give feedback.
-
|
@gjosh91 Yes that document is not upto date but the common troubleshooting steps remains the same mostly. |
Beta Was this translation helpful? Give feedback.
-
ERROR:
Request Failed (530)
There is no secondary storage VM for downloading template to image store secondary.
My Setup:
All running on Ubuntu 22
Management Server : IP - 192.168.1.122
NFS Server : IP - 192.168.1.182
KVM - 192.168.1.44
Required :
Add ISO from local storage .
Just for your perspective . The dashboard is showing my System VMs are running one is console proxy and the other is secondary storage vm.
My infrastructure is showing that I have a primary storage and a secondary storage. My Dashboard is showing the memory allocated to my primary storage but not to my secondary storage.
My mount point are also functioning and no error in log.
Any guidance will be appreciated . I have also attached the screenshot for agent logs and management logs for your reference.
Management server log -
Please note - I do not think the error related to certificate is cause for the error I am receiving. Correct me if I am wrong.
2024-07-25 12:46:16,560 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Executing command [/bin/bash -c du -sh /var/log/cloudstack/management/management-server.log | cut -f '1' ].
2024-07-25 12:46:16,564 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Successfully executed process [17716] for command [/bin/bash -c du -sh /var/log/cloudstack/management/management-server.log | cut -f '1' ].
2024-07-25 12:46:16,565 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Executing command [/bin/bash -c df -h /var/log/cloudstack/management/management-server.log | grep -v Filesystem | awk '{print "on disk " $1 " mounted on " $6 " (" $5 " full)"}' ].
2024-07-25 12:46:16,571 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Successfully executed process [17719] for command [/bin/bash -c df -h /var/log/cloudstack/management/management-server.log | grep -v Filesystem | awk '{print "on disk " $1 " mounted on " $6 " (" $5 " full)"}' ].
2024-07-25 12:46:16,573 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersdirect.capacity, 122261
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersdirect.count, 20
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersdirect.used, 122261
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersmapped.capacity, 76568
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersmapped.count, 5
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersmapped.used, 76568
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-MarkSweep.count, 6
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-MarkSweep.time, 1147
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-Scavenge.count, 33
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-Scavenge.time, 249
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail jvmname, 15681@hcimg-HP-ProBook-440-G5
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail jvmuptime, 1409565
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail jvmvendor, Ubuntu OpenJDK 64-Bit Server VM 11.0.23+9-post-Ubuntu-1ubuntu122.04.1 (11)
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memoryheap.committed, 953155584
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memoryheap.init, 130023424
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memoryheap.usage, 0.1845245948043442
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.committed, 194682880
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.init, 7667712
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.max, -1
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.usage, -1.89691784E8
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.used, 189691784
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30991228070175436
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.14020948580378564
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2614160862465425
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.Compressed-Class-Space.usage, 0.011386938393115997
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.Metaspace.usage, 0.9769060697991255
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.PS-Eden-Space.usage, 0.31128890661153935
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.PS-Old-Gen.usage, 0.1051116015875309
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.PS-Survivor-Space.usage, 0.1280120481927711
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.committed, 1147838464
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.init, 137691136
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.max, 1908932607
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.used, 541936800
2024-07-25 12:46:16,577 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail threadsdeadlocks, []
2024-07-25 12:46:16,577 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail threadsnew.count, 0
2024-07-25 12:46:16,578 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail threadstimed_waiting.count, 74
2024-07-25 12:46:16,581 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"fc5923e9-3eac-46b0-93b3-3edbf9b595fb","collectionTime":"Jul 25, 2024, 12:46:16 PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":953155584,"freeJvmMemoryBytes":602002456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1409455,"jvmStartTime":1721890367082,"availableProcessors":8,"loadAverage":0.435546875,"totalInit":137691136,"totalUsed":540842440,"totalCommitted":1147838464,"pid":15681,"jvmName":"15681@hcimg-HP-ProBook-440-G5","jvmVendor":"Ubuntu","jvmVersion":"11.0.23+9-post-Ubuntu-1ubuntu122.04.1","osDistribution":"Ubuntu 22.04.4 LTS","agentCount":1,"heapMemoryUsed":352245016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":11,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":112,"systemMemoryTotal":8206192640,"systemMemoryFree":700325888,"systemMemoryUsed":1294708,"systemMemoryVirtualSize":8336060416,"logInfo":"/var/log/cloudstack/management/management-server.log using: 34M\non disk /dev/nvme0n1p2 mounted on / (11% full)","systemTotalCpuCycles":13586.599,"systemLoadAverages":[0.44,0.49,0.58],"systemCyclesUsage":[352458,122206,5591261],"dbLocal":true,"usageLocal":false,"systemBootTime":"Jul 25, 2024, 10:38:22 AM","kernelVersion":"5.15.0-116-generic"} to 233349941812470
2024-07-25 12:46:16,581 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Cluster PDU 233349941812470 -> 233349941812470. agent: 0, pdu seq: 24, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"fc5923e9-3eac-46b0-93b3-3edbf9b595fb","collectionTime":"Jul 25, 2024, 12:46:16 PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":953155584,"freeJvmMemoryBytes":602002456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1409455,"jvmStartTime":1721890367082,"availableProcessors":8,"loadAverage":0.435546875,"totalInit":137691136,"totalUsed":540842440,"totalCommitted":1147838464,"pid":15681,"jvmName":"15681@hcimg-HP-ProBook-440-G5","jvmVendor":"Ubuntu","jvmVersion":"11.0.23+9-post-Ubuntu-1ubuntu122.04.1","osDistribution":"Ubuntu 22.04.4 LTS","agentCount":1,"heapMemoryUsed":352245016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":11,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":112,"systemMemoryTotal":8206192640,"systemMemoryFree":700325888,"systemMemoryUsed":1294708,"systemMemoryVirtualSize":8336060416,"logInfo":"/var/log/cloudstack/management/management-server.log using: 34M\non disk /dev/nvme0n1p2 mounted on / (11% full)","systemTotalCpuCycles":13586.599,"systemLoadAverages":[0.44,0.49,0.58],"systemCyclesUsage":[352458,122206,5591261],"dbLocal":true,"usageLocal":false,"systemBootTime":"Jul 25, 2024, 10:38:22 AM","kernelVersion":"5.15.0-116-generic"}
2024-07-25 12:46:16,581 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Executing ClusterServicePdu with service URL: https://127.0.0.1:9090/clusterservice
2024-07-25 12:46:16,585 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Exception from : https://127.0.0.1:9090/clusterservice, method : null, exception :
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <127.0.0.1> doesn't match any of the subject alternative names: [fe80:0:0:0:da73:d3bc:cbd3:976e, 192.168.1.122, hcimg-HP-ProBook-440-G5, cloudstack.internal]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at com.cloud.cluster.ClusterServiceServletImpl.executePostMethod(ClusterServiceServletImpl.java:142)
at com.cloud.cluster.ClusterServiceServletImpl.execute(ClusterServiceServletImpl.java:105)
at com.cloud.cluster.ClusterManagerImpl.onSendingClusterPdu(ClusterManagerImpl.java:276)
at com.cloud.cluster.ClusterManagerImpl$1.runInContext(ClusterManagerImpl.java:236)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-25 12:46:16,588 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Cluster PDU 233349941812470 -> 233349941812470 completed. time: 6ms. agent: 0, pdu seq: 24, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"fc5923e9-3eac-46b0-93b3-3edbf9b595fb","collectionTime":"Jul 25, 2024, 12:46:16 PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":953155584,"freeJvmMemoryBytes":602002456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1409455,"jvmStartTime":1721890367082,"availableProcessors":8,"loadAverage":0.435546875,"totalInit":137691136,"totalUsed":540842440,"totalCommitted":1147838464,"pid":15681,"jvmName":"15681@hcimg-HP-ProBook-440-G5","jvmVendor":"Ubuntu","jvmVersion":"11.0.23+9-post-Ubuntu-1ubuntu122.04.1","osDistribution":"Ubuntu 22.04.4 LTS","agentCount":1,"heapMemoryUsed":352245016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":11,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":112,"systemMemoryTotal":8206192640,"systemMemoryFree":700325888,"systemMemoryUsed":1294708,"systemMemoryVirtualSize":8336060416,"logInfo":"/var/log/cloudstack/management/management-server.log using: 34M\non disk /dev/nvme0n1p2 mounted on / (11% full)","systemTotalCpuCycles":13586.599,"systemLoadAverages":[0.44,0.49,0.58],"systemCyclesUsage":[352458,122206,5591261],"dbLocal":true,"usageLocal":false,"systemBootTime":"Jul 25, 2024, 10:38:22 AM","kernelVersion":"5.15.0-116-generic"}
Agent log -
Beta Was this translation helpful? Give feedback.
All reactions