Fix ROCm 6.x training error #42
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
|
Ohhh, this is awesome! 🥳 Thanks for testing and for the fix. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After this fix, AMD GPU build should start working via the new ROCm 6.x docker environment 🎉.
The following is the hardware setup and initial test result.
root@5fb916383b63:/code/build# time ./opensplat /data/banana -n 2000 Using CUDA Reading 14241 points Loading /data/banana/images/frame_00001.JPG Loading /data/banana/images/frame_00002.JPG Loading /data/banana/images/frame_00003.JPG Loading /data/banana/images/frame_00004.JPG Loading /data/banana/images/frame_00005.JPG Loading /data/banana/images/frame_00006.JPG Loading /data/banana/images/frame_00008.JPG Loading /data/banana/images/frame_00009.JPG Loading /data/banana/images/frame_00010.JPG Loading /data/banana/images/frame_00011.JPG Loading /data/banana/images/frame_00013.JPG Loading /data/banana/images/frame_00014.JPG Loading /data/banana/images/frame_00015.JPG Loading /data/banana/images/frame_00016.JPG Step 10: 0.208075 Step 20: 0.237863 Step 30: 0.181474 Step 40: 0.209521 Step 50: 0.160717 Step 60: 0.205582 Step 70: 0.147323 Step 80: 0.150524 Step 90: 0.147064 Step 100: 0.144563 Step 110: 0.125039 Step 120: 0.142733 Step 130: 0.189102 Step 140: 0.136359 Step 150: 0.138213 Step 160: 0.136127 Step 170: 0.178151 Step 180: 0.13395 Step 190: 0.142111 Step 200: 0.131241 Step 210: 0.178902 Step 220: 0.105384 Step 230: 0.137437 Step 240: 0.134837 Step 250: 0.128858 Step 260: 0.120589 Step 270: 0.174268 Step 280: 0.12046 Step 290: 0.102276 Step 300: 0.119286 Step 310: 0.129531 Step 320: 0.124861 Step 330: 0.123276 Step 340: 0.121402 Step 350: 0.115895 Step 360: 0.159928 Step 370: 0.0886419 Step 380: 0.109183 Step 390: 0.0915482 Step 400: 0.109655 Step 410: 0.111456 Step 420: 0.110098 Step 430: 0.0880052 Step 440: 0.0919362 Step 450: 0.0903232 Step 460: 0.108424 Step 470: 0.124387 Step 480: 0.10601 Step 490: 0.16722 Step 500: 0.0900435 Step 510: 0.117083 Step 520: 0.163578 Step 530: 0.1069 Step 540: 0.118836 Step 550: 0.105309 Step 560: 0.156448 Step 570: 0.164224 Step 580: 0.108291 Step 590: 0.153091 Step 600: 0.121695 Added 9591 gaussians, new count 23832 Culled 9895 gaussians, remaining 13937 Step 610: 0.166405 Step 620: 0.110598 Step 630: 0.125043 Step 640: 0.084933 Step 650: 0.0845598 Step 660: 0.119641 Step 670: 0.117407 Step 680: 0.117641 Step 690: 0.101347 Step 700: 0.150917 Added 15037 gaussians, new count 28974 Culled 5439 gaussians, remaining 23535 Step 710: 0.0916314 Step 720: 0.154107 Step 730: 0.124596 Step 740: 0.103116 Step 750: 0.120372 Step 760: 0.0796096 Step 770: 0.117717 Step 780: 0.115493 Step 790: 0.0769396 Step 800: 0.0984292 Added 22787 gaussians, new count 46322 Culled 8011 gaussians, remaining 38311 Step 810: 0.101665 Step 820: 0.118277 Step 830: 0.153499 Step 840: 0.0754875 Step 850: 0.112758 Step 860: 0.0972251 Step 870: 0.0860925 Step 880: 0.11283 Step 890: 0.108513 Step 900: 0.0681005 Added 31275 gaussians, new count 69586 Culled 10132 gaussians, remaining 59454 Step 910: 0.14729 Step 920: 0.0950705 Step 930: 0.148599 Step 940: 0.105241 Step 950: 0.106126 Step 960: 0.145349 Step 970: 0.106236 Step 980: 0.0896305 Step 990: 0.0987537 Step 1000: 0.0637969 Added 40189 gaussians, new count 99643 Culled 12016 gaussians, remaining 87627 Step 1010: 0.0732109 Step 1020: 0.0837388 Step 1030: 0.0838937 Step 1040: 0.103043 Step 1050: 0.0619884 Step 1060: 0.0830906 Step 1070: 0.122514 Step 1080: 0.0703483 Step 1090: 0.126708 Step 1100: 0.0943512 Added 48113 gaussians, new count 135740 Culled 13658 gaussians, remaining 122082 Step 1110: 0.141307 Step 1120: 0.0895955 Step 1130: 0.0609604 Step 1140: 0.0559767 Step 1150: 0.0964497 Step 1160: 0.0889039 Step 1170: 0.0929496 Step 1180: 0.0889424 Step 1190: 0.0522798 Step 1200: 0.06965 Added 55532 gaussians, new count 177614 Culled 14556 gaussians, remaining 163058 Step 1210: 0.138581 Step 1220: 0.115415 Step 1230: 0.065682 Step 1240: 0.116656 Step 1250: 0.113661 Step 1260: 0.0887083 Step 1270: 0.0662218 Step 1280: 0.0500107 Step 1290: 0.121573 Step 1300: 0.112594 Added 61540 gaussians, new count 224598 Culled 15636 gaussians, remaining 208962 Step 1310: 0.0591685 Step 1320: 0.0816074 Step 1330: 0.0896751 Step 1340: 0.0858046 Step 1350: 0.048018 Step 1360: 0.0954068 Step 1370: 0.0753745 Step 1380: 0.0552086 Step 1390: 0.0540024 Step 1400: 0.0960227 Added 64682 gaussians, new count 273644 Culled 16057 gaussians, remaining 257587 Step 1410: 0.0906328 Step 1420: 0.0786731 Step 1430: 0.0466605 Step 1440: 0.0808401 Step 1450: 0.0787142 Step 1460: 0.0566929 Step 1470: 0.0433018 Step 1480: 0.0594259 Step 1490: 0.0671842 Step 1500: 0.0391344 Added 67010 gaussians, new count 324597 Culled 16008 gaussians, remaining 308589 Step 1510: 0.117637 Step 1520: 0.0458945 Step 1530: 0.106889 Step 1540: 0.0520811 Step 1550: 0.0548199 Step 1560: 0.0660458 Step 1570: 0.0406176 Step 1580: 0.0684525 Step 1590: 0.054903 Step 1600: 0.0789318 Added 66922 gaussians, new count 375511 Culled 15448 gaussians, remaining 360063 Step 1610: 0.0779538 Step 1620: 0.0453232 Step 1630: 0.100514 Step 1640: 0.0383056 Step 1650: 0.042757 Step 1660: 0.0451564 Step 1670: 0.0513167 Step 1680: 0.0797595 Step 1690: 0.096207 Step 1700: 0.0456395 Added 67374 gaussians, new count 427437 Culled 15828 gaussians, remaining 411609 Step 1710: 0.04383 Step 1720: 0.0661522 Step 1730: 0.0685059 Step 1740: 0.0884331 Step 1750: 0.0449189 Step 1760: 0.0433282 Step 1770: 0.0476067 Step 1780: 0.0420889 Step 1790: 0.0565312 Step 1800: 0.0900256 Added 66777 gaussians, new count 478386 Culled 15137 gaussians, remaining 463249 Step 1810: 0.065665 Step 1820: 0.0536985 Step 1830: 0.0370926 Step 1840: 0.0438131 Step 1850: 0.0828206 Step 1860: 0.0422622 Step 1870: 0.0545112 Step 1880: 0.0589518 Step 1890: 0.0361636 Step 1900: 0.0545315 Added 65206 gaussians, new count 528455 Culled 14989 gaussians, remaining 513466 Step 1910: 0.0546802 Step 1920: 0.0598722 Step 1930: 0.0569776 Step 1940: 0.053701 Step 1950: 0.0508809 Step 1960: 0.0445479 Step 1970: 0.050296 Step 1980: 0.0339478 Step 1990: 0.0498263 Step 2000: 0.0350233 Added 63772 gaussians, new count 577238 Culled 14569 gaussians, remaining 562669 Wrote splat.ply real 1m12.359s user 1m14.793s sys 0m4.042sGPU usage:
First AMD GPU based 3DGS:
