small

RWKV · Aug 28, 2023 · e452213 · e452213
1 parent fe597e3
commit e452213
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/notebook/experiment/rwkv-x-exp/v5-small-model/v5-L6-ctx4k-models-part1.ipynb b/notebook/experiment/rwkv-x-exp/v5-small-model/v5-L6-ctx4k-models-part1.ipynb
@@ -8,7 +8,7 @@
     "# RWKV v5 Small model experiment\n",
     "\n",
     "- 6 layers\n",
-    "- 2048 embedding size\n",
+    "- 64-1024 embedding size\n",
     "\n",
     "A series of small model training expriments, to see \"how small can we go\" for v5\n",
     "\n",