BuiltinByteString literals aren't UTF8 encoded. #6655

Unisay · 2024-11-12T16:18:17Z

BuiltinByteString literals are utf-8 decoded in the plugin.
Haskell.IsString BuiltinByteString instance delegates to the Haskell ByteString instance.
Unit tests to verify that bytes in range from 0 to 255 (\x00 - \xFF) found in string literals are passed to the BuiltinByteString without the UTF8 conversion.

...lc/evaluation/builtin/semantics/bls12_381_G1_scalarMul/mul-neg-one/mul-neg-one.uplc.expected

Unisay · 2024-11-12T17:05:24Z

plutus-tx-plugin/src/PlutusTx/Compiler/Expr.hs

@@ -699,13 +732,21 @@ compileExpr e = traceCompilation 2 ("Compiling expr:" GHC.<+> GHC.ppr e) $ do
    _                             -> throwPlain $ CompilationError "No info for Pair builtin"

  -- TODO: Maybe share this to avoid repeated lookups. Probably cheap, though.
-  (stringTyName, sbsName) <- case (Map.lookup ''Builtins.BuiltinString nameInfo, Map.lookup 'Builtins.stringToBuiltinString nameInfo) of
+  (stringTyName, sbsName) <-
+    case


Wrap long line

...t-cases/uplc/evaluation/builtin/semantics/replicateByte/case-07/case-07.uplc.budget.expected

Unisay · 2024-11-12T20:20:37Z

plutus-tx-plugin/src/PlutusTx/Compiler/Expr.hs

-    | let name = GHC.getName n
-    , name == GHC.unpackCStringName || name == GHC.unpackCStringUtf8Name ->
-      stringExprContent expr
+data StringExprContentAs = AsBytes | AsText


The stringExprContent function is used to extract bytes for the BuiltinByteString as well as for the BuiltinString literals;
I have added this datatype as its parameter to distinguish between these scenarios as we want to utf-8 decode BuiltinByteString literals only.

Unisay · 2024-11-12T20:24:05Z

plutus-tx-plugin/src/PlutusTx/Compiler/Expr.hs

+This isn't a full UTF-8 decoder: it only decodes the subset of UTF-8 that
+is expected to be found in bytestring literals: 0x00 - 0xFF
+-}
+fromUtf8 :: [Word8] -> Maybe [Word8]


I tried using Data.Text.Encoding.decodeUtf8 and it didn't work: the error was saying about invalid utf-8 bytes. Maybe the implementation in GHC is not compatible with the one in Text (Currently GHC ships with at least five UTF-8 implementations)
It turned out that decoding rules are not super complex so I have implemented them below.

There is a unit test to prove it works correctly on bytes 00 - FF

Unisay · 2024-11-12T20:27:51Z

plutus-tx-plugin/src/PlutusTx/Compiler/Expr.hs

-  (bsTyName, sbbsName) <- case (Map.lookup ''Builtins.BuiltinByteString nameInfo, Map.lookup 'Builtins.stringToBuiltinByteString nameInfo) of
-    (Just t1, Just t2) -> pure (GHC.getName t1, GHC.getName t2)
-    _                  -> throwPlain $ CompilationError "No info for ByteString builtin"
+  (builtinByteStringTyName, sbbsName) <-


Wrap another long line

Unisay · 2024-11-12T20:29:11Z

plutus-tx-plugin/src/PlutusTx/Compiler/Expr.hs

+            Just tc ->
+              if
+                | GHC.getName tc == builtinByteStringTyName ->
+                    case stringExprContent AsBytes (strip content) of


Extracting content as bytes undoes the UTF8 encoding for BuiltinByteString

Unisay · 2024-11-12T20:29:39Z

plutus-tx-plugin/src/PlutusTx/Compiler/Expr.hs

+                      Just bs ->
+                        pure $ PIR.Constant annMayInline $ PLC.someValue bs
+                | GHC.getName tc == stringTyName ->
+                    case stringExprContent AsText (strip content) of


Extracting content as text leaves original UTF8 encoding for BuiltinString

Unisay · 2024-11-12T20:30:23Z

plutus-tx-plugin/src/PlutusTx/Compiler/Expr.hs

-          throwPlain $
-            CompilationError $
-              "Text literal with invalid UTF-8 content: " <> (T.pack $ show err)
+    (strip -> GHC.Var n) `GHC.App` (strip -> stringExprContent AsText -> Just bs)


Wrapped long line

Unisay · 2024-11-12T20:34:02Z

plutus-tx-plugin/test/AssocMap/Spec.hs

@@ -91,21 +91,23 @@ map2 =
        \n ->
          let m1 =
                Data.AssocMap.unsafeFromList
-                  [ (n PlutusTx.+ 1, "one")


The fact that this test relies on UTF-8 encoding for BuiltinByteStrings is accidental

I've re-worked the test to use Integers instead of BuiltinByteStrings.

Unisay self-assigned this Nov 12, 2024

Unisay temporarily deployed to github-pages November 12, 2024 16:18 — with GitHub Actions Inactive

Unisay commented Nov 12, 2024

View reviewed changes

...lc/evaluation/builtin/semantics/bls12_381_G1_scalarMul/mul-neg-one/mul-neg-one.uplc.expected Outdated Show resolved Hide resolved

Unisay commented Nov 12, 2024

View reviewed changes

kwxm reviewed Nov 12, 2024

View reviewed changes

...t-cases/uplc/evaluation/builtin/semantics/replicateByte/case-07/case-07.uplc.budget.expected Outdated Show resolved Hide resolved

Unisay commented Nov 12, 2024

View reviewed changes

Unisay force-pushed the yura/builtin-byte-string-literals branch from 0f08f16 to 439f7e5 Compare November 13, 2024 10:54

Unisay temporarily deployed to github-pages November 13, 2024 10:54 — with GitHub Actions Inactive

Unisay force-pushed the yura/builtin-byte-string-literals branch from 439f7e5 to b4f1bb2 Compare November 13, 2024 10:57

Unisay temporarily deployed to github-pages November 13, 2024 10:57 — with GitHub Actions Inactive

Unisay temporarily deployed to github-pages November 13, 2024 11:09 — with GitHub Actions Inactive

Unisay force-pushed the yura/builtin-byte-string-literals branch from 8c7865f to 7cd0c62 Compare November 13, 2024 11:20

Unisay temporarily deployed to github-pages November 13, 2024 11:20 — with GitHub Actions Inactive

Unisay requested review from zliu41 and a team November 13, 2024 11:21

Unisay marked this pull request as ready for review November 13, 2024 11:21

Unisay force-pushed the yura/builtin-byte-string-literals branch from 7cd0c62 to 03011f0 Compare November 13, 2024 11:23

Unisay temporarily deployed to github-pages November 13, 2024 11:23 — with GitHub Actions Inactive

Unisay force-pushed the yura/builtin-byte-string-literals branch from 03011f0 to b718e0b Compare November 13, 2024 12:20

Unisay temporarily deployed to github-pages November 13, 2024 12:20 — with GitHub Actions Inactive

BuiltinByteString literals aren't UTF8 encoded.

5ff83c0

Unisay force-pushed the yura/builtin-byte-string-literals branch from b718e0b to 5ff83c0 Compare November 13, 2024 13:29

Unisay temporarily deployed to github-pages November 13, 2024 13:29 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BuiltinByteString literals aren't UTF8 encoded. #6655

BuiltinByteString literals aren't UTF8 encoded. #6655

Unisay commented Nov 12, 2024 •

edited

Loading

Unisay Nov 12, 2024

Unisay Nov 12, 2024

Unisay Nov 12, 2024 •

edited

Loading

Unisay Nov 12, 2024 •

edited

Loading

Unisay Nov 12, 2024

Unisay Nov 12, 2024

Unisay Nov 12, 2024

Unisay Nov 12, 2024

Unisay Nov 12, 2024

Unisay Nov 13, 2024

BuiltinByteString literals aren't UTF8 encoded. #6655

Are you sure you want to change the base?

BuiltinByteString literals aren't UTF8 encoded. #6655

Conversation

Unisay commented Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Unisay Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Unisay Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Unisay commented Nov 12, 2024 •

edited

Loading

Unisay Nov 12, 2024 •

edited

Loading

Unisay Nov 12, 2024 •

edited

Loading