Add JSON5 support #655

ks-yim · 2021-12-07T16:53:38Z

Motivation:

Support JSON5 #538

Modifications:

Server & ArmeriaCentralDogma client JSON5 support
- getFile() as Entry<JsonNode> but can still get the raw content via #contentAsText()
- watchFile() as Entry<JsonNode> but can still get the raw content via #contentAsText()
- getFile() & watchFile() with JSON_PATH
- mergeFiles()
Update Admin UI's ace editor version to 1.4.13
- To support JSON5 syntax highlighting

Results:

Closes Support JSON5 #538

CLAassistant · 2021-12-07T16:53:51Z

All committers have signed the CLA.

minwoox · 2021-12-08T02:57:18Z

This looks awesome. 😆

codecov · 2021-12-10T07:50:56Z

Codecov Report

Attention: Patch coverage is 94.39252% with 6 lines in your changes missing coverage. Please review.

Project coverage is 70.34%. Comparing base (0ed7a9a) to head (849bdb2).
Report is 251 commits behind head on main.

Files with missing lines	Patch %	Lines
...ntraldogma/client/armeria/ArmeriaCentralDogma.java	80.00%	2 Missing ⚠️
...java/com/linecorp/centraldogma/internal/Json5.java	88.88%	2 Missing ⚠️
.../java/com/linecorp/centraldogma/common/Change.java	91.66%	1 Missing ⚠️
...n/java/com/linecorp/centraldogma/common/Entry.java	92.85%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #655      +/-   ##
============================================
+ Coverage     70.04%   70.34%   +0.29%     
- Complexity     3415     3458      +43     
============================================
  Files           349      350       +1     
  Lines         13482    13544      +62     
  Branches       1454     1466      +12     
============================================
+ Hits           9444     9528      +84     
+ Misses         3153     3135      -18     
+ Partials        885      881       -4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

minwoox

Sorry for the late review. 😅
Left some suggestions. 😉

...java-armeria/src/main/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogma.java

common/src/main/java/com/linecorp/centraldogma/internal/Jackson.java

ikhoon · 2021-12-20T08:28:48Z

common/src/main/java/com/linecorp/centraldogma/common/Change.java

@@ -65,10 +66,20 @@
    static Change<String> ofTextUpsert(String path, String text) {
        requireNonNull(text, "text");
        validateFilePath(path, "path");
-        if (EntryType.guessFromPath(path) == EntryType.JSON) {


It seems not to make sense to use ofTextUpsert for not JSON but JSON5.
How about modifying Change.ofJsonUpsert(String path, String jsonText) to take JSON5?

static Change<JsonNode> ofJsonUpsert(String path, String jsonText) { final JsonNode jsonNode = ...; return new DefaultChange<>(path, ChangeType.UPSERT_JSON, jsonNode, jsonText); }

I think we can give jsonText a higher priority to serialize the data.

Just wanted to point out we may need to use contentAsText conditionally for serialization right before sending on wire. It's not elegant, but I think this is worth it considering the gain 👍

ArmeriaCentralDogma.java

private static ArrayNode toJson(Iterable<? extends Change<?>> changes) { ... if (c.path().endsWith(".json5")) { changeNode.put("content", c.contentAsText()); } else { final Class<?> contentType = c.type().contentType(); if (contentType == JsonNode.class) { changeNode.set("content", (JsonNode) c.content()); } else if (contentType == String.class) { changeNode.put("content", (String) c.content()); } } ... }

jrhee17

The PR overall looks really nice! Left some random ideas/comments 😄 Looking forward to this feature a lot 👍

jrhee17 · 2021-12-29T13:15:41Z

common/src/main/java/com/linecorp/centraldogma/common/Change.java

@@ -65,10 +66,20 @@
    static Change<String> ofTextUpsert(String path, String text) {
        requireNonNull(text, "text");
        validateFilePath(path, "path");
-        if (EntryType.guessFromPath(path) == EntryType.JSON) {


Just wanted to point out we may need to use contentAsText conditionally for serialization right before sending on wire. It's not elegant, but I think this is worth it considering the gain 👍

ArmeriaCentralDogma.java

private static ArrayNode toJson(Iterable<? extends Change<?>> changes) { ... if (c.path().endsWith(".json5")) { changeNode.put("content", c.contentAsText()); } else { final Class<?> contentType = c.type().contentType(); if (contentType == JsonNode.class) { changeNode.set("content", (JsonNode) c.content()); } else if (contentType == String.class) { changeNode.put("content", (String) c.content()); } } ... }

...java-armeria/src/main/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogma.java

jrhee17 · 2021-12-29T13:59:32Z

server/src/test/java/com/linecorp/centraldogma/server/internal/api/ContentServiceV1Test.java

@@ -633,6 +671,72 @@ void editFileWithTextPatch() throws IOException {
                    .isEqualTo("text in some file.\n");
        }

+        @Test
+        void editJson5FileWithTextPatch() throws IOException {


For now, I think text patch is fine but this feature might become awkward if we want to prettify/normalize our json5 input. (Also, I don't think anyone would want to text patch json files..)

Ideally, I think json5 should support jsonPatch instead, but we will need a parser which I don't think is very trivial.

I'm leaning more towards not allowing text patches for json5 (at least for the client side) and considering later if we receive a feature request/or we can mark it as unstable somehow)

@jrhee17 I was thinking of supporting both APPLY_JSON_PATCH and APPLY_TEXT_PATCH for JSON5 file but if updated via APPLY_JSON_PATCH it loses all JSON5 features and degraded to ordinary JSON.
I am not sure if it is the right direction, though...

I will ping you again when I revise the PR and hope to get some advice from there..!

ks-yim · 2022-01-04T01:51:17Z

...-armeria/src/test/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogmaTest.java

    }

    @Test
-    void pushMirrorsJsonFileToMetaRepository() throws UnknownHostException {


Moved to it/MetadataPushTest.java.

ks-yim · 2022-01-04T01:51:27Z

...-armeria/src/test/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogmaTest.java

@@ -57,33 +78,94 @@ void testEncodePathPattern() {
    }

    @Test
-    void pushFileToMetaRepositoryShouldFail() throws UnknownHostException {


Moved to it/MetadataPushTest.java.

ks-yim · 2022-01-04T06:43:50Z

@jrhee17 @ikhoon @minwoox I've re-written the PR as per your advices, PTAL 🙏.

Use EntryType.JSON for JSON5
Use dedicated mapper for JSON5
Give higher priority to contentAsText property when transmitting JSON5 content of Entry or Change over the wire.

Sorry that it's delayed. 😓

minwoox

You have beautifully implemented it. 😄
Left some minor comments.

minwoox · 2022-01-04T13:16:45Z

...java-armeria/src/main/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogma.java

+            try {
+                return unsafeCast(Entry.ofJson(revision, entryPath, json5Text));
+            } catch (JsonParseException e) {
+                throw new CentralDogmaException("failed to parse JSON5 content as JSON: " + json5Text);


nit:

Suggested change

throw new CentralDogmaException("failed to parse JSON5 content as JSON: " + json5Text);

throw new CentralDogmaException("failed to parse JSON5 content as JSON: " + json5Text, e);

minwoox · 2022-01-04T13:23:04Z

...java-armeria/src/main/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogma.java

@@ -1015,7 +1016,9 @@ private static ArrayNode toJson(Iterable<? extends Change<?>> changes) {
            changeNode.put("path", c.path());
            changeNode.put("type", c.type().name());
            final Class<?> contentType = c.type().contentType();
-            if (contentType == JsonNode.class) {
+            if (maybeJson5(c.path())) {


Could it be just isJson5?

common/src/main/java/com/linecorp/centraldogma/common/Change.java

minwoox · 2022-01-04T13:28:00Z

common/src/main/java/com/linecorp/centraldogma/common/Change.java

+            if (maybeJson5(path)) {
+                jsonNode = Json5.readTree(jsonText);
+                return new DefaultChange<>(path, ChangeType.UPSERT_JSON, jsonNode, jsonText);
+            } else {
+                jsonNode = Jackson.readTree(jsonText);
+                return new DefaultChange<>(path, ChangeType.UPSERT_JSON, jsonNode);
+            }


It seems like we can inline the variable and return early:

Suggested change

if (maybeJson5(path)) {

jsonNode = Json5.readTree(jsonText);

return new DefaultChange<>(path, ChangeType.UPSERT_JSON, jsonNode, jsonText);

} else {

jsonNode = Jackson.readTree(jsonText);

return new DefaultChange<>(path, ChangeType.UPSERT_JSON, jsonNode);

}

if (maybeJson5(path)) {

return new DefaultChange<>(path, ChangeType.UPSERT_JSON, Json5.readTree(jsonText), jsonText);

}

return new DefaultChange<>(path, ChangeType.UPSERT_JSON, Jackson.readTree(jsonText));

minwoox · 2022-01-04T13:32:41Z

common/src/main/java/com/linecorp/centraldogma/common/Entry.java

@@ -69,7 +72,22 @@
     */
    public static Entry<JsonNode> ofJson(Revision revision, String path, String content)
            throws JsonParseException {
-        return ofJson(revision, path, Jackson.readTree(content));
+        return maybeJson5(path) ? new Entry<>(revision, path, EntryType.JSON, Json5.readTree(content), content)


Let's do requireNonNull(content, "content") before we use it. 😉

minwoox · 2022-01-04T13:48:28Z

it/src/test/java/com/linecorp/centraldogma/it/MetaRepositoryPushTest.java

+import com.linecorp.centraldogma.common.Revision;
+import com.linecorp.centraldogma.testing.junit.CentralDogmaExtension;
+
+public class MetaRepositoryPushTest {


nit:

Suggested change

public class MetaRepositoryPushTest {

class MetaRepositoryPushTest {

minwoox · 2022-01-04T13:50:38Z

it/src/test/java/com/linecorp/centraldogma/it/WatchTest.java

+
+            // Make change to a relevant file.
+            final PushResult result = client.push(
+                    dogma.project(), dogma.repo1(), Revision.HEAD, "Edit test1.json5",


nit:

Suggested change

dogma.project(), dogma.repo1(), Revision.HEAD, "Edit test1.json5",

dogma.project(), dogma.repo1(), Revision.HEAD, "Add test1.json5",

because we are adding it? 😄
Could you also add a test case that only changes the comments of a JSON5 file?
I think we should specially treat the case not to notify the watchers.

I think we should specially treat the case not to notify the watchers.

This is a nice idea!!
But there seems to be no easy way to detect if the actual JSON content is changed before notifying watchers.

Maybe we can add the following code at GitRepository.java:L1497 (GitRepository#notifyWatchers) but it doesn't seem to be a right choice (it looks more costly than the savings we get from not notifying watchers).

case MODIFY: if (isJson5(entry.getOldPath())) { try (ObjectReader reader = jGitRepository.newObjectReader()) { final JsonNode oldJsonNode = Json5.readTree(reader.open(entry.getOldId().toObjectId()).getBytes()); final JsonNode newJsonNode = Json5.readTree(reader.open(entry.getNewId().toObjectId()).getBytes()); if (oldJsonNode == newJsonNode) { break; } } catch (Exception e) { // Do nothing. } } /* fall-thru */

Oops, I missed this comment. 😄
I thought we could just manipulate here:
https://github.com/line/centraldogma/blob/master/server/src/main/java/com/linecorp/centraldogma/server/storage/repository/RepositoryUtil.java#L182

If it gets too complicated, let's make a separate PR for that. 😄

RepositoryUtil.java#L182

Seems to be a good place to put this check..!
I didn't know that it eventually gets the old content from the repo..!

Let me work on it in this PR.

And it seems we don't even need a change. It already works in a way you wanted it to be.
I've just added a test case to prove it..!

I didn't know that it's going to work. That's awesome. 😄

minwoox · 2022-01-04T14:02:56Z

server/src/main/java/com/linecorp/centraldogma/server/internal/api/DtoConverter.java

+    private static <T> EntryDto convert(Repository repository, Revision revision, String path,
+                                        EntryType type, @Nullable T content) {


We don't need <T> anymore throughout this class :

Suggested change

private static <T> EntryDto convert(Repository repository, Revision revision, String path,

EntryType type, @Nullable T content) {

private static EntryDto convert(Repository repository, Revision revision, String path,

EntryType type, @Nullable Object content) {

minwoox · 2022-01-04T14:06:19Z

server/src/main/java/com/linecorp/centraldogma/server/internal/api/DtoConverter.java

        requireNonNull(entry, "entry");
        if (withContent && entry.hasContent()) {
-            return convert(repository, revision, entry.path(), entry.type(), entry.content());
+            // TODO(ks-yim): Not sure DtoConverter having business logic and it looks hacky to receive
+            //   queryType to handle JSON_PATH query for JSON5 files.


I don't have a better solution. 😅
Let's leave it as a known issue. 👍

Not sure of 'DtoConverter' having business logic and it looks hacky to receive queryType to handle JSON_PATH query for JSON5 files.

Agreed. QueryType and isJson5() don't match with DTO. How about taking a Function rather than adding a specific logic here?

public static EntryDto convert(Repository repository, Revision revision, Entry<?> entry, boolean withContent, Function<Entry, ?> contentConverter) { return convert(repository, revision, entry.path(), entry.type(), contentCoverter.apply(entry)); }

minwoox · 2022-01-04T14:10:54Z

server/src/main/java/com/linecorp/centraldogma/server/internal/mirror/GitMirror.java

-                            final JsonNode jsonNode = Jackson.readTree(content);
-                            changes.putIfAbsent(localPath, Change.ofJsonUpsert(localPath, jsonNode));
+                            final String jsonText = new String(content, StandardCharsets.UTF_8);
+                            changes.putIfAbsent(localPath, Change.ofJsonUpsert(localPath, jsonText));


How about also adding the variant method that takes byte[] so that we can remove String conversion?

static Change<JsonNode> ofJsonUpsert(String path, byte[] jsonBytes) {...}

ks-yim · 2022-01-06T15:18:25Z

@minwoox I've resolved most of your comments except the feature of not notifying watchers for any changes that does not affect the actual content of JSON5.

And I added a validation before committing a patch on JSON5 files (since they are updated via text patch..).

Plus, I found a weird bug with Jackson's ALLOW_SINGLE_QUOTES features when it deserializes JSON5 from a byte-array source.

I've summarized the issue at Json5#readTree(byte[]) method...:

// String conversion is a workaround for Jackson bug with 'ALLOW_SINGLE_QUOTES' feature
// when it deserializes JSON5 from byte array source.
// If double quotes are used within a single quoted string, it either raises JsonParseException or
// removes double quotes while deserializing.
// e.g. 'I can use "double quotes" here' deserialized into "I can use double quotes here"
//      'I can use double quotes "here"' raises JsonParseException.

I will report the issue to Jackson but I think we should always convert byte[] into String before deserializing JSON5 content as of now.

minwoox · 2022-01-07T01:35:28Z

And I added a validation before committing a patch on JSON5 files (since they are updated via text patch..).

👍

I will report the issue to Jackson but I think we should always convert byte[] into String before deserializing JSON5 content as of now.

Oops, I didn't notice that. 😅 Yeah, let's report it to the upstream and use String for now.

except the feature of not notifying watchers for any changes that does not affect the actual content of JSON5.

Could you tell me more about it, please?

jrhee17

Overall looks good! 🙇 Left a couple more comments.

jrhee17 · 2022-01-11T08:26:51Z

common/src/main/java/com/linecorp/centraldogma/common/DefaultChange.java

@@ -106,6 +110,7 @@

        this.path = path;
        this.content = content;
+        this.contentAsText = contentAsText;


I wonder if we should also add json5 specific logic when performing json deserialization

centraldogma/common/src/main/java/com/linecorp/centraldogma/common/DefaultChange.java

Lines 38 to 40 in 3642633

static DefaultChange<?> deserialize(@JsonProperty("type") ChangeType type,

@JsonProperty("path") String path,

@JsonProperty("content") @Nullable JsonNode content) {

which may be done when replicating logs

centraldogma/server/src/main/java/com/linecorp/centraldogma/server/internal/replication/ZooKeeperCommandExecutor.java

Line 1034 in 3642633

final ReplicationLog<?> log = Jackson.readValue(bytes, ReplicationLog.class);

Hmm.. thanks for pointing this out. I think we should handle the case that Change#ofJsonUpsert is used to add a new JSON5 file(ChangeType.ADD) as it relies on contentAsText() which isn't serialized into replication log.

No need to care about Change#ofJsonUpsert to modify the file content(ChangeType.MODIFY) since it internally converted into Change.ofTextPatch.

I think we should have Change#contentAsText in the serialized content to make it work (or a simple workaround would be internally converting Change#ofJsonUpsert to Change#ofTextUpsert like we did for ChangeType.MODIFY).

Sorry about the late reply 😅

I wonder if it's difficult to tweak behavior so that serde happens in the same way as we've been doing so far (using Change#ofJsonUpsert(String path, String jsonText)).

I think we should have Change#contentAsText in the serialized content to make it work

I personally think we don't want contentAsText for all file types. If we want to selectively include contentAsText depending on the file path, I also think the increased complexity would be similar to the other idea.

Let me know if you feel differently though 😅

I wonder if it's difficult to tweak behavior so that serde happens in the same way as we've been doing so far

The type of content in Change#ofJsonUpsert is JsonNode because of the decision we've made for JSON5 entry type (use JsonNode over String).

Maybe we can do the tweak that Change to a JSON5 file have TextNode as its content and do if-else in all places where change gets applied, but this also seems to me extremely unmanageable.

I agree that we should avoid unnecessary complexity, but either way(tweaking serde vs tweak `Change) is not very appealing 😅..

The 3rd(and the most simple) option would be converting Change#ofJsonUpsert request to Change#ofTextUpsert internally like we already did for ChangeType.MODIFY.

~~It turns out that the 3rd option will never work since the serializing and sending replication log happens earlier than the point that we can do conversion.~~

Had a chat with @minwoox and @jrhee17 about this and I was advised to put denormalized JSON5 Change to replication logs to preserve its original content.

I tried this approach only to find there's no easy way to pass the original JSON5 content to replication log even with this. NormalizingPushCommand#changes gives the original JSON5 content in it, but once it gets JSON-serialized for replication log it loses its JSON5 content since contentAsText is ignored in the serialization process.

Meanwhile, it turned out that 3rd option that I mentioned above works just fine.
It works by normalizing JSON5 Change#ofJsonUpsert to Change#ofTextUpsert and then putting the normalized change to replication log (refer to GitRepository.java:L803-804).

jrhee17 · 2022-01-11T10:01:13Z

server/src/main/java/com/linecorp/centraldogma/server/internal/api/DtoConverter.java

+            // TODO(ks-yim): Not sure of 'DtoConverter' having business logic and it looks hacky to receive
+            //   queryType to handle JSON_PATH query for JSON5 files.
+            return convert(repository, revision, entry.path(), entry.type(),
+                           isJson5(entry) && queryType != QueryType.JSON_PATH ? entry.contentAsText()


Q) Did we agree to move this functionality to api v2?
Since type == JSON and content is json5, I feel like this will break backwards-compatibility

Haven't talked about V2 part, yet.

Since type == JSON and content is json5, I feel like this will break backwards-compatibility

I don't think there will be a breaking change with ArmeriaCentralDogma client unless users are using deprecated Query.identity(...) because JSON5 can now only be queried via Query.ofText(..) and this PR still supports it.

The only breaking change that I can think of is that users won't be able to upsert JSON5 text with Change.ofTextUpsert(...) after this PR on anymore, though I doubt there will be many users who do this.

But I am not sure if this PR won't break admin UI and its APIs somehow.
Let me check this out.

Let me be more specific since I'm starting to doubt myself as well 😅 I've run the test ContentServiceV1Test#getFile to check this behavior

I was thinking of the following scenario:

The user sends a simple file request, the entry value at this line is Entry{revision=2, path=/foo.json5, type=JSON, content={a: 'bar'}}

ArmeriaCentralDogma in this PR parses this entry no problem

Old clients or clients for other languages (i.e. python, go, etc..) can't parse this entry. At the very least, the old version java client can't parse json5.

No problem if this scenario isn't an issue 😅 (hopefully this is just my misunderstanding 🙏 )

I was thinking of the following scenario:

Oh, I've thought CD client's getFile API always accept Query but it seems not.
There could be a compatibility issue if a user uses the following API.

The return type of Entry#content() retrieved via this API is no longer String but TextNode for JSON5 files.

default CompletableFuture<Entry<?>> getFile(String projectName, String repositoryName, Revision revision, String path) { @SuppressWarnings("unchecked") final CompletableFuture<Entry<?>> f = (CompletableFuture<Entry<?>>) (CompletableFuture<?>) getFile(projectName, repositoryName, revision, Query.of(QueryType.IDENTITY, path)); return f; }

Also had a chat about this and decided to go for ContentServiceV2 which will be handled in another PR.

…tent

minwoox · 2022-02-21T14:03:46Z

I think it's almost done, excellent job, @ks-yim!
https://github.com/line/centraldogma/runs/5271252784?check_suite_focus=true#step:5:1211
Could you check the test failures, please? 😄

ks-yim · 2022-02-22T02:05:33Z

@minwoox Thanks for the patience.

It seems Jackson's ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER feature doesn't work well in Windows..? I should test it in my Windows laptop.

minwoox · 2022-02-22T02:18:25Z

@minwoox Thanks for the patience.

I really appreciate all the efforts. Please, don't worry about it. 😄

It seems Jackson's ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER feature doesn't work well in Windows..? I should test it in my Windows laptop.

Oops, let me also take a look at it.

ikhoon · 2022-02-22T02:26:03Z

common/src/main/java/com/linecorp/centraldogma/common/Change.java

        final JsonNode jsonNode;
        try {
+            if (isJson5(path)) {
+                jsonNode = Json5.readTree(jsonText);
+                return new DefaultChange<>(path, ChangeType.UPSERT_JSON, jsonNode, jsonText);
+            }
            jsonNode = Jackson.readTree(jsonText);
-        } catch (IOException e) {
+            return new DefaultChange<>(path, ChangeType.UPSERT_JSON, jsonNode);


nit: Could move jsonNode into try..catch block?

ikhoon · 2022-02-22T05:08:37Z

...ain/java/com/linecorp/centraldogma/server/internal/storage/repository/git/GitRepository.java

+                        final EntryType oldEntryType = !isJson5(oldPath) ? EntryType.guessFromPath(oldPath)
+                                                                         : EntryType.TEXT;


EntryType.guessFromPath() seems not to satisfy our requirements anymore.
How about introducing a new type such as FileType.

enum FileType { JSON, JSON5, DIRECTORY, TEXT, UNKNOWN; // ?? // This method would replace all EntryType.guessFromPath() usages. public static FileType guessFromPath(String path) { ... } // If there are some requirements. EntryType toEntryType() { ... } }

ikhoon · 2022-02-22T05:18:48Z

...ain/java/com/linecorp/centraldogma/server/internal/storage/repository/git/GitRepository.java

-                        if (!Objects.equals(newJsonNode, oldJsonNode)) {
-                            applyPathEdit(dirCache, new InsertJson(changePath, inserter, newJsonNode));
-                            numEdits++;
+                        if (!isJson5(changePath)) {


I'd like to minimize to use isJson5() from our codebase and make this type follow the change.type().
Question: Could we set UPSERT_TEXT to change.type() if JSON5 is specified when creating Change.ofJsonUpsert()?

ikhoon · 2022-02-22T05:25:53Z

common/src/main/java/com/linecorp/centraldogma/common/Entry.java

-               Objects.equals(content, that.content);
+               Objects.equals(content, that.content) &&
+               // Compare 'contentAsText' for JSON5 entries.
+               (!isJson5(path) || Objects.equals(contentAsText, that.contentAsText));


Could we simplify?

Suggested change

(!isJson5(path) || Objects.equals(contentAsText, that.contentAsText));

Objects.equals(contentAsText, that.contentAsText);

ikhoon · 2022-02-22T06:05:52Z

server/src/main/java/com/linecorp/centraldogma/server/internal/api/DtoConverter.java

        requireNonNull(entry, "entry");
        if (withContent && entry.hasContent()) {
-            return convert(repository, revision, entry.path(), entry.type(), entry.content());
+            // TODO(ks-yim): Not sure DtoConverter having business logic and it looks hacky to receive
+            //   queryType to handle JSON_PATH query for JSON5 files.


Not sure of 'DtoConverter' having business logic and it looks hacky to receive queryType to handle JSON_PATH query for JSON5 files.

Agreed. QueryType and isJson5() don't match with DTO. How about taking a Function rather than adding a specific logic here?

public static EntryDto convert(Repository repository, Revision revision, Entry<?> entry, boolean withContent, Function<Entry, ?> contentConverter) { return convert(repository, revision, entry.path(), entry.type(), contentCoverter.apply(entry)); }

ikhoon · 2022-02-22T06:16:00Z

...java-armeria/src/main/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogma.java

@@ -978,7 +979,9 @@ private static ArrayNode toJson(Iterable<? extends Change<?>> changes) {
            changeNode.put("path", c.path());
            changeNode.put("type", c.type().name());
            final Class<?> contentType = c.type().contentType();
-            if (contentType == JsonNode.class) {
+            if (isJson5(c.path())) {


How about adding the FileType mentioned to Change so that we check file type more statically?

public interface Change<T> { FileType fileType() { ... } }

jyterencekim · 2023-04-13T10:31:00Z

👀 @ks-yim Thanks for your work. I was just wondering whether this is still in progress.

ks-yim · 2023-04-14T07:05:09Z

@jyterencekim No 😭. Stopped working on this for now. You can take this issue if you want..

Just some notes tracing back an old memory..

This PR somewhat has achieved JSON5 support in CentralDogma, but is littered with hacky workarounds and improvisations. As @ikhoon has pointed out in his suggestion of adding FileType, a more structured approach is needed so that other config formats(.yaml, .toml, etc.) support can be added without pain later in the future, too.

Although JSON5 can be parsed as a normal JSON, parsed data can't be reversed back into the original JSON5 content so the data should be treated as EntryType.TEXT when it is patched & stored but should support ALL EntryType.JSON features such as..:

getFile() as Entry<JsonNode> but can still get the raw content via #contentAsText()
watchFile() as Entry<JsonNode> but can still get the raw content via #contentAsText()
getFile() & watchFile() with JSON_PATH
mergeFiles()

minwoox · 2023-04-17T08:39:08Z

Let us continue to work on this after adding API v2. @ks-yim Sorry to burden you. You are free to go. 😆

@jyterencekim Could you use a plain text file for JSON5 before we implement this? If you add the validation logic before pushing a file, I think you can use JSON5 without problems. 🤔

ks-yim · 2023-04-18T10:49:57Z

@minwoox thanks you for all the valuable feedback you gave while I am working on this 😄.

Closing this PR in favor of API v2.

minwoox · 2023-04-18T11:39:32Z

What I meant was I was going to work on this PR again after adding V2. 😓 Could you reopen this?

ks-yim · 2023-04-18T23:40:48Z

@minwoox Oops, my mistake 🤣. Reopened the PR! Looking forward to the rework!

ikhoon added the new feature label Dec 8, 2021

ks-yim force-pushed the json5-support branch 6 times, most recently from a8d64e2 to 5723c68 Compare December 10, 2021 05:39

ks-yim marked this pull request as ready for review December 10, 2021 07:35

ks-yim requested review from ikhoon, jrhee17, minwoox and trustin as code owners December 10, 2021 07:35

ks-yim changed the title ~~JSON5 support~~ Add JSON5 support Dec 11, 2021

minwoox reviewed Dec 15, 2021

View reviewed changes

...java-armeria/src/main/java/com/linecorp/centraldogma/client/armeria/ArmeriaCentralDogma.java Outdated Show resolved Hide resolved

common/src/main/java/com/linecorp/centraldogma/internal/Jackson.java Outdated Show resolved Hide resolved

ikhoon reviewed Dec 20, 2021

View reviewed changes

jrhee17 reviewed Dec 29, 2021

View reviewed changes

ks-yim added 4 commits January 3, 2022 12:47

JSON5 upsert

0a070ab

Add unit tests

96900f3

Add JSON5 support to ArmeriaCentralDogma

5ba66fe

Add ITs

e384b9d

ks-yim force-pushed the json5-support branch from 954b8e6 to e384b9d Compare January 4, 2022 01:45

ks-yim commented Jan 4, 2022

View reviewed changes

Clean up

b780570

minwoox reviewed Jan 4, 2022

View reviewed changes

Merge master

ade904f

Avoid string convertion in Entry#ofJson

cd45a77

Show server error if 'add file' fails in admin UI

2384e54

Add test case for watcher not notified unless JSON5 content changes

e0f6fbb

jrhee17 reviewed Jan 11, 2022

View reviewed changes

Normalize JSON5 upsert as text upsert to properly replicate JSON5 con…

a3c3e64

…tent

ks-yim force-pushed the json5-support branch from 3b0b431 to a3c3e64 Compare February 2, 2022 06:56

Fix WatchJson5 test

d7f23f6

ks-yim force-pushed the json5-support branch from 3a818e7 to d7f23f6 Compare February 2, 2022 09:06

ks-yim added 2 commits February 22, 2022 00:22

Remove lineBreaks in test JSON5 to fix windows test

a8d6adb

Fix ResponseTimeout in unit test

bcaef45

ks-yim added 2 commits February 22, 2022 11:18

Fix editJson5FileWithJsonPatch test

dc4e076

Fix windows test failure due to carriage return

cb9ae45

ikhoon reviewed Feb 22, 2022

View reviewed changes

Merge branch 'master' into json5-support

849bdb2

ks-yim closed this Apr 18, 2023

ks-yim deleted the json5-support branch April 18, 2023 10:50

ks-yim restored the json5-support branch April 18, 2023 23:32

ks-yim reopened this Apr 18, 2023

minwoox mentioned this pull request Mar 4, 2024

Support JSON5 #538

Open

	throw new CentralDogmaException("failed to parse JSON5 content as JSON: " + json5Text);
	throw new CentralDogmaException("failed to parse JSON5 content as JSON: " + json5Text, e);

	public class MetaRepositoryPushTest {
	class MetaRepositoryPushTest {

	dogma.project(), dogma.repo1(), Revision.HEAD, "Edit test1.json5",
	dogma.project(), dogma.repo1(), Revision.HEAD, "Add test1.json5",

		private static <T> EntryDto convert(Repository repository, Revision revision, String path,
		EntryType type, @Nullable T content) {

	static DefaultChange<?> deserialize(@JsonProperty("type") ChangeType type,
	@JsonProperty("path") String path,
	@JsonProperty("content") @Nullable JsonNode content) {

		final EntryType oldEntryType = !isJson5(oldPath) ? EntryType.guessFromPath(oldPath)
		: EntryType.TEXT;

	(!isJson5(path) \|\| Objects.equals(contentAsText, that.contentAsText));
	Objects.equals(contentAsText, that.contentAsText);

Add JSON5 support #655

Are you sure you want to change the base?

Add JSON5 support #655

Conversation

ks-yim commented Dec 7, 2021 • edited Loading

Motivation:

Modifications:

Results:

CLAassistant commented Dec 7, 2021 • edited Loading

minwoox commented Dec 8, 2021

codecov bot commented Dec 10, 2021 • edited Loading

Codecov Report

minwoox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrhee17 Dec 29, 2021 • edited Loading

Choose a reason for hiding this comment

jrhee17 left a comment

Choose a reason for hiding this comment

jrhee17 Dec 29, 2021 • edited Loading

Choose a reason for hiding this comment

jrhee17 Dec 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ks-yim commented Jan 4, 2022 • edited Loading

minwoox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ks-yim Jan 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ks-yim commented Jan 6, 2022 • edited Loading

minwoox commented Jan 7, 2022

jrhee17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ks-yim Jan 25, 2022 • edited Loading

Choose a reason for hiding this comment

ks-yim Jan 25, 2022 • edited Loading

Choose a reason for hiding this comment

ks-yim Feb 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minwoox commented Feb 21, 2022

ks-yim commented Feb 22, 2022 • edited Loading

minwoox commented Feb 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jyterencekim commented Apr 13, 2023

ks-yim commented Apr 14, 2023

minwoox commented Apr 17, 2023

ks-yim commented Apr 18, 2023

minwoox commented Apr 18, 2023

ks-yim commented Apr 18, 2023

ks-yim commented Dec 7, 2021 •

edited

Loading

CLAassistant commented Dec 7, 2021 •

edited

Loading

codecov bot commented Dec 10, 2021 •

edited

Loading

jrhee17 Dec 29, 2021 •

edited

Loading

jrhee17 Dec 29, 2021 •

edited

Loading

jrhee17 Dec 29, 2021 •

edited

Loading

ks-yim commented Jan 4, 2022 •

edited

Loading

ks-yim Jan 6, 2022 •

edited

Loading

ks-yim commented Jan 6, 2022 •

edited

Loading

ks-yim Jan 25, 2022 •

edited

Loading

ks-yim Jan 25, 2022 •

edited

Loading

ks-yim Feb 2, 2022 •

edited

Loading

ks-yim commented Feb 22, 2022 •

edited

Loading

minwoox commented Feb 22, 2022 •

edited

Loading