Proposal for discussion - remove GraphQL CMS endpoints in CMS 6

[emteknetnz]

Currently, there are two distinct systems for serving JSON in the Silverstripe CMS: the "regular" Silverstripe controller endpoints defined in private static $url_handlers and the GraphQL endpoints. Having two different standards creates additional overhead for everyone involved.

Current Situation

The Silverstripe CMS uses both regular controller endpoints and GraphQL endpoints. This has leads to unnecessary development complexity and time wasted on higher maintenance efforts than otherwise would be the case if there was better standardisation. For the community, the perception that GraphQL should be used has imposed barrier on community contributions.

Proposal

We propose to entirely remove GraphQL endpoints for Silverstripe CMS 6 and replace their functionality with regular controller endpoints.

This change applies only to CMS endpoints, such as those used by elemental and asset-admin. This does not affect the status of the silverstripe/graphql module as a supported module which some projects use to create public facing JSON APIs.

Rationale

Benefits of Standardising on Regular Silverstripe Controllers:

• Lower Barriers to Customization: Community members won't need to learn GraphQL in addition to regular Silverstripe endpoints when making CMS modules.
• Reduced Maintenance Burden: Maintaining a single standard reduces complexity for the CMS squad.

Why not standardise on GraphQL Instead?

• Perceived Difficulty: GraphQL is perceived to be difficult to use in the context of Silverstripe CMS.
• Low usage: GraphQL endpoints constitute only about 18% of CMS endpoints. Standardising on GraphQL would require significantly more effort than standardising on regular endpoints. The numbers of endpoints are detailed below in Appendix One.

Considerations

• Upgrade Pain: Projects using custom GraphQL CMS endpoints or modifying existing ones will face difficulties during the upgrade to CMS 6 as their old GraphQL endpoints will no longer work
• Mitigation Strategies:
  • Provide extension hooks in new Silverstripe controller endpoints to facilitate customization.
  • Create documentation on migrating from CMS GraphQL endpoints to regular controller endpoints.
  • Offer assistance for third-party modules or projects needing migration support.

Note on prior work done

Some work to remove GraphQL from the CMS has already progressed quite far. Comments where raised on that issue expressing that there should have been a public discussion before embarking on this work. Within the internal product team at Silverstripe there is agreement that we should look to better standardise the API layer in the CMS. However, in retrospect a public discussion should have come first to get feedback from others. Apologies for doing this out of order.

Appendix One: Endpoint counts

A simple script was created to count the number of $url_handler endpoints compared to GraphQL queries and mutations across supported modules. The script was run on an install of silverstripe/recipe-kitchen-sink. Note that the cms graphql queries/mutations are used in silverstripe/versioned-admin. The results are as follows:

$url_handlers:

framework: 24
mfa: 18
asset-admin: 12
gridfield-bulk-editing-tools: 9
silverstripe-advancedworkflow: 7
campaign-admin: 7
admin: 6
blog: 4
linkfield: 4
restfulserver: 4
cms: 3
silverstripe-elemental: 2
silverstripe-queuedjobs: 1
cwp: 1
session-manager: 1
sharedraftcontent: 1
frameworktest: 1
realme: 1
reports: 1
environmentcheck: 1
registry: 1
assets: 1

Total: 110


GraphQL:

asset-admin: 12
silverstripe-elemental: 9
cms: 3

Total: 24

@silverstripe/core-team

[sminnee]

So to be clear - is the idea to deprecate the use of GraphQL for powering the CMS?

GraphQL has been in place for 8 years. If it's still causing issues, it seems reasonable to declare the use of GraphQL for driving the CMS a failed experiment, although I note that's a big call.

If module maintainers have made use of GraphQL, would the GraphQL endpoints be available in a module?

Do we have any sense of how many modules would fall into that category>

[emteknetnz]

So to be clear - is the idea to deprecate the use of GraphQL for powering the CMS?

Yes, deprecate the use of GraphQL for powering the CMS in CMS 5 though keep using it, and remove it entirely in CMS 6 where the endpoints are switched over to use regular Silverstripe controllers.

If module maintainers have made use of GraphQL, would the GraphQL endpoints be available in a module?

I'm not really sure. If they've defined their own schema entirely independently from admin schema, then, to the best of my knowledge it should work fine, just as a project defining its own frontend schema should still work fine. However if there's any sort of reliance on the PHP code in asset-admin, elemental, cms, or any of the JS code in admin, then it won't work at all.

Take what I say with a grain of salt though I'm not an expert on the inner workings of Silvestripes implementation of graphql. No one actively working on the code-base is anymore, that's part of the problem.

Do we have any sense of how many modules would fall into that category

I don't have any data on that sorry, and I don't know how I would obtain it. My assumption is that it's very low, though that's just my assumption.

In terms of "upgrade pain", I don't think would be that bad. I do think that any CMS modules that are using graphql could be ported over to use regular Silverstripe controllers with admin authentication in a pretty short amount of time.

[michalkleiner]

There are still people that have the upgrade from 3 to 4 in memory and basically I would like to prevent similar headaches.
I'm not sure how many people wrote custom grapqhl handlers and resolvers in their modules for the CMS for e.g. custom fields, how many used it to power some front-end apps in a hybrid mode or used the CMS in a full headless mode.
If we decide to embark on this (as it seems that's been already the direction anyway) it would be great if we could minimise the upgrade/removal effort and possibly keep the code for the endpoints somewhat available in an optional module or so. We might have a picture of what it would mean for the cms and the supported modules, but we probably don't know what community devs already used graphql for and how they are (or not) relying on it.
Would this possibly warrant a short blog post asking for comments from module devs?

[emteknetnz]

To understand the landscape better, I've done a quick bit of research to try and gauge GraphQL usage in modules. I think this could also be used as a proxy for estimated CMS GraphQL usage (not frontend schemas) within projects where no code was opened source. I think this gives better data that a blog post will which will likely have a very patchy response at best.

For the research I had a look on ssmods.com and ordered by number of installs and filtered by 'module' and '5.*'. I looked for the 50 most popular modules that were not supported modules or "by silverstripe" and then looked at the GitHub repository and scanned for GraphQL usage e.g. a _graphql config folder, or *Resolver.php files in the src directory.

From what I could see, only 1 out of the top 50 non-supported/by-silverstripe modules used GraphQL, that's only 2%. That strongly indicates that CMS GraphQL usage is very low. I'd say that "upgrade pain" relating to the GraphQL admin API going away in the CMS is going be complete non-issue for the vast majority of projects.

https://ssmods.com/#~(cfi~(Type~(~(vtm~'module~ivl~'module))~Supports~(~(vtm~'5.*2a~ivl~'5.*2a)))~csr~(sdi~'desc~sct~'Installs))

Uses GraphQL

• https://github.com/elliot-sawyer/silverstripe-link

Does not use GraphQL

• https://github.com/jonom/silverstripe-betternavigator
• https://github.com/sheadawson/silverstripe-linkable
• https://github.com/tractorcow/silverstripe-autocomplete
• https://github.com/WPP-Public/akqa-nz-silverstripe-menumanager
• https://github.com/Rhym/silverstripe-cms-theme
• https://github.com/stevie-mayhew/silverstripe-svg
• https://github.com/kinglozzer/silverstripe-metatitle
• https://github.com/tractorcow/silverstripe-colorpicker
• https://github.com/lekoala/silverstripe-debugbar
• https://github.com/sheadawson/silverstripe-dependentdropdownfield
• https://github.com/phptek/silverstripe-sentry
• https://github.com/axllent/silverstripe-trailing-slash
• https://github.com/dnadesign/silverstripe-elemental-list
• https://github.com/lekoala/silverstripe-cms-actions
• https://github.com/tractorcow/silverstripe-opengraph
• https://github.com/stevie-mayhew/hasoneedit
• https://github.com/WPP-Public/silverstripe-colorpalette
• https://github.com/jonom/silverstripe-text-target-length
• https://github.com/lekoala/silverstripe-softdelete
• https://github.com/axllent/silverstripe-email-obfuscator
• https://github.com/silverstripe-terraformers/silverstripe-embargo-expiry
• https://github.com/WPP-Public/akqa-nz-silverstripe-responsive-images
• https://github.com/tractorcow/silverstripe-robots
• https://github.com/WPP-Public/akqa-nz-silverstripe-hashpath
• https://github.com/sheadawson/silverstripe-zenvalidator
• https://github.com/dynamic/silverstripe-country-dropdown-field
• https://github.com/dynamic/silverstripe-elemental-baseobject
• https://github.com/pstaender/silverstripe-redis-cache
• https://github.com/axllent/silverstripe-scaled-uploads
• https://github.com/firebrandhq/silverstripe-phonelink
• https://github.com/nathancox/silverstripe-hasoneautocompletefield
• https://github.com/axllent/silverstripe-form-fields
• https://github.com/WPP-Public/akqa-nz-silverstripe-cacheinclude
• https://github.com/axllent/silverstripe-version-truncator
• https://github.com/silverleague/silverstripe-ideannotator
• https://github.com/bummzack/sortablefile
• https://github.com/wilr/silverstripe-googlesitemaps
• https://github.com/silvershop/silverstripe-hasonefield
• https://github.com/Rhym/silverstripe-color-field
• https://github.com/dynamic/silverstripe-elemental-blog
• https://github.com/axllent/silverstripe-bootstrap-forms
• https://github.com/dynamic/silverstripe-elemental-accordion
• https://github.com/benmanu/silverstripe-simple-styleguide
• https://github.com/dynamic/silverstripe-geocoder
• https://github.com/dynamic/silverstripe-elemental-embedded-code
• https://github.com/WPP-Public/akqa-nz-silverstripe-wkhtml
• https://github.com/silvershop/silverstripe-sqlquerylist
• https://github.com/fromholdio/silverstripe-gridfield-limiter
• https://github.com/plastyk/silverstripe-dashboard

Only elliot-sawyer/silverstripe-link (formally gorriecoe/silverstripe-link) uses GraphQL. In terms of migration to CMS 6, there are a couple of options:

• Migrate the modules API layer - replace the GraphQL implementation in the module with something very similar to LinkFieldController in silverstripe/linkfield v4. I'm happy to provide guidance here.
• Migrate projects using the module to silverstripe/linkfield v4 - there's a build task specifically for this.

[michalkleiner]

Thanks for the data @emteknetnz. It's possible it might be a non-issue and devs may even welcome the change. It might seem that I'm against progress and moving forward, but it's probably more about just making sure we minimise surprises and communicate such substantial changes earlier and in a better way - and I do appreciate I may not have such better way up my sleeve and I believe the intentions are good.

[michalkleiner]

@forsdahl any comments from Europe?

[forsdahl]

@michalkleiner I can't speak for everyone in Europe, but we at Creamarketing have done a lot of admin customizations over the years, but we have yet to do any custom GraphQL endpoints. All our own admin modules and customizations use normal Silverstripe endpoints. The only thing we have slightly customized regarding GraphQL is some logic in the FolderTypeResolver, so as long as some new corresponding Silverstripe Controller endpoint for AssetAdmin can be customized, we're good.

In fact we have some plans on doing a Sveltekit based headless admin for Silverstripe, and for that project having normal Silverstripe endpoints for everything in admin would be easier than having a mix of JSON endpoints and GraphQL endpoints.

[NightJar]

GraphQL has been in place for 8 years. If it's still causing issues, it seems reasonable to declare the use of GraphQL for driving the CMS a failed experiment, although I note that's a big call.

I admit, I never understood the drive to use GraphQL to power the CMS in the first place. It is a useful tool in some cases... but many cases in the CMS are not that case, as evidenced by the low pickup (although partially probably also due to quite a learning curve). Still, I'm not sure I'd call it a failed experiement. Though neither should that prevent the team from moving on.

There are still people that have the upgrade from 3 to 4 in memory and basically I would like to prevent similar headaches

Anyone who bought into GraphQL has already been bitten in the GQL major update... ironically also from 3 to 4.
I have known people to quit Silverstripe CMS entirely because of it.

If there's anyone left, I don't think they'd much appreciate being 100% disjointed again. Although I guess this would only apply to admin extensions this time.

for the research I had a look on ssmods.com

The canonical source of truth for composer based packages is packagist.com, which names 32 dependencies (including core & supported modules).

https://packagist.org/packages/silverstripe/graphql/dependents?order_by=downloads

This of course will list neither private modules nor projects.

---

I have no love for GraphQL; while I find the concept easy enough to grasp, I find the Silverstripe implementation of it (particularly since v4) extremely obtuse, overly complex, and 100% obfuscatory. I couldn't even follow the execution path to a very simple ultimately 1 line fix. silverstripe/silverstripe-versioned-admin#255 (comment)

At the same time I feel like this is a bit of a step backward. More particularly the all or nothing approach. Rebuilding asset admin feels like a grand expensive effort in more of a witch hunt than a focus on progress. While I feel it fine to revert the CMS to "normal" endpoints (RESTful or whatever), I think the task should be done over time rather than in the short ~half-year until the CMS 6 is due, per the major release schedule expectations.

It is possible - as current - to have both "endpoint options" at once.

I'm not an expert on the inner workings of Silvestripes implementation of graphql. No one actively working on the code-base is anymore, that's part of the problem.

Other than the obvious "Silverstripe Ltd. has no expertise with GQL anymore" it appears as there's no real driving reason for this change.

[sminnee]

"None of the maintainers have GraphQL expertise" is probably a more useful framing than "Silverstripe Ltd. has no expertise with GQL anymore".

[michalkleiner]

Any other comments, @silverstripe/core-team?

I personally don't think we will get anywhere conclusive here and the work is progressing towards the removal of GraphQL for CMS 6 regardless.

Other than the learning of "let's do an RFC before we do the work next time" there's probably not much to talk about.

[kinglozzer]

No real comments on removing it from the CMS, other than echo-ing the need for an RFC. I appreciate that if there’s no expertise in the open-source team available to work on it then it seems like an open and shut case, and pushback against it would be pretty frustrating, but if we’re trying to foster more community involvement then decisions like this (even if they risk being unpopular) need to be made in the open, and not semi-retrospectively.

I’ve not done any CMS customisations I can recall that touch GraphQL, only React. I think to have seen the real benefits of GraphQL in the CMS, we would’ve needed to go all-in with React. That’s never happened, I doubt it ever will at this point.

One question I have though: is this a step toward dropping Ltd’s support for the graphql module entirely? Does Ltd use GraphQL in their own builds? I think losing official support would be a pretty big blow to Silverstripe’s offering as a framework, so that’s my main concern off the back of this

[GuySartorelli]

One question I have though: is this a step toward dropping Ltd’s support for the graphql module entirely?

There are no current plans to do that. Though sentiments about the module vary from person to person 😅

[chillu]

IMHO anything that makes the CMS easier to maintain right now is a good choice for the longevity of the wider ecosystem. There was ample time for a) the community to pick up the "GraphQL in CMS" use case, and b) step up as maintainers to make a more complex (powerful?) CMS viable.

My worry would be that this ends up as a tactical "half measure" that the CMS has been beset by over the years, where there's not enough capacity to follow through on a coherent alternative model in one upgrade cycle (presumably at least including some customizeable RESTful endpoints for frontend-heavy UIs like AssetAdmin or Elemental). The key will be community involvement to get transparency of current customisations (which is where most of the upgrade pain will be).

A bit of 20/20 hindsight: The decision for a more frontend-heavy and API powered CMS was made at a time when we had vastly more resources as CMS maintainers, and was meant to be a "two birds with one stone" architecture - since we also needed better first party structured data support outside of the CMS. And I learned the hard way that some features aren't easily portable between languages/runtimes: The canonical GraphQL implementation is in Node because it it's very efficient to load all the data structures into memory on boot across requests. With PHP's "shared nothing" request processing, we needed to ensure this step is fast for each request. Which lead to caching, which lead to complexity...