The future of NCL in the ESMValTool #1057
Comments
A bit of good news here is that they are translating this particular example to Python as a pilot, see: NCAR/ncl#64 (comment) |
|
From the roadmap it also looks like there will be training available. |
|
Thanks for picking this up, @mattiarighi !
I completely agree that this is a process, cannot happen over night, and should have (almost) zero bearing on v2.0 (the exception just being that, all else being equal, I wouldn't prefer NCL for new developments).
@bouweandela has that covered. I think therefore CVDP isn't a blocker, but maybe we should collect all such packages that we have in some place (extra ticket, wiki?)?
That is unfortunate, but adding substantial new NCL code seems to go in the wrong direction. In most cases those developers will have to adapt to the changing environment for reasons entirely outside of ESMValTool.
👍 Though of course, as you know, intentions aside, the limits will always be funding and man-power. Usually, few people with the necessary skill are eager to maintain things after a few years without the mandate to change them.
To formalize this: With the release after v2.0 or with v2.0 itself, we should formally deprecate it. Imho, we can already now give the general recommendation to rather prefer python, making it clear that NCL contributions are still acceptable for now. But if somebody is on the fence on what to choose, they should go for python.
Hm, this shouldn't be specific to python. If it seems that the burden on python contributions is higher than on other languages, we should rather see how we can achieve the same high standards in those languages. At the moment we are rather lenient there, probably for lack of automatic qa. As an example, we know that, no matter the language, functions should be short. I find around 45 lines as an upper limit a good guide rule. If in python you make much longer functions, you are likely to get a complexity warning or too-many-local-variables from our code checkers. In NCL this malpractice is still wide-spread. Bottom line: Quality standards should be equally high for all languages, the effort comparable.
Interesting question. Looking at the list of contributors (here for the private repository for those who have access), my impression is that most would be comfortable with python. What do you think? Of course, this is just a personal impression. To get a better picture we could do a survey, best limited to those having contributed at least one commit, or try automatic numbercrunching, eg see what fraction of lines added per person where in NCL. I think a survey can be done quickly with (eg) google forms; number crunching would take more effort. It would be very interesting to hear from NCL developers on this. |
|
I got 0 NCL lines 🤣 So this will be easier to discuss when we will have split ESMValTool (at least) into core and diagnostics. The core is and will be Python; the diagnostics are a bit more relaxed and the shepherding of the scientists developing them towards Python (and away from NCL) can be done in a bit more of a relxaed way rather than at gun point if it was for the core |
|
DKRZ is still organizing NCL workshops for the users, so they are probably not planning to retire NCL support soon. |
Just remember that the end of python 2 support seemed to be far far away |
I am working with Met Office software which is (as expected) Python2 and everytime I try and update Py2 deps I get warnings about the looming dat of January 1st 2020 😁 |
|
There are some news about the future of NCL: http://www.ncl.ucar.edu/Document/Pivot_to_Python/september_2019_update.shtml "To summarize the status of NCL, the project is "feature frozen". CISL does not have plans at this time to add new features to NCL. We will, however, prepare maintenance releases containing critical bug fixes and user-contributed code on an infrequent basis. Thus, there is no reason for members of the community with substantial investments in NCL to consider porting their NCL code to Python (unless you want to take advantage of the improved scalability afforded by our use of Dask)." |
|
A new update from 11 November 2020 by the NCL authors:
|
|
NCL in conda-forge has been a bit neglected for a while. This means that it has not been (re-)build against more recent versions of the usual libraries (libgdal, libnetcdf, proj,...) which makes it hard for us to install recent versions that are required by other parts of the ESMValTool eco system. I have spent a good chunk of the last week and the beginning of this week trying to improve the situation, but this is a real drain on resources and it is not clear how long we can keep this up. |
|
Related to this, you might be interested in the python GeoCAT package, which translates many of the original NCL function in Python. It could be a way to encourage NCL users to move to Python... |
|
I think it really is time to phase out NCL. The conda-forge build is broken again and it is clear now that NCAR is no longer performing maintenance. The existing builds will continue to work for a while and we can make some effort to keep the car running, but sooner rather than later this becomes too much of a burden. I propose that we develop a concrete plan for the removal of NCL. Something along the lines of
@ESMValGroup/esmvaltool-coreteam, @ESMValGroup/esmvaltool-developmentteam, thoughts? |
|
There are NCL PRs (e.g. #2156, which should be merged because it was described in one of the ESMValTool paper) which have been there kind of forever and I'm afraid nobody has the resources to convert that all to python. |
|
Hi @zklaus I fully support we migrate from NCL to Python soon, but that is a very resource-intensive task, who gonna do it? Points 1 and 2 are most welcome, afraid 3 is needed, but much harder than it sounds
where do you see that? GA and feedstock tests still passing... |
TBD. But one thing is crystal clear: It will become infeasible to have NCL as a conda-forge dependency. The only question is when.
There are two PRs outstanding to rebuild for new versions of Proj and HDF. Both are failing. If they are not fixed, installing NCL from conda-forge will block updating Proj and HDF, which in turn will prevent the installation of dependencies like netcdf and iris when they depend on those. This is some time in the future, for sure, but as you note, we will need some time.
Not sure what you are referring to: Tests in this repo and the core one? Our own feedstock? The NCL feedstock? |
Gotcha! I was suspecting it was one of the big deps Yeah I was referring to our esmvaltool GA tests (which obv don't contain the pkg build test no more) and the esmvaltool feedstock tests |
|
cheers for raising this BTW, Klaus! It's getting very tight indeed |
|
This might be a good point for discussion in the upcoming workshop https://github.com/ESMValGroup/ESMValTool/discussions/2588. At the moment, slightly less than half of all diagnostic code is NCL, but maybe Python and/or R code is just more compact. It seems unlikely that there will ever be enough time to translate all (or even many) NCL diagnostics to Python. Maybe the @ESMValGroup/scientific-lead-development-team could think about which diagnostics are required as a minimum for the tool to remain interesting from a scientific point of view? I do agree that it would be good to stop introducing new NCL diagnostics in the tool, as seems a waste of time in the longer run. If installation from conda really becomes a problem and dropping support for NCL is just not feasible from a scientific point of view, an alternative to completely dropping support for NCL would be to just no longer install NCL from conda. The installation of the NCL interpreter would then be left to the users. They might be able to install the NCL interpreter
|
|
I am not really sure I fully understand why some people are so eager to get rid of NCL. Even if installation from conda gets too difficult to maintain, there are other options as @bouweandela pointed out. At the moment, 17 recipes with about 100 diagnostic scripts are based on NCL. Among those are some of our flagship diagnostics such as IPCC AR5, SMPI and perfmetrics. I do not see any realistic possibility to convert so many diagnostics to Python. When transitioning from ESMValTool v1.1 to v2.0, a lot of diagnostics could not be converted because it was way beyond our resources (even though we had quite a few coding workshops). And that did not even involve changing the original programming language. Dropping NCL would change the ESMValTool significantly from what has been advertised over the last years and for this reason alone, I am strongly advising against such a step. In the end, it does not matter too much if people would have to install NCL on their own, the main point is that diagnostics such as IPCC AR5 are in and could (at least in theory) be used. That said, I also find it difficult to simply decide to trash the work of quite a few contributers over the last years just like that. |
|
Note that flagship diagnostics will soon also include code used for IPCC AR6 (first PR in #2533). I think a majority of the ESMValTool produced figures for AR6 are based on NCL diagnostics (new ones but also already existing ones). I guess here as well, a translation of the diagnostics to Python is not feasible in practice... It would be quite unfortunate to continue advertising ESMValTool for the produced AR6 figures and at the same time not being able to merge the code to the main branch and not being able to use the Tool to reproduce these figures because support for NCL was completely dropped... I support the idea of considering alternative ways to install the NCL interpreter if such installation from conda becomes problematic. |
|
I think there is some misunderstanding. It certainly is not my intention to "trash" anybody's work. Quite the contrary, I will come back to that. But it is my assessment that NCL will become unavailable. That is not due to some weird behavior by conda-forge, but because NCL's code quality is not up to modern standards and because it depends on other libraries that are still developing and that we need to keep updated, such as Proj. As for alternate installation methods, note that they all will face similar issues as conda-forge. For Even if things can be kept running, it will never be made to parallelize or updated to deal with CF 1.10 or UGRID, which means that we risk breaking another promise of ESMValTool. After all, the AR6 recipes (just as an example) are not interesting to produce the exact same figures from the exact same data---those are published---but rather because it is easy to add CMIP7 and CMIP8 models in there. Will NCL diagnostics be able to handle that? I doubt it. Will all of this happen overnight? No. Within one year? Probably not. Two? Perhaps. Three? Probably. Some of this may be mitigated by maintenance. However, NCAR's GeoCAT team does seem to take a rather hands-off approach to maintenance. Have a look at the NCL issue tracker and think whether you want to rely on that level of support. Also note, that the two packages touted at the time as a replacement and bridge to Python, PyNGL and PyNIO also have since been placed into "maintenance mode" and require patches just to make them compile. It seems currently the CI on the official NCL repo is not able to build NCL. Despite conda being the officially suggested mode of installation and the promise to make bugfix releases and keep the conda install working, you have to go back to May 2020 to find contributions from the NCL team to the conda build. The community has tried to apply enough duct tape to hold things together. I myself have spent weeks to keep NCL and PyNGL running in the current conda-forge. We can keep this up for a bit longer, but I won't be doing that in 5 years. What I am saying is: With NCL, we are on a train that is slowly falling apart. We still have time, let's salvage what we can by making a plan now. Let's not wait until it just disintegrates below us. So returning to what I said at the top: I don't want to trash the work. I don't want to lose those diagnostics. That is precisely why we should think about how best to deal with them now. Then we will have some time and we will be able to preserve the most important diagnostics and recipes for use with CMIP7, 8, and beyond. Perhaps they will even acquire a few new tricks along the way. |
|
In case we need to contact NCL developers, the following is the complete list of users that have touched Edit: To clarify, this is git authors of ncl files currently in the main branch. It does not capture branches that are not integrated yet and it does also not capture possible authors/collaborators outside of git, which particularly means authors of ESMValTool version one diagnostics that have later been ported by someone else. This seems reasonable to me because those are old contributions and absent further involvement since then, we probably don't expect those contributors to resume maintenance of their code. Possibly Active
Retired
Technical
|
|
Although I'm not writing new software in NCL, I'm involved in pull requests containing NCL as a maintainer for a tool from @irenecionni (#2156) and there is a PR planned for things from the AR6 repository, where I build up on an existing NCL cmorizer. In the second case I maybe could convert it to python, for the PR #2156 I don't see that I, or @irenecionni would have the resources to change these diagnostics to python. |
Could it be an interesting internship/MSc graduation project? |
Could you maybe also give estimates of how many NCL functions and lines of code we have in the main branch? It would give an idea of the effort to translate NCL code into Python given the number of persons listed above. But this would not count NCL code waiting to be merged in some PRs (e.g. #2156), or AR6 diagnostics... So this effort would certainly be larger than that. |
|
Here is the number of all non-empty non-comment lines of all ncl files in
|
|
A brief introduction: I am the author of the CVDP, and for many years served as a scientific advisor to the NCL Development Team. Since the dissolution of the NCL Dev Team and the formation of the GeoCAT Dev Team, I no longer am directly involved with NCL/GeoCAT development efforts, and do not speak for GeoCAT. However, I have been following this discussion for a while, and thought it was important to speak up, but only after I discussed the situation with the GeoCAT Team lead @erogluorhan. Since the dissolution of the NCL Dev Team, I have been assured many times by NCAR/GeoCAT leadership that NCL builds will be supported as long as the community asks for it. There is no doubt at the present time that the community (beyond ESMValTool) still wants supported builds. @erogluorhan has stated providing stable builds via conda is the number one NCL-related priority for the GeoCAT team. Another priority of the team is to support community contributions to NCL. There is no doubt that these priorities have not been treated as such over the past couple of years, and there are 2 principal reasons why: 1 - GeoCAT has not had the personnel to be able to support all of their external packages and to pursue their funded mandate to build new python tools. 2 - Due to retirements and people switching jobs, there is very little NCL institutional knowledge left within GeoCAT. I find it completely understandable that NCL users and system/package maintainers have expressed frustration with the current situation. Regardless, for now, GeoCAT has agreed to evaluate and fix NCL's conda builds and to address @zklaus proj pull request NCAR/ncl#173, both in short order. Some other items from the discussion above that I would like to chime in on:
I hope the above provides some clarity with respect to NCL and advances the discussion! |
|
Thanks, @phillips-ad for your comments. It's nice to hear from the NCL side! It would be great if the NCAR could pick up the maintenance of NCL again. In that case, we may push the other bugfixes that reside only in the conda-forge feedstock at the moment upstream as well. This may even have to happen before the integration of NCAR/ncl#173 to make it possible again for NCL's CI to build NCL. Please don't hesitate to ask for assistance with that if you would like that. On the other items you raised:
We do make frequent use of CVDP via ESMValTool at my institute, so I am very happy to hear that you are still interested in maintaining it and possibly moving it to Python. Perhaps you might want to consider doing that directly in ESMValTool. This way, you can benefit from a lot of the standard preprocessors for data fixing, regridding, spatial and temporal statistics, and more. Where there are gaps in the preprocessors, adding those needed for CVDP might prove useful beyond that application for recipes and we can likely give a little bit of support in these developments. Thanks again for you input, and do get/stay in touch about anything we can do to help with NCL or CVDP. |
|
Hello all, I am Orhan from the GeoCAT team. @phillips-ad already shared significant insights about our team and NCL commitments, and I'd like to make a few more:
|
|
Hi @erogluorhan, it's really good to hear from you, too! I understand that there are no long term plans for NCL and that it is difficult to maintain such software with limited resources. With the CI in place, it should be straightforward to integrate the proj6 port and the boz fixes. From there, we might be able to tackle some of the open issues. Here is probably not the place to discuss how to fix the NCL CI, but let me know if you want to take it up in the NCL repo or any other way. |
|
Hi @zklaus , yes, let us not clutter this repo much, but just to quickly answer: I am currently working on getting the CI up and running again. My first priority would be to get Github Actions configured as you pointed out my PR (This would also help GeoCAT team address future maintenance needs hopefully more timely in NCL as we have much experience with Github actions). However, if I feel like setting up Github Actions would take much time, I'd also consider fixing th existing CircleCI config just to expedite the things. Let us follow up the rest of this conversation under either NCAR/ncl PR#170 or your NCAR/ncl PR#173. |
The discussion about the future of NCL support in the ESMValTool has come up quite frequently in the last weeks in various GitHub issues here.
The reason behind this discussion is (should be) well known: NCAR has decided to stop the development of NCL, which has been put into maintenance mode. Graphics routine have been translated to Python and are available via the PyNGL library. This means that NCL is still available and will be available "for the foreseeable future" (as NCAR itself states in its letter to users), but the transition to Python is recommended.
Some general considerations to trigger the discussion:
@ESMValGroup/esmvaltool-coreteam
@zklaus @hb326 @LisaBock @ruthlorenz (please tag other non-core members who might be interested in this topic).
The text was updated successfully, but these errors were encountered: