Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

init sankey #770

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

init sankey #770

wants to merge 29 commits into from

Conversation

mabudz
Copy link

@mabudz mabudz commented Jan 7, 2024

Create sankey diagram and add tutorial.

The approach here creates the sankey diagram by using the sankey plot function of the pyam package. This function requires a mapping dictionary. The mapping dict is created by using the newly created automatic report message::sankey and a utility function sankey_mapper()map_for_sankey.

This pull request is related to the draft pull request in message_data , which can be removed, since the sankey diagram has been implemented in the pyam package.

Note

The new key is now only added when calling Reporter.add_sankey().

How to review

General approach should be reviewed. Also, I am not sure if the util function sankey_mapper() map_for_sankey() is at the right place.

PR checklist

  • Continuous integration checks all ✅
  • Add or expand tests; coverage checks both ✅
  • Add, expand, or update documentation.
  • Update release notes.

@CLAassistant
Copy link

CLAassistant commented Jan 7, 2024

CLA assistant check
All committers have signed the CLA.

@glatterf42
Copy link
Member

Thanks for this PR :)
Getting the tests to pass is a bit tricky since the API for pyam changed when they went from version 1.9.0 to 2.0.0 (as you might expect) and our code needs to satisfy both versions since we test on Python <= 3.9, which is not supported for pyam 2.0.0. So let me take a closer look, maybe we'd want to use the function that pyam's sankey function wraps around directly, after all.

@glatterf42 glatterf42 added this to the 3.9 milestone Jan 11, 2024
@glatterf42 glatterf42 self-assigned this Jan 11, 2024
@glatterf42 glatterf42 added the enh New features & functionality label Jan 11, 2024
@glatterf42
Copy link
Member

Thanks @daymontas1 for stepping up here, much appreciated :)

You should be able to test the PR as is and see how you like it's usefulness, but there are a few things that I would like to see happen before we merge it:

  • Please rebase on main to keep it up to date with current developments.
  • Please migrate the main functionality from util/__init__.py to somewhere more appropriate. E.g. report/sankey.py if it's a reporting tool or util/sankey.py if there are more general use cases.
  • Please check if we can use plotly directly instead of pyam's wrapper functionality. This should allow us to utilize more design options and avoids some dependency constraints (see above). See some examples and their docs.
  • Please confirm that we want to integrate the sankey functionality in the reporting as it currently stands. I'd probably ask at least @khaeru for his opinion.
  • Please write a test if possible to confirm that sankey_mapper maps the correct values to the correct keys. We might also need to adapt one or two reporting tests that check the number of reporting steps if this stays in the default reporting workflow.

Please feel free to ask about any of these steps if you need help :)

@gidden
Copy link
Member

gidden commented Mar 13, 2024

Hi all, to simplify here, I would simply limit this functionality to pyam > 2 - we can do a version check for this on the fly and raise an error.

@@ -141,6 +141,7 @@
"message::costs",
"message::emissions",
),
("message::sankey", "concat", "out::pyam", "in::pyam"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would get rid of this, and simply do the concatenation outside of reporter (e.g., in the jupyter notebook here)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept this for now as I don't know what the Reporter actually does and it seems useful to just get the dataframe format we need from the Reporter. But please elaborate :)

Copy link

@daymontas1 daymontas1 May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have created a new file, sankey.py, within the report directory, which extracts Sankey diagram functionality from init.py. Fridolin (@glatterf42), could you please confirm if this is what we want? Additionally, I have updated the westeros_sankey.ipynb file to ensure compatibility with the new sankey.py file in the report directory. I have annotated these changes in the westeros_sankey.ipynb below. Also, in this case, the following line inside the init.py file must be either removed or commented out:
("message::sankey", "concat", "out::pyam", "in::pyam")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @daymontas1, these changes sound good to me. For the line that has become redundant, please remove it rather than commenting it out, this keeps the code clean.
Could you please push your changes to the branch so that we can take a look and see how the tests are doing?
If you don't have write access to @mabudz's fork, please let us know so that we can figure out a solution :)

message_ix/util/__init__.py Outdated Show resolved Hide resolved
message_ix/util/__init__.py Outdated Show resolved Hide resolved
message_ix/util/__init__.py Outdated Show resolved Hide resolved
@glatterf42

This comment was marked as outdated.

@glatterf42

This comment was marked as outdated.

@glatterf42

This comment was marked as outdated.

@glatterf42 glatterf42 modified the milestones: 3.9, 3.10 May 23, 2024
@daymontas1

This comment was marked as outdated.

@glatterf42

This comment was marked as outdated.

@mabudz

This comment was marked as outdated.

@glatterf42

This comment was marked as outdated.

@khaeru

This comment was marked as outdated.

@glatterf42

This comment was marked as outdated.

Copy link

codecov bot commented Jun 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.6%. Comparing base (ef66e37) to head (30d8d32).
Report is 20 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##            main    #770     +/-   ##
=======================================
- Coverage   95.6%   95.6%   -0.1%     
=======================================
  Files         46      47      +1     
  Lines       4344    4383     +39     
=======================================
+ Hits        4154    4191     +37     
- Misses       190     192      +2     
Files with missing lines Coverage Δ
message_ix/report/__init__.py 100.0% <100.0%> (ø)
message_ix/tests/test_report.py 100.0% <100.0%> (ø)
message_ix/tests/test_tutorials.py 97.6% <ø> (ø)
message_ix/tests/test_util.py 100.0% <100.0%> (ø)
message_ix/util/sankey.py 100.0% <100.0%> (ø)

... and 1 file with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enh New features & functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants