Update base project config files #228

jkbhagatio · 2023-08-16T18:28:28Z

Updated base config files.

Please update your aeon python env! See the env*.yml files or requirements*.txt files or pyproject.toml file depending on what you use to manage your aeon python env!

Biggest noticeable dev changes will be pre-commit running auto-formatting and checks via black, ruff, and pyright.

Also github actions CI will run on pushes to some branches, and PRs to 'main'.

Summary by CodeRabbit

New Feature: Introduced GitHub Actions workflows for automated merging, squashing of commits, and running codebase checks and tests. This will enhance the development process and ensure code quality.
Refactor: Updated Python environment configuration files to include new tools for auto-formatting and code checks. This will help maintain a consistent coding style across the project.
Chore: Upgraded the default Python version to python3.11 and updated versions of pre-commit hooks. Added new pre-commit hooks for ruff and pyright. This will improve code security and maintainability.
New Feature: Added a new file for common preprocessing functions in aeon/preprocessing/init.py. This will streamline data preprocessing tasks.

@coderabbitai: ignore

JaerongA · 2023-08-16T20:04:13Z

@jkbhagatio Why not consolidate the dependency installment into pyproject.toml, like

...
[project.optional-dependencies]
dev = [
  "bandit"
  "black"
  "flake8"
    ... 
]

gpu = [
    "cupy",
    "dask"
]
...

and then a user can only run e.g., pip install -e ".[gpu]" ?

jkbhagatio · 2023-08-16T23:18:20Z

@JaerongA ah yeah sure, I could add a gpu section to pyproject.toml, so pip install could be done directly from pyproject.toml, good point.

I'd still keep the env yml files because afaik there's no way to conda/mamba install from a pyproject.toml?

I guess I can delete the requirements*.txt files though?

JaerongA · 2023-08-16T23:58:29Z

@jkbhagatio Sounds good.. yeah, it may only install from pip. Not sure how it can be tweaked to install from conda.

.pre-commit-config.yaml

.github/workflows/build_env_run_tests.yml

github-actions · 2023-08-27T01:31:17Z

CodeRabbit

Walkthrough

The proposed changes primarily focus on enhancing the development environment setup and introducing new workflows for better code management. The updates include new tools for auto-formatting, code checks, and CI integration. GitHub Actions workflows are introduced to automate merging and squashing commits from 'main' to 'prod', and for building the environment, running checks, tests, and generating coverage reports. The Python version is updated to python3.11 and pre-commit hooks are updated with new ones added. A new file for common preprocessing functions is also added.

Changes

File	Summary
Config	Updates related to development environment setup, including new tools for auto-formatting, code checks, and CI integration. Users are requested to update their Python environment configuration files.
.github/workflows/gpt_pr_review.yml, .github/workflows/squash_merge_to_prod.yml, .github/workflows/build_env_run_tests.yml	Introduces a GitHub Actions workflow that automatically merges and squashes commits from the 'main' branch into the 'prod' branch. Also, introduces a workflow for building the environment, running codebase checks and tests, and generating test coverage reports.
.pre-commit-config.yaml	Updates the base config files. Changes the default Python version to python3.11 and updates the versions of pre-commit hooks. Adds new pre-commit hooks for ruff and pyright. Includes commented-out setup for pytest-py.
aeon/preprocessing/init.py	Added a new file for common preprocessing functions. No changes to existing code.

Uplevel your code reviews with CodeRabbit Pro

CodeRabbit Pro

If you like this project, please support us by purchasing the Pro version. The Pro version has advanced context, superior noise reduction and several proprietary improvements compared to the open source version.

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 131f0fd and 747777b commits.

Files selected (9)

.flake8 (1)
.github/workflows/build_env_run_tests.yml (1)
.github/workflows/gpt_pr_review.yml (1)
.github/workflows/squash_merge_to_prod.yml (1)
.pre-commit-config.yaml (2)
aeon/io/reader.py (1)
env_config/.python-version (1)
readme.md (1)
requirements.txt (1)

Files ignored due to filter (5)

env.yml
env_config/env.yml
env_config/env_dev.yml
env_config/env_gpu.yml
pyproject.toml

Files not summarized due to errors (9)

.github/workflows/gpt_pr_review.yml (nothing obtained from openai)
.flake8 (nothing obtained from openai)
aeon/io/reader.py (nothing obtained from openai)
.github/workflows/build_env_run_tests.yml (nothing obtained from openai)
.github/workflows/squash_merge_to_prod.yml (nothing obtained from openai)
.pre-commit-config.yaml (nothing obtained from openai)
env_config/.python-version (nothing obtained from openai)
requirements.txt (nothing obtained from openai)
readme.md (nothing obtained from openai)

Files not reviewed due to errors (9)

.github/workflows/gpt_pr_review.yml (no response)
.flake8 (no response)
.github/workflows/squash_merge_to_prod.yml (no response)
.pre-commit-config.yaml (no response)
aeon/io/reader.py (no response)
.github/workflows/build_env_run_tests.yml (no response)
readme.md (no response)
requirements.txt (no response)
env_config/.python-version (no response)

Review comments generated (0)

Review: 0
LGTM: 0

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 747777b and 2f2af11 commits.

Files selected (2)

.github/workflows/gpt_pr_review.yml (1)
.github/workflows/squash_merge_to_prod.yml (1)

Files skipped from review due to trivial changes (1)

.github/workflows/squash_merge_to_prod.yml

Review comments generated (1)

Review: 0
LGTM: 1

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between 2f2af11 and ee35412 commits.

Files selected (4)

.github/workflows/build_env_run_tests.yml (1)
.github/workflows/squash_merge_to_prod.yml (1)
.pre-commit-config.yaml (2)
aeon/preprocessing/init.py (1)

Files ignored due to filter (2)

env_config/env_dev.yml
pyproject.toml

Files skipped from review due to trivial changes (2)

.github/workflows/squash_merge_to_prod.yml
aeon/preprocessing/init.py

Review comments generated (15)

Review: 13
LGTM: 2

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

.pre-commit-config.yaml

.github/workflows/build_env_run_tests.yml

github-actions

CodeRabbit

Commits

Files that changed from the base of the PR and between ee35412 and baec063 commits.

Files selected (1)

.github/workflows/build_env_run_tests.yml (1)

Files ignored due to filter (2)

pyproject.toml
setup.cfg

Review comments generated (5)

Review: 5
LGTM: 0

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

.github/workflows/build_env_run_tests.yml

glopesdev

Looks mostly okay, left a few comments for consideration and we can discuss further at the meeting. I have also renamed the PR for clarity. Happy to do a different title but just "Config" is too terse to include in release notes.

.github/workflows/build_env_run_tests.yml

glopesdev · 2023-08-30T10:03:48Z

.github/workflows/build_env_run_tests.yml

+
+      # Only run codebase checks and tests for ubuntu.
+      - name: ruff
+        if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'


Could we replace all instances of string 3.11 below with an environment variable to make it easier to change the python version for all the steps in the future?

It should be possible to do this using workflow environment variables: https://docs.github.com/en/actions/learn-github-actions/variables#defining-environment-variables-for-a-single-workflow

yup good idea, easy magic number to avoid

Oops, actually no! Looks like env variables are only supported for steps, but not for the strategy subfields in an actions workflow: https://stackoverflow.com/questions/74072206/github-actions-use-variables-in-matrix-definition

+1 code-rabbit =p

@jkbhagatio ah, failed to notice that, but we can still to get the values inside using matrix variables.

From the updated file it looks like you already did that right? So it should be all good now.

@jkbhagatio Another option, if we are really not going to run tests on any other python version in the matrix, would be to simply remove the checks altogether no? Python is already setup correctly using conda_incubator so why not just assume that these checks are not necessary?

glopesdev · 2023-08-30T10:07:06Z

.github/workflows/gpt_pr_review.yml

How long does it take to run this workflow? I still have mixed feelings about leaning on this even if only to generate PR summaries and recommendations. It failed to notice easy opportunities to refactor, such as the environment variables comment I made above, and my fear is always that this will discourage people from reading the code changes in detail.

At least on my end I will still do a full pass review by hand, but I guess if everyone is happy I am also happy to ignore it :)

Mm it was pretty quick, within a couple minutes, faster than build_env_run_tests

my fear is always that this will discourage people from reading the code changes in detail.
For me at least I'd actually say the opposite - it will probably make me read the code in more detail because I'll read it anyway and then probably pay extra attention to its suggestions.

Happy to revisit discussion on it in future after we see more examples of what it comes up with, but at least today people seemed intrigued : )

.github/workflows/squash_merge_to_prod.yml

docs/env_setup/local/miniconda_conda_local_setup.md

aeon/io/reader.py

aeon/processing/__init__.py

glopesdev · 2023-08-30T10:31:33Z

docs/devs/readme.md

+- Ensure dev dependencies are installed in your Aeon Python environment
+- Ensure pre-commit hooks are installed in your Aeon Python environment: run `pre-commit install` in the activated environment
+- Use [pyan3](https://github.com/Technologicat/pyan) for generating callgraphs


Can we elaborate here on what we are supposed to do with the output of pyan3? I can see how call graphs might be useful when reading the code base in some cases, but do we need to generate them when developing? Or is it more of a general tooling recommendation?

This was more of a reminder to myself. I found pyan when I wanted to generate diagrams for callgraphs for readers.

It's just a random point, but I can move it, any thoughts on where it should live?

Ok, decided just to remove this point from here (well now from 'contributing.md', given I've deleted this file)

glopesdev · 2023-08-30T10:35:25Z

docs/devs/readme.md

I was wondering if this README file should be renamed as CONTRIBUTING.md? Seems like these are guidelines for anyone wanting to contribute to the repo. If we use the standard GitHub naming convention it seems like GH will automatically add links to relevant places, e.g. issues and PRs: https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors#about-contributing-guidelines

.pre-commit-config.yaml

glopesdev

Almost there, just a few minor suggestions left.

glopesdev · 2023-08-31T08:05:09Z

.github/workflows/build_env_run_tests.yml

+
+      # Only run codebase checks and tests for ubuntu.
+      - name: ruff
+        if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'


@jkbhagatio Another option, if we are really not going to run tests on any other python version in the matrix, would be to simply remove the checks altogether no? Python is already setup correctly using conda_incubator so why not just assume that these checks are not necessary?

glopesdev · 2023-08-31T08:06:54Z

aeon/io/reader.py

Looks like a change is still being introduced here in that the white line was entirely removed. Maybe just revert the file to the version in the previous commit?

ah yeah sounds good

Another option, if we are really not going to run tests on any other python version in the matrix, would be to simply remove the checks altogether no?

yep, can just remove the python-version checks here

glopesdev · 2023-08-31T08:12:45Z

pyproject.toml

+description = '''
+    Code for managing acquired data from Project Aeon experiments. Includes general file IO,
+    data QC, querying, and analysis modules.
+'''
 authors = [
  { name = "Jai Bhagat", email = "[email protected]" },
  { name = "Goncalo Lopes", email = "[email protected]" },


Can you change my email here to g.lopes AT neurogears.org?

glopesdev · 2023-08-31T13:07:30Z

Sounds good, just the email change missing?

jkbhagatio · 2023-08-31T13:22:21Z

@glopesdev

Sounds good, just the email change missing?

It shows up here for me

glopesdev · 2023-08-31T17:05:48Z

These look good to me now, but maybe check what's up with ruff and the non-windows CI builds?

jkbhagatio · 2023-08-31T18:03:05Z

@glopesdev

These look good to me now, but maybe check what's up with ruff and the non-windows CI builds?

Yep the tests are failing on Ubuntu because ruff doesn't approve of formatting, and the build failing on macos is related to a DJ wheel not being built properly - something @ttngu207 is looking into

glopesdev · 2023-09-01T06:27:32Z

@jkbhagatio pyright still seems to be missing from pyproject.toml if I get this right? Might be easier to open both yml and toml side by side and make them identical.

Can conda bootstrap from pyproject.toml too? Would be preferable to have a single source of truth for dependencies.

jkbhagatio · 2023-09-01T08:08:37Z

@jkbhagatio pyright still seems to be missing from pyproject.toml if I get this right? Might be easier to open both yml and toml side by side and make them identical.

Can conda bootstrap from pyproject.toml too? Would be preferable to have a single source of truth for dependencies.

Oops, will fix that. No unfortunately, conda can't use pyproject as of now
conda/conda#10633

…hub actions

jkbhagatio requested review from glopesdev, lochhh, ttngu207 and JaerongA August 16, 2023 18:28

lochhh reviewed Aug 17, 2023

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

lochhh reviewed Aug 17, 2023

View reviewed changes

.github/workflows/build_env_run_tests.yml Show resolved Hide resolved

jkbhagatio marked this pull request as ready for review August 26, 2023 23:26

jkbhagatio mentioned this pull request Aug 27, 2023

Checklist for 'reformat' PR #235

Closed

6 tasks

github-actions bot reviewed Aug 27, 2023

View reviewed changes

github-actions bot reviewed Aug 28, 2023

View reviewed changes

jkbhagatio requested a review from lochhh August 28, 2023 21:00

glopesdev mentioned this pull request Aug 30, 2023

Add pose reader classes supporting ingestion of SLEAP tracking data #206

Merged

glopesdev changed the title ~~Config~~ Update base project config files Aug 30, 2023

glopesdev requested changes Aug 30, 2023

View reviewed changes

jkbhagatio requested a review from glopesdev August 31, 2023 00:23

glopesdev requested changes Aug 31, 2023

View reviewed changes

jkbhagatio requested a review from glopesdev August 31, 2023 12:49

glopesdev approved these changes Aug 31, 2023

View reviewed changes

jkbhagatio added 20 commits September 1, 2023 16:46

Cleaned up dev deps and added gpt-review and squash-merge-to-prod git…

2ef50a5

…hub actions

Updated actions workflows

058596e

Updated github actions

51a062b

fixed errors in pyproject.toml

729cdb1

Troubleshooting pip + pyproject env install via github actions

41c9ab8

Updated docs

148d2c8

Added some pre-commit checks

f206511

Updated docs

ad6bda6

Specified python version 3.11

d96101a

Undid formatting change

9d571ff

Removed 'processing' subpackage - will add in future PR

b6dceb5

Updated contributing.md

a019c28

Fixed py version in actions workflow

429ca0c

Fixed bug in actions workflow

e040d67

Removed python version checks

6c009ce

Reverted formatting change

a98b31d

Small doc fix

fbcd6d0

Updated black deps and pre-commit

033aff4

Fixed deps

7c46705

Addedd pyright dep

c6bafb4

jkbhagatio force-pushed the config branch from 22e6e74 to c6bafb4 Compare September 1, 2023 16:47

jkbhagatio merged commit f1ab685 into main Sep 1, 2023
5 of 7 checks passed

github-actions bot pushed a commit that referenced this pull request Sep 1, 2023

Merge pull request #228 from SainsburyWellcomeCentre/config

8bc586c

jkbhagatio linked an issue Oct 3, 2023 that may be closed by this pull request

Remove all non conda/mamba set-up instructions from docs #127

Closed

jkbhagatio deleted the config branch July 23, 2024 16:37

Update base project config files #228

Update base project config files #228

Conversation

jkbhagatio commented Aug 16, 2023 • edited Loading

Summary by CodeRabbit

JaerongA commented Aug 16, 2023

jkbhagatio commented Aug 16, 2023

JaerongA commented Aug 16, 2023

github-actions bot commented Aug 27, 2023 • edited Loading

Walkthrough

Changes

CodeRabbit Pro

github-actions bot left a comment

Choose a reason for hiding this comment

Chat with CodeRabbit Bot (@coderabbitai)

Code suggestions

Pausing incremental reviews

github-actions bot left a comment

Choose a reason for hiding this comment

Chat with CodeRabbit Bot (@coderabbitai)

Code suggestions

Pausing incremental reviews

github-actions bot left a comment

Choose a reason for hiding this comment

Chat with CodeRabbit Bot (@coderabbitai)

Code suggestions

Pausing incremental reviews

github-actions bot left a comment

Choose a reason for hiding this comment

Chat with CodeRabbit Bot (@coderabbitai)

Code suggestions

Pausing incremental reviews

glopesdev left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkbhagatio Aug 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glopesdev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glopesdev commented Aug 31, 2023

jkbhagatio commented Aug 31, 2023

glopesdev commented Aug 31, 2023

jkbhagatio commented Aug 31, 2023

glopesdev commented Sep 1, 2023

jkbhagatio commented Sep 1, 2023 • edited Loading

jkbhagatio commented Aug 16, 2023 •

edited

Loading

github-actions bot commented Aug 27, 2023 •

edited

Loading

Chat with CodeRabbit Bot (`@coderabbitai`)

Chat with CodeRabbit Bot (`@coderabbitai`)

Chat with CodeRabbit Bot (`@coderabbitai`)

Chat with CodeRabbit Bot (`@coderabbitai`)

glopesdev left a comment •

edited

Loading

jkbhagatio Aug 31, 2023 •

edited

Loading

jkbhagatio commented Sep 1, 2023 •

edited

Loading