2021-07-21 planning discussion

[kgodey]

We're switching to a new format this week where each topic will be in its own top-level post and this post will only contain a list of topics. Please respond to each topic individually.

• New discussion format
• New monthly improvement milestones
• Retrospective
• Active work
• Milestone deadline updates
• Upcoming deadlines
• Data types and filters review

@mathemancer, @ghislaine, @pavish, and @eito-fis: your responses are required, but everyone is encouraged to comment if you have something to add, regardless of level of involvement with the project.

[kgodey]

New discussion format

We are going to be trying a new format for the meeting discussion where each section will be its own post. There is no dedicated "Questions" section any more. Instead, each section will have its own questions at the bottom.

What does everyone think of the new discussion format?

[pavish]

I think this format works a lot better.

[kgodey]

I like this format as well. I'm hoping it'll promote more discussion.

[ghislaineguerin]

I love the new format too. Thanks

[mathemancer]

Agreed. I think this will make responding more natural.

[kgodey]

New monthly improvement milestones

We often want to make general improvements to our code or processes that aren't necessarily related to one of the features in our milestones. We've been putting them in feature related milestones for now, but that makes it harder to track completion of a feature.

To solve the problem, I've created six new milestones, due at the end of every month. I've moved all the issues I could find that were general improvements to those.

• 2021-07 improvements
  • Now contains the following issues:
    • Design
      • Create button component in the component library.
    • Frontend
      • Serve built static files in production
    • Infrastructure
      • Set up staging environment for Mathesar
• 2021-08 improvements
  • Now contains the following issues:
    • Backend
      • Schema sqlalchemy engine caching is per-instance
    • Design
      • Create input component for the component library.
      • Create tabs component for the component library.
      • Create sidebar component for the component library
      • Create collapsable headers component for the component library.
    • Frontend
      • Frontend unit test cases: Readonly table view
      • Frontend component: Navigation Tree
      • Frontend component: Tabs
      • Frontend component: Pagination
      • Frontend utils: CancellablePromise, FilterUtils
      • Frontend components: Select
        • ARIA accessibility for Select component
        • Documentation for Select component
        • Unit tests for Select component
    • Infrastructure
      • Set up automatic deployment to staging server
      • Setup storybook deployment for frontend components
      • Setup API documentation and Postman collection
• 2021-09 improvements
  • Now contains the following issues:
    • Design
      • [Wiki] Add design toolbox section
      • Setup an Icon library with credits to creators
• 2021-10 improvements
• 2021-11 improvements
• 2021-12 improvements
  • Now contains the following issues:
    • Infrastructure
      • Setup bot for auto dependency updates for npm packages
      • Setup GH action to raise an issue when npm audit fails in workflow, instead of failing the workflow.
      • Set up GitHub Action to triage issues automatically when an issue has a status

Any thoughts about this change or the way I've allocated issues to it?

[mathemancer]

I think these allocations make sense for now.

Is it possible to automate bug reports going into these? If so, is it worth the trouble to automate?

[kgodey]

I don't think it's worth the effort to automate, triaging doesn't take me much time, and sometimes we do want bug reports to go into specific milestones (e.g. the grouping count bug that was fixed recently was essential to make grouping actually work properly).

[kgodey]

Retrospective

This is a summary of work we completed in the last week. See the end for questions.

• 02. Basic Filtering, Sorting, Grouping (Due: Jul 23)
  • PR for Spreadsheet view Spreadsheet view, sorting, grouping #363 merged
• 03. Editable Tables (Due: Jul 28)
  • No work this week.
• 2021-07 improvements (Due: Jul 31)
  • Created and documented script for pre-commit linter hook
  • Design work for button component in the Figma design library is ongoing and in review
  • Work on setting up a Mathesar staging server is almost complete
  • Work on setting up a frontend build to support the Mathesar staging server is underway
• 04. Working with Schemas (Due: Aug 02)
  • Design spec for working with schemas was finalized
  • Schema editing at DB & API levels
  • Schema deletion at DB & API levels
• 05. Tables from File Import (Due: Aug 09)
  • Design spec for import status indicators was finalized
  • PR Add column type targets to API #366 was merged.
    • This reorganized some things w.r.t. how columns are handled in the service, and also results in listing the valid target types to which a column can be altered in the relevant API views.
• 06. Working with Tables (Due: Aug 16)
  • Design spec for basic table operations was finalized
  • Design spec for working with columns and constraints was started and is in review
  • Table name editing at DB and API level
  • Table name deletion at DB and API level
  • Empty table creation via API
• 07. Initial Data Types (Due: Aug 23)
  • Lots of tickets were updated (split up into smaller pieces)
  • Design of data types has commenced, with text data type currently in review
• 08. Working with Views (Due: Aug 30)
  • Some new tickets were created to start speccing this milestone out, especially design tickets
  • Working with relationships was moved to this milestone from "Initial Data Types"
• 2021-08 improvements (Due: Aug 31)
  • Work on ARIA accessibility for Select component is underway
  • Frontend utils: CancellablePromise, FilterUtils is underway
• 10. Multiple Database Support (Due: Sep 06)
  • New databases endpoint

Questions:

• What went well this week?
• What could have gone better this week?
• Are there any process improvements we should consider implementing based on what happened this week?

[kgodey]

What went well this week?

I think we got a lot done. I'm glad that design is moving faster and I'm especially glad that we are way closer to having staging up than when I was working on it.

It's also nice to have people from outside the core team contributing. @dhruvkb got a couple of PRs merged, and it was nice to see the PR from @mr-gabe49 come in today. Thank you!

What could have gone better this week?

Creating issues took / is taking a lot more time than I anticipated, which isn't leaving me much time for other things. I think it's worth it because it will speed up implementation, but I'm looking forward to getting a couple of milestones ahead so that I can work on some of those issues.

Are there any process improvements we should consider implementing based on what happened this week?

Compiling the retrospective and active work portions of these updates is a pain. Since most people voted them useful last week, I think I may actually put some effort into automating them since it will save everybody time, not just me.

[pavish]

What went well this week?

The spreadsheet layout is now stabilised. I'm looking forward to help get staging up, so that @ghislaineguerin and I can be on the same page.

What could have gone better this week?

I've done considerable amount of work this week, but I'm inching very slowly towards the deadlines. I estimated the first few milestones to take a lot of time on the frontend, but it's more than anticipated. I am hoping to get things fast tracked by the mid milestones around 9 or 10, where we could be ahead of the deadlines.

Are there any process improvements we should consider implementing based on what happened this week?

None, from my end.

[ghislaineguerin]

What went well this week?

I think I'm finding it easier in defining just enough to start review and engage in a more collaborative process. Experimenting w quick calls w @kgodey also helped to speed up design.

What could have gone better this week?

I think I could spend a bit less time doing research for terminology and concepts, and post more questions to review when I'm unsure.

Are there any process improvements we should consider implementing based on what happened this week?

I think not.

[mathemancer]

What went well this week?

I consciously took less time on Github issues / comments / messaging, and more on implementation / coding. This felt more productive, and I was able to get more actual code merged as a result.

What could have gone better this week?

I dropped the ball on participating in this discussion in a timely manner. I'll try to pay more attention in the future.

Are there any process improvements we should consider implementing based on what happened this week?

None come to mind.

[kgodey]

Active Work

This is work planned for the next week.

• 02. Basic Filtering, Sorting, Grouping (Due: Jul 23)
  • @pavish: Sorting and grouping implementation in review
  • @pavish: Filtering is being implemented
  • @kgodey: Fix undesirable record grouping behavior
• 03. Editable Tables (Due: Jul 28)
  • @pavish: Implement record editing
  • @pavish: Implement record addition
  • @pavish: Implement record deletion
• 2021-07 improvements (Due: Jul 31)
  • @pavish: Frontend build setup for staging deloyment is ongoing
  • @andyharley-cci: Complete staging server setup once the frontend build setup is done
  • @Dhruvi16: Complete work on button component in Figma
• 04. Working with Schemas (Due: Aug 02)
  • @dhruvkb: Implement schema list page
  • @eito-fis: DB syncing is not triggered when objects are fetched without involving the API
  • @eito-fis: AttributeError if there's no schemas synced to the database
• 05. Tables from File Import (Due: Aug 09)
  • @mathemancer: Complete work for data checking / import preview API flow.
• 06. Working with Tables (Due: Aug 16)
  • @ghislaineguerin: Complete design spec for working with columns and constraints
  • @kgodey: Update GitHub issues in this milestone once design spec is complete
  • @eito-fis: API to upload file from URL
  • @eito-fis: Allow user to create a new table via pasting from spreadsheet
  • @kgodey: CRUD API for table constraints
• 07. Initial Data Types (Due: Aug 23)
  • @ghislaineguerin: Designs for data types
  • @kgodey: Update GitHub issues in this milestone as designs get finalized
  • @kgodey: Update GitHub issues to contain more details on how to handle type options
  • @powellc: Implementation of boolean type
• 08. Working with Views (Due: Aug 30)
  • @kgodey: Finish making backend and design issues for this milestone
• 09. Data Modeling (Due: Aug 30)
  • @kgodey: Finish making backend and design issues for this milestone
• 2021-08 improvements (Due: Aug 31)
  • @andyharley-cci: Complete setup of automatic deployment to staging
  • @Dhruvi16: Work on Input component in the design library
  • @Dhruvi16: Work on Tabs component in the design library
  • @mathemancer: Schema sqlalchemy engine caching is per-instance
  • @patelaryan7751: Improve code around frontend utilities
  • @mr-gabe49: ARIA accessibility for Select component

Questions and comments welcome.

[kgodey]

Milestone deadline updates

I went through and updated deadlines up to milestone 15 in the roadmap to be more generous. I'm not going to list all of the updated deadlines here - please look at the Roadmap for the new plan.

This does mean that the deadlines in September and October are more compressed, but I think they're still doable because:

• we're doing more foundational work in the first couple of months that should hopefully be complete by then.
• we're hiring more engineers and will hopefully have two more people on the team then.

Thoughts and feedback welcome!

[mathemancer]

I think it makes sense, especially with the increase in engineering team size.

[kgodey]

Upcoming deadlines

We have two deadlines due in the next week.

• 02. Basic Filtering, Sorting, Grouping (Due: Jul 23)
• 03. Editable Tables (Due: Jul 28)

@mathemancer pointed out that #352 is in Milestone 2 (due Friday) and was unassigned. Since it seems like a pretty small issue, I've assigned it to myself and I'll plan to get it done tomorrow if possible. @pavish, you have the rest of the issues – any blockers?

[pavish]

No blockers.

[kgodey]

Data types and filters review

Design for milestone 7, 'Initial Data Types', is in progress. Due to the scope of data types, the design will touch filter functionality again. Any implementation details about filters left unaddressed during milestone 2, 'Basic Filtering,' can be revisited. It might be of value to organize a discussion around layout details and sidebar states (visible vs. hidden).

This was written by @ghislaineguerin.