Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP][Dist] RAF Sharding - Phase 2 #123

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

[WIP][Dist] RAF Sharding - Phase 2 #123

wants to merge 9 commits into from

Conversation

Tonny-Gu
Copy link
Contributor

Description

(Brief description on what this PR is about)

This PR mainly contains the Expansion Rules and Infer Hints parts.

Checklist

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

cc @awslabs/raf-reviewer

@Tonny-Gu
Squashed commit of the following:
e22975b 
commit 4131923
Merge: 61f3dd0 9e36319
Author: NekoDaemon <[email protected]>
Date:   Tue Aug 23 14:28:20 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/raf into sharding-pr1

commit 61f3dd0
Merge: dd26fbc 3136651
Author: NekoDaemon <[email protected]>
Date:   Mon Aug 8 02:20:36 2022 +0800

    Merge branch 'awslabs:main' into sharding

commit dd26fbc
Author: NekoDaemon <[email protected]>
Date:   Wed Aug 3 18:57:55 2022 +0000

    impl

commit d7deb20
Author: NekoDaemon <[email protected]>
Date:   Sun Jul 31 19:10:08 2022 +0000

    impl

commit 6acdac5
Merge: c1ff936 9f26c8e
Author: NekoDaemon <[email protected]>
Date:   Mon Aug 1 02:48:04 2022 +0800

    Merge branch 'awslabs:main' into sharding

commit c1ff936
Author: NekoDaemon <[email protected]>
Date:   Sun Jul 31 18:46:46 2022 +0000

    impl

commit 4468f2d
Author: NekoDaemon <[email protected]>
Date:   Sun Jul 31 09:44:02 2022 +0000

    impl

commit e191d2b
Author: NekoDaemon <[email protected]>
Date:   Tue Jul 19 18:22:34 2022 +0000

    impl

commit 08f8fb0
Author: NekoDaemon <[email protected]>
Date:   Tue Jul 5 22:28:57 2022 +0000

    impl

commit ef45da3
Author: NekoDaemon <[email protected]>
Date:   Tue Jun 28 16:30:07 2022 +0000

    impl

commit 8e9e83e
Author: NekoDaemon <[email protected]>
Date:   Tue Jun 28 08:31:11 2022 +0000

    impl

commit 5394633
Author: NekoDaemon <[email protected]>
Date:   Tue Jun 14 17:52:43 2022 +0000

    fix

commit 65a896d
Author: NekoDaemon <[email protected]>
Date:   Tue Jun 14 17:52:23 2022 +0000

    fix

commit ce663a7
Author: NekoDaemon <[email protected]>
Date:   Tue Jun 14 16:29:39 2022 +0000

    refactor

commit 1727b4f
Merge: f124494 c8ddbc9
Author: NekoDaemon <[email protected]>
Date:   Sun Jun 12 16:14:20 2022 +0800

    Merge branch 'awslabs:main' into sharding

commit f124494
Merge: 10749ad f980111
Author: NekoDaemon <[email protected]>
Date:   Fri Apr 22 17:01:58 2022 +0000

    Merge branch 'main' of https://github.com/Tonny-Gu/raf into sharding

commit 10749ad
Author: NekoDaemon <[email protected]>
Date:   Fri Apr 22 17:00:42 2022 +0000

    refactor

commit adf58ec
Merge: ccadefc 4d1d033
Author: NekoDaemon <[email protected]>
Date:   Wed Apr 13 15:38:17 2022 +0000

    Merge branch 'main' of https://github.com/Tonny-Gu/raf into sharding

commit ccadefc
Author: NekoDaemon <[email protected]>
Date:   Mon Apr 11 15:34:48 2022 +0000

    test

commit 7869cf1
Author: NekoDaemon <[email protected]>
Date:   Sat Apr 2 12:26:00 2022 +0000

    impl

commit c2b9133
Author: NekoDaemon <[email protected]>
Date:   Sat Mar 19 10:09:48 2022 +0000

    refactor

commit 4c80afb
Author: NekoDaemon <[email protected]>
Date:   Fri Mar 18 18:42:43 2022 +0000

    trace shardspec

commit 2133a71
Author: NekoDaemon <[email protected]>
Date:   Fri Mar 18 09:19:49 2022 +0000

    refactor

commit 8a8b783
Author: NekoDaemon <[email protected]>
Date:   Thu Mar 17 16:12:21 2022 +0000

    refactor

commit bd00a6c
Author: NekoDaemon <[email protected]>
Date:   Wed Mar 16 10:21:37 2022 +0000

    lint

commit 8f590b8
Merge: ec654df b87a69d
Author: NekoDaemon <[email protected]>
Date:   Wed Mar 16 10:14:26 2022 +0000

    Merge branch 'multi-comm' of github.com:Tonny-Gu/meta into sharding

commit ec654df
Merge: 0e74b40 1899126
Author: NekoDaemon <[email protected]>
Date:   Wed Mar 16 16:18:34 2022 +0800

    Merge branch 'meta-project:main' into sharding

commit b87a69d
Merge: 4b3ba0a 1899126
Author: NekoDaemon <[email protected]>
Date:   Wed Mar 16 16:18:25 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 4b3ba0a
Merge: 0e06a41 df740a1
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 15 20:18:54 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 0e06a41
Merge: c58a495 7afd415
Author: NekoDaemon <[email protected]>
Date:   Fri Mar 11 23:29:36 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit c58a495
Author: NekoDaemon <[email protected]>
Date:   Fri Mar 11 15:29:16 2022 +0000

    fix

commit bd2c3d7
Merge: ad29efa aa423a2
Author: NekoDaemon <[email protected]>
Date:   Wed Mar 9 17:39:40 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit ad29efa
Author: NekoDaemon <[email protected]>
Date:   Wed Mar 9 09:38:52 2022 +0000

    clean

commit 66d2984
Merge: 2df68d2 20eb5a1
Author: NekoDaemon <[email protected]>
Date:   Wed Mar 9 01:11:11 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 2df68d2
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 8 17:10:37 2022 +0000

    nested tuple

commit b22ba52
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 8 09:25:07 2022 +0000

    fix

commit 0e74b40
Merge: 44214ac 20eb5a1
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 8 16:38:02 2022 +0800

    Merge branch 'meta-project:main' into sharding

commit 995bb1f
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 8 08:37:16 2022 +0000

    nested tuple

commit eefd9eb
Merge: 5407151 279dba8
Author: NekoDaemon <[email protected]>
Date:   Sat Mar 5 14:59:25 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 5407151
Author: NekoDaemon <[email protected]>
Date:   Sat Mar 5 06:56:58 2022 +0000

    clean

commit 44214ac
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 1 14:47:15 2022 +0000

    fix

commit dfc60f2
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 1 12:32:16 2022 +0000

    merge

commit 86b4e69
Merge: 46eec3e df572d2
Author: NekoDaemon <[email protected]>
Date:   Tue Mar 1 12:22:52 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into sharding

commit c924e80
Merge: efea2ff df572d2
Author: NekoDaemon <[email protected]>
Date:   Sun Feb 27 22:52:29 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit efea2ff
Author: NekoDaemon <[email protected]>
Date:   Fri Feb 25 15:06:46 2022 +0000

    fix

commit 43b2100
Author: NekoDaemon <[email protected]>
Date:   Fri Feb 25 14:59:30 2022 +0000

    merge

commit 4f95e2a
Merge: 1dbe464 654a121
Author: NekoDaemon <[email protected]>
Date:   Fri Feb 25 14:48:24 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into multi-comm

commit 46eec3e
Merge: 5989fe1 7e595b5
Author: NekoDaemon <[email protected]>
Date:   Tue Feb 22 13:29:35 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into sharding

commit 1dbe464
Author: NekoDaemon <[email protected]>
Date:   Tue Feb 22 13:12:54 2022 +0000

    clean

commit 6e8a865
Merge: a4deaed 7e595b5
Author: NekoDaemon <[email protected]>
Date:   Tue Feb 22 19:56:18 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit a4deaed
Author: NekoDaemon <[email protected]>
Date:   Sat Feb 19 08:26:22 2022 +0000

    fix

commit 1bff858
Author: NekoDaemon <[email protected]>
Date:   Sat Feb 19 08:01:51 2022 +0000

    license

commit 9032d21
Merge: 8e02780 8a941b8
Author: NekoDaemon <[email protected]>
Date:   Sat Feb 19 07:51:56 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into multi-comm

commit 8e02780
Merge: 1f994f0 e736065
Author: NekoDaemon <[email protected]>
Date:   Sat Feb 19 07:41:58 2022 +0000

    Merge branch 'comm-refactor' of github.com:Tonny-Gu/meta into multi-comm

commit e736065
Author: NekoDaemon <[email protected]>
Date:   Sat Feb 19 07:39:41 2022 +0000

    fix

commit 31e20ff
Author: NekoDaemon <[email protected]>
Date:   Fri Feb 11 15:13:46 2022 +0000

    refactor

commit ec73ffd
Merge: 3395d68 aa3b2a8
Author: NekoDaemon <[email protected]>
Date:   Fri Feb 11 14:51:48 2022 +0800

    Merge branch 'meta-project:main' into comm-refactor

commit 1f994f0
Merge: 3395d68 aa3b2a8
Author: NekoDaemon <[email protected]>
Date:   Fri Feb 11 14:51:35 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 3395d68
Merge: e9ede20 d0da258
Author: NekoDaemon <[email protected]>
Date:   Thu Feb 10 17:52:48 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit e9ede20
Merge: ef2138e ba516c2
Author: NekoDaemon <[email protected]>
Date:   Wed Feb 9 22:13:06 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit ef2138e
Author: NekoDaemon <[email protected]>
Date:   Wed Feb 9 13:23:02 2022 +0000

    fix

commit 967ab79
Author: NekoDaemon <[email protected]>
Date:   Wed Feb 9 12:09:54 2022 +0000

    fix

commit 026fc34
Author: NekoDaemon <[email protected]>
Date:   Wed Feb 9 04:06:38 2022 +0000

    lint

commit b62320b
Merge: b676af6 08f6ffa
Author: NekoDaemon <[email protected]>
Date:   Wed Feb 9 11:58:16 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit b676af6
Author: NekoDaemon <[email protected]>
Date:   Wed Feb 9 03:57:25 2022 +0000

    fix gpu test

commit 6b1bc9b
Author: NekoDaemon <[email protected]>
Date:   Tue Feb 8 15:27:46 2022 +0000

    fix cpu test

commit 4c5b083
Merge: bccc991 41fc64e
Author: NekoDaemon <[email protected]>
Date:   Tue Feb 8 10:15:11 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into multi-comm

commit bccc991
Author: NekoDaemon <[email protected]>
Date:   Tue Jan 4 07:01:26 2022 +0000

    lint

commit e4d256e
Author: NekoDaemon <[email protected]>
Date:   Tue Jan 4 06:49:18 2022 +0000

    lint

commit 3f03bff
Author: NekoDaemon <[email protected]>
Date:   Mon Jan 3 16:37:44 2022 +0000

    lint

commit 8d8d898
Author: NekoDaemon <[email protected]>
Date:   Mon Jan 3 13:59:26 2022 +0000

    implement

commit 230828d
Author: NekoDaemon <[email protected]>
Date:   Mon Jan 3 13:20:55 2022 +0000

    implement

commit a671d38
Author: NekoDaemon <[email protected]>
Date:   Mon Jan 3 10:31:47 2022 +0000

    implement

commit af7b85b
Author: NekoDaemon <[email protected]>
Date:   Sun Jan 2 17:07:56 2022 +0000

    test & lint

commit b40e9d1
Author: NekoDaemon <[email protected]>
Date:   Sun Jan 2 16:33:22 2022 +0000

    implement

commit a8b875e
Author: NekoDaemon <[email protected]>
Date:   Sat Jan 1 15:13:12 2022 +0000

    test

commit b93a6ab
Merge: e6548da 6215bfc
Author: NekoDaemon <[email protected]>
Date:   Sat Jan 1 15:02:43 2022 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into multi-comm

commit e6548da
Author: NekoDaemon <[email protected]>
Date:   Sat Jan 1 14:23:57 2022 +0000

    implement

commit 47701b5
Author: NekoDaemon <[email protected]>
Date:   Sun Dec 19 12:27:35 2021 +0000

    implement

commit e45b60a
Author: NekoDaemon <[email protected]>
Date:   Tue Dec 14 08:19:30 2021 +0000

    implement

commit 56409b7
Merge: c8c7381 5ca2012
Author: NekoDaemon <[email protected]>
Date:   Mon Dec 13 14:56:37 2021 +0800

    Merge branch 'meta-project:master' into multi-comm

commit c8c7381
Author: NekoDaemon <[email protected]>
Date:   Mon Dec 13 06:54:48 2021 +0000

    update

commit 97f5aac
Author: NekoDaemon <[email protected]>
Date:   Sat Dec 11 08:21:32 2021 +0000

    update

commit 5d1a7b9
Author: NekoDaemon <[email protected]>
Date:   Fri Dec 10 16:12:34 2021 +0000

    fix

commit 5989fe1
Author: NekoDaemon <[email protected]>
Date:   Tue Nov 9 06:25:30 2021 +0000

    merge

commit 61ae89b
Merge: 4bbd779 5114fb6
Author: NekoDaemon <[email protected]>
Date:   Tue Nov 9 06:24:05 2021 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into sharding

commit 4bbd779
Author: NekoDaemon <[email protected]>
Date:   Tue Nov 9 05:57:41 2021 +0000

    refactor

commit 3e2a3ae
Author: NekoDaemon <[email protected]>
Date:   Fri Oct 29 03:45:31 2021 +0000

    run!

commit 11d08d2
Author: NekoDaemon <[email protected]>
Date:   Tue Oct 26 13:40:00 2021 +0000

    implement

commit cf306f7
Author: NekoDaemon <[email protected]>
Date:   Tue Oct 19 16:13:52 2021 +0000

    implement

commit 527141f
Merge: 8e4271e eae1132
Author: NekoDaemon <[email protected]>
Date:   Tue Oct 19 00:54:47 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 8e4271e
Author: NekoDaemon <[email protected]>
Date:   Mon Oct 18 16:53:41 2021 +0000

    implement

commit c22c5a0
Merge: 75ddb68 c25a152
Author: NekoDaemon <[email protected]>
Date:   Thu Oct 14 05:47:56 2021 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into sharding

commit 75ddb68
Author: NekoDaemon <[email protected]>
Date:   Thu Oct 14 05:45:48 2021 +0000

    implement

commit 971c93e
Author: NekoDaemon <[email protected]>
Date:   Tue Sep 28 09:05:02 2021 +0000

    refactor

commit c67fbdf
Author: NekoDaemon <[email protected]>
Date:   Tue Sep 28 07:49:53 2021 +0000

    implement

commit c676ee5
Author: NekoDaemon <[email protected]>
Date:   Wed Sep 22 15:57:38 2021 +0000

    implement

commit 9b30495
Merge: dea0149 b5845ae
Author: NekoDaemon <[email protected]>
Date:   Wed Sep 22 15:16:01 2021 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into sharding

commit dea0149
Author: NekoDaemon <[email protected]>
Date:   Wed Sep 15 13:05:12 2021 +0000

    implement

commit c28de27
Merge: 6137b98 caa3650
Author: NekoDaemon <[email protected]>
Date:   Mon Sep 13 22:41:22 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 6137b98
Author: NekoDaemon <[email protected]>
Date:   Mon Sep 13 14:39:51 2021 +0000

    implement

commit 7587a1f
Author: Tonny-Gu <[email protected]>
Date:   Sat Sep 11 13:51:37 2021 +0000

    implement

commit 7315d93
Author: Tonny-Gu <[email protected]>
Date:   Fri Sep 10 03:58:02 2021 +0000

    fix bug

commit b26566a
Merge: 7d0a519 8b24b09
Author: NekoDaemon <[email protected]>
Date:   Thu Sep 9 14:44:45 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 7d0a519
Author: Tonny-Gu <[email protected]>
Date:   Thu Sep 9 06:43:40 2021 +0000

    refactor and fix bug

commit 25bb65d
Author: Tonny-Gu <[email protected]>
Date:   Tue Sep 7 09:41:08 2021 +0000

    implement

commit deb3d43
Author: Tonny-Gu <[email protected]>
Date:   Mon Sep 6 03:52:12 2021 +0000

    fix bug

commit a08fd84
Merge: af6df28 469357d
Author: NekoDaemon <[email protected]>
Date:   Mon Sep 6 11:24:59 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit af6df28
Merge: ebf6dc6 08dcf2e
Author: NekoDaemon <[email protected]>
Date:   Mon Aug 30 15:12:02 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit ebf6dc6
Merge: 94f90ba 0110e38
Author: NekoDaemon <[email protected]>
Date:   Fri Aug 27 15:17:31 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 94f90ba
Merge: ec43e77 c87ae8e
Author: Tonny-Gu <[email protected]>
Date:   Thu Aug 26 09:27:10 2021 +0000

    Merge remote-tracking branch 'origin/master' into sharding

commit ec43e77
Author: Tonny-Gu <[email protected]>
Date:   Thu Aug 26 09:21:44 2021 +0000

    implement

commit 50a90f9
Author: Tonny-Gu <[email protected]>
Date:   Mon Aug 23 11:34:13 2021 +0000

    implement

commit 4a50d9d
Author: Tonny-Gu <[email protected]>
Date:   Sat Aug 21 08:04:06 2021 +0000

    extend shardspec family

commit 38d46bd
Merge: 595cd9c 3ed4598
Author: NekoDaemon <[email protected]>
Date:   Thu Aug 19 17:48:56 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 595cd9c
Author: Tonny-Gu <[email protected]>
Date:   Thu Aug 19 09:23:58 2021 +0000

    implement

commit 637d4aa
Merge: 77ad1b2 f6f87a3
Author: NekoDaemon <[email protected]>
Date:   Wed Aug 18 14:47:41 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 77ad1b2
Merge: 3de8293 e0e003e
Author: NekoDaemon <[email protected]>
Date:   Tue Aug 17 18:13:44 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 3de8293
Author: Tonny-Gu <[email protected]>
Date:   Tue Aug 17 10:09:42 2021 +0000

    implement

commit d8cf092
Author: Tonny-Gu <[email protected]>
Date:   Mon Aug 16 12:32:30 2021 +0000

    add ShardSpec, ShardOpAttrs, InitShardOpAttrs Pass
@Tonny-Gu
lint
2c556e7
@Tonny-Gu Tonny-Gu mentioned this pull request Aug 23, 2022
Open
@Tonny-Gu
Copy link
Contributor Author

@hgt312 @comaniac This PR is ready except for the unit test part.

hgt312
hgt312 reviewed Sep 28, 2022
View reviewed changes
include/raf/pass.h
Pass ExpandShardOpCall();

/*!
* \brief .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing doc

src/pass/sharding.cc

namespace shard_pass {

class ShardOpCallAttrsSetter : public ExprMutator {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider using mixedmodemutator/exprrewriter for these passes

@Tonny-Gu Tonny-Gu mentioned this pull request Nov 1, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hgt312 hgt312 hgt312 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Tonny-Gu @hgt312