Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not hardcode the numbers of sync nodes #75

Closed
3 tasks done
hellodword opened this issue Jun 28, 2024 · 3 comments
Closed
3 tasks done

Do not hardcode the numbers of sync nodes #75

hellodword opened this issue Jun 28, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@hellodword
Copy link
Contributor

Have you read a contributing guide?

  • I have read CONTRIBUTING.md
  • I have searched the existing requests and didn't find any that were similar
  • I have considered creating a pull request instead and want to proceed

Clear and concise description of the problem

ANY_SYNC_NODE_1_HOST=any-sync-node-1
ANY_SYNC_NODE_1_PORT=1001
ANY_SYNC_NODE_1_ADDRESSES=${ANY_SYNC_NODE_1_HOST}:${ANY_SYNC_NODE_1_PORT}
ANY_SYNC_NODE_1_QUIC_PORT=1011
ANY_SYNC_NODE_1_QUIC_ADDRESSES=${ANY_SYNC_NODE_1_HOST}:${ANY_SYNC_NODE_1_QUIC_PORT}
ANY_SYNC_NODE_2_HOST=any-sync-node-2
ANY_SYNC_NODE_2_PORT=1002
ANY_SYNC_NODE_2_ADDRESSES=${ANY_SYNC_NODE_2_HOST}:${ANY_SYNC_NODE_2_PORT}
ANY_SYNC_NODE_2_QUIC_PORT=1012
ANY_SYNC_NODE_2_QUIC_ADDRESSES=${ANY_SYNC_NODE_2_HOST}:${ANY_SYNC_NODE_2_QUIC_PORT}
ANY_SYNC_NODE_3_HOST=any-sync-node-3
ANY_SYNC_NODE_3_PORT=1003
ANY_SYNC_NODE_3_ADDRESSES=${ANY_SYNC_NODE_3_HOST}:${ANY_SYNC_NODE_3_PORT}
ANY_SYNC_NODE_3_QUIC_PORT=1013
ANY_SYNC_NODE_3_QUIC_ADDRESSES=${ANY_SYNC_NODE_3_HOST}:${ANY_SYNC_NODE_3_QUIC_PORT}

--t tree \
--t tree \
--t tree \

--addresses ${ANY_SYNC_NODE_1_ADDRESSES} \
--addresses ${ANY_SYNC_NODE_2_ADDRESSES} \
--addresses ${ANY_SYNC_NODE_3_ADDRESSES} \

yq --indent 2 --inplace ".account.signingKey |= \"${NETWORK_SIGNING_KEY}\"" account3.yml
yq --indent 2 --inplace ".account.signingKey |= \"${NETWORK_SIGNING_KEY}\"" account5.yml

cat "${NETWORK_FILE}" docker-generateconfig/etc/common.yml storage/docker-generateconfig/account3.yml docker-generateconfig/etc/coordinator.yml \
> ${DEST_PATH}/any-sync-coordinator/config.yml
echo "INFO: Generate config files for filenode"
cat "${NETWORK_FILE}" docker-generateconfig/etc/common.yml storage/docker-generateconfig/account4.yml docker-generateconfig/etc/filenode.yml \
> ${DEST_PATH}/any-sync-filenode/config.yml
echo "INFO: Generate config files for consensusnode"
cat "${NETWORK_FILE}" docker-generateconfig/etc/common.yml storage/docker-generateconfig/account5.yml docker-generateconfig/etc/consensusnode.yml \
> ${DEST_PATH}/any-sync-consensusnode/config.yml

Suggested solution

I'm not sure, maybe jinja2 is a good choice?

Alternative

No response

Additional context

No response

@hellodword hellodword added the enhancement New feature or request label Jun 28, 2024
@fb929
Copy link
Collaborator

fb929 commented Jun 28, 2024

this will be too complicated: as it will also require simultaneous templating of docker-compose.yml files, .env, and the generator, along with post-processing configurations.
moreover, this does not align with the goals of this repository

@hellodword
Copy link
Contributor Author

hellodword commented Jun 29, 2024

I think it is already too complicated. There are too many shell scripts and Python scripts, and hardcoding makes customization difficult.

Furthermore, are three nodes by default necessary for self-hosted and all-in-one users? Are there any design reasons for this?

Meanwhile, env.py and setListenIp.py are dynamic, they're good starts for removing hardcodings:

for name,value in envVars.items():
if re.match(r"^(ANY_SYNC_.*_PORT)$", name) and value == listenPort:
if re.match(r"^(ANY_SYNC_.*_QUIC_PORT)$", name):
# skip port, if PORT == QUIC_PORT
continue

@fb929
Copy link
Collaborator

fb929 commented Jul 10, 2024

The minimum number of any-sync-nodes required for the cluster to operate is 3.
Increasing this number is not practical because, architecturally, self-hosted can only operate within a single docker-host.
Additionally, at the moment, any-sync-node does not support rebalancing - you cannot change the number of nodes in an existing cluster, as this will lead to errors and potential data loss.

@fb929 fb929 closed this as completed Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants