From 7192e14d87d93767768e2baa774276f96da2d994 Mon Sep 17 00:00:00 2001 From: despiegk <> Date: Sat, 14 Oct 2023 17:50:14 +0300 Subject: [PATCH] mdbook for operations, ... --- book/src/tfgrid4_architecture/.collection | 1 + .../architecture/architecture.md | 96 ++++++++++++++++ .../architecture/concepts.md | 37 +++++++ .../tfgrid4_architecture/architecture/dsl.md | 73 ++++++++++++ .../architecture/tfgrid_actors.md | 104 ++++++++++++++++++ .../concepts/build_sandbox.md | 23 ++++ .../concepts/build_stack.md | 7 ++ .../tfgrid4_architecture/concepts/flist4.md | 20 ++++ book/src/tfgrid4_operations/.collection | 1 + .../src/tfgrid4_operations/operations_team.md | 31 ++++++ book/src/todo/.collection | 1 + book/src/todo/project_3bot.md | 39 +++++++ book/src/todo/story_sandbox.md | 75 +++++++++++++ book/src/todo/users.md | 18 +++ tfcloud/mycelium/mycelium.md | 9 +- tfcloud/mycelium/mycelium_faq.md | 11 +- 16 files changed, 539 insertions(+), 7 deletions(-) create mode 100644 book/src/tfgrid4_architecture/.collection create mode 100644 book/src/tfgrid4_architecture/architecture/architecture.md create mode 100644 book/src/tfgrid4_architecture/architecture/concepts.md create mode 100644 book/src/tfgrid4_architecture/architecture/dsl.md create mode 100644 book/src/tfgrid4_architecture/architecture/tfgrid_actors.md create mode 100644 book/src/tfgrid4_architecture/concepts/build_sandbox.md create mode 100644 book/src/tfgrid4_architecture/concepts/build_stack.md create mode 100644 book/src/tfgrid4_architecture/concepts/flist4.md create mode 100644 book/src/tfgrid4_operations/.collection create mode 100644 book/src/tfgrid4_operations/operations_team.md create mode 100644 book/src/todo/.collection create mode 100644 book/src/todo/project_3bot.md create mode 100644 book/src/todo/story_sandbox.md create mode 100644 book/src/todo/users.md diff --git a/book/src/tfgrid4_architecture/.collection b/book/src/tfgrid4_architecture/.collection new file mode 100644 index 0000000..273e3ab --- /dev/null +++ b/book/src/tfgrid4_architecture/.collection @@ -0,0 +1 @@ +name:architecture \ No newline at end of file diff --git a/book/src/tfgrid4_architecture/architecture/architecture.md b/book/src/tfgrid4_architecture/architecture/architecture.md new file mode 100644 index 0000000..942822f --- /dev/null +++ b/book/src/tfgrid4_architecture/architecture/architecture.md @@ -0,0 +1,96 @@ + + +## Circle + Actor + +```mermaid + + +graph TB + user1[Member of Circle OurAdmin] .->|3script\ne.g. through chat| actor1 + subgraph Circle1 [A Circle with name *OurAdmin*] + actor1[Actor] + modeldsl[Model DSL] + worlddsl[World DSL] + systemdsl[System DSL] + modeldb[Actor Mem DB] + actor1-->worlddsl + worlddsl-->modeldsl + modeldsl-->modeldb + worlddsl-->systemdsl + end + + +``` + +A Circle has a main actor which will have recipe's calling world DLS based on context. + +The World DSL abstracts access to model dsl as well as system dsl. + +The user which is a member of the circle, communicate to the circle actor in this case using chat and 3script. + +## Circle DB + Backend + +All structured data is stored in actor memory db and version controlled optionally through quantum safe storage system or a git based system. + +```mermaid +graph TB + subgraph Circle1 [A Circle with name *OurAdmin*] + actor1[Actor] + modeldsl[Model DSL] + worlddsl[World DSL] + systemdsl[System DSL] + modeldb[Actor Mem DB] + actor1-->worlddsl + worlddsl-->modeldsl + modeldsl-->modeldb + worlddsl-->systemdsl + end + + subgraph GitSystem1 [GitSystem 1] + git1[version controlled storage of
structured data and content] + end + + subgraph GitSystem2 [GitSystem 2] + git2[version controlled storage of
structured data and content] + end + + modeldb -->|git over mycelium| git1 + modeldb -->|git over mycelium| git2 + + +``` + +The redundant git stor at backend makes sure data data cannot be lost, multiple actors can work together, full version control is being achieved + +Mycelium is our secure P2P network. + + +```mermaid +graph TB + subgraph Circle1 [Circle 1] + actor1[Actor 1] + modeldb[Actor Mem DB] + end + subgraph Circle2 [Circle 1] + actor2[Actor 2] + modeldb2[Actor Mem DB] + end + + subgraph GitSystem1 [GitSystem 1] + git1[version controlled storage of
structured data and content] + end + + subgraph GitSystem2 [GitSystem 2] + git2[version controlled storage of
structured data and content] + end + + modeldb -->|git over mycelium| git1 + modeldb -->|git over mycelium| git2 + modeldb2 -->|git over mycelium| git1 + modeldb2 -->|git over mycelium| git2 + actor1 <--> actor2 + + +``` + + diff --git a/book/src/tfgrid4_architecture/architecture/concepts.md b/book/src/tfgrid4_architecture/architecture/concepts.md new file mode 100644 index 0000000..dad05e3 --- /dev/null +++ b/book/src/tfgrid4_architecture/architecture/concepts.md @@ -0,0 +1,37 @@ + + + + +## Actor DNA + +Is knowledge to teach actors how to deal with incoming requests. + +Its a collection of Recipes which are the registration of knowledge as is needed for the Actor. + +### How to give knowledge to an Actor + +The Actor DSL is a language allowing an expert to define Actor Recipe's which are teachings of knowledge around specific topic. E.g. an Actor can learn how to deal with a customer if a customer has an issue and is looking for support. + +The Actor DSL will use the Model DSL and System DSL to define the knowledge in. + +Think about it like a recipe for a cook in a kitchen. + +## Circle + + +- A group of people working together on a project (set of tasks, milestones, governance area, IT environment, …). +- Each Circle has a unique id, called CID = Circle ID +- Data models are often stored in mem and linked to a circle. + +## Actors & Circles + +Actors are linked to Circles. + +One actor can never operate in an external circle. + +Actors can send messages to other actors of other circles, this happens by default using 3 script messages. + +## 3script messages + +Each Circle Actor has a message queue, and incoming messages (3script) are parsed to actions which are given to right components in crystallib or other v library. + diff --git a/book/src/tfgrid4_architecture/architecture/dsl.md b/book/src/tfgrid4_architecture/architecture/dsl.md new file mode 100644 index 0000000..a1d64f8 --- /dev/null +++ b/book/src/tfgrid4_architecture/architecture/dsl.md @@ -0,0 +1,73 @@ + +## DSL's + +A Domain Specific Language. +We believe a DSL's need to be human readable and easy to expand. + +We use 3script and vlang to define our DSL's in. + +There are different types of DSLs + +- Model DSL = manage structured data, in a consistent way (Vlang & 3script) + - the memory of our digital world, uses rootobjects at the base +- System DSL = translate from a specific system/machine/tool to our vlang language (Vlang only) + - is a tool + - e.g. docker.machine_start would be interface to docker to get a machine started + - e.g. smtp.message_send is sending message in specific way +- World DSL = translate from a world usecase (can be tech or non tech), to the specific System / Model DSL (Vlang & 3script) + - is like an abstraction layer of the world, each actionr results in a recipe which is using the different DSL's + - e.g. cloud.machine_start would be world dsl, does not have to know about docker or KVM + - e.g. communication.message_send would be independent of implementation e.g. mail, sms, ... +- Circle DSL + - Each Circle can expose parts of a world DSL, rights can be defined on that world DSL. + - e.g. Admin's of Circle can do cloud.* but not a normal member, e.g. Consensus of Members needed to do ... + - Each Circle has 1 actor, this actor gets incoming messages (based on 3script) and will look for the right World DSL to execute on. + - Each Circle has its own memory structure in which the data can be kept, this data is organize din such a way that only the circle actor can access and modify the data. + +### Model DSL + +Is a language (3script or V) which allows actors to manipulate a data model. +All data is organized following rootobjects, the rootobjects can be loaded in memory and linked to a circle. + +- A root object is a complex data type, each rootobject as a unique ID (OID - Object ID). +- Each rootobject is linked to a Circle. +- 3Script or VLang can be used to manipulate (CREATE, DELETE, UPDATE) the Root Objects. + +Each RootObject has a factory through which we can do our crud actions + +- ingest 3script +- get, create, delete, update (will return copy) +- object to 3script +- ... + +#### Example root objects + +- User +- Contact +- Event (calendar) +- Story +- Milestone +- … + +### System DSL = SAL + +SAL = System Abstraction Layer + +Is like set of tools to manage a system e.g. manage virtual machines in a computer. The language used to manipulate a system is a DSL by itself but with specific purpose. + +SAL’s DON’T keep state, they are a tool, it's a language to be able to automate or work with systems. + +Examples would be + +- docker.container.stop +- docker.container.create + +### World DSL = WAL + +WAL = World Abstraction Layer + +Make a language which is independent of the physical implementation, will call the underlying model and system DSL's + + + +!!def_define name:'dsl,system_dsl,sal,dal,wal,model_dsl,actor_dsl' \ No newline at end of file diff --git a/book/src/tfgrid4_architecture/architecture/tfgrid_actors.md b/book/src/tfgrid4_architecture/architecture/tfgrid_actors.md new file mode 100644 index 0000000..1b23354 --- /dev/null +++ b/book/src/tfgrid4_architecture/architecture/tfgrid_actors.md @@ -0,0 +1,104 @@ + + +# TFGrid main actors + +A Circle actor has the capability to communicate wiht 3Nodes, which are computers running on the TFGrid. + +Each 3Node has an actor inside who has as function to manage the provisioning of compute, storage and network capacity. + +The 3Node actor will report back to a farming actor (see below) + +```mermaid + + +graph TB + subgraph Circle1 [A Circle with name *OurAdmin*] + actor1[Actor] + end + + subgraph 3Node1 [TFGrid 3Node] + actor_3node1[Actor for 3Node] + actor_3node1 .- compute1 + actor_3node1 .- storage1 + actor_3node1 .- network1 + compute1[Compute] + storage1[Storage] + network1[Network] + end + + subgraph 3Node2 [TFGrid 3Node] + actor_3node2[Actor for 3Node] + actor_3node2 .- compute2 + actor_3node2 .- storage2 + actor_3node2 .- network2 + compute2[Compute] + storage2[Storage] + network2[Network] + end + + subgraph 3Node3 [TFGrid 3Node] + actor_3node3[Actor for 3Node] + actor_3node3 .- network3 + actor_3node3 .- compute3 + actor_3node3 .- storage3 + compute3[Compute] + storage3[Storage] + network3[Network] + end + + network2 .-|Mycelium P2P Encrypted Network| network1 + network2 .-network3 + network1 .- network3 + actor1 ---|3script rpc over mycelium \nreliable message bus| actor_3node1 + actor1 --- actor_3node2 + actor1 --- actor_3node3 + +``` + +## farming actors + + +```mermaid + + +graph TB + subgraph Circle1 [A Circle with name *OurAdmin*] + actor1[Actor] + end + + subgraph 3Node1 [TFGrid 3Node] + actor_3node1[Actor for 3Node] + actor_3node1 .- compute1 + actor_3node1 .- storage1 + actor_3node1 .- network1 + compute1[Compute] + storage1[Storage] + network1[Network] + end + + subgraph 3Node2 [TFGrid 3Node] + actor_3node2[Actor for 3Node] + actor_3node2 .- compute2 + actor_3node2 .- storage2 + actor_3node2 .- network2 + compute2[Compute] + storage2[Storage] + network2[Network] + end + + subgraph FarmingActor [Farming Actor] + farmingactor[Actor for Farmer] + + end + + actor1 ---|3script rpc over mycelium \nreliable message bus| actor_3node1 + actor1 --- actor_3node2 + actor_3node1 -->|capacity utilization\nand availability| farmingactor + actor_3node2 --> farmingactor + actor1 -->|ask information of availability\nby means of 3script| farmingactor + +``` + +The Farming Actor will stor information of utilization & capacity in an internal DB (can be queried), so that every actor can find out how much capacity there is, is being used... + +The farming actor will also do monitoring. \ No newline at end of file diff --git a/book/src/tfgrid4_architecture/concepts/build_sandbox.md b/book/src/tfgrid4_architecture/concepts/build_sandbox.md new file mode 100644 index 0000000..04cec7a --- /dev/null +++ b/book/src/tfgrid4_architecture/concepts/build_sandbox.md @@ -0,0 +1,23 @@ +# Build Sandbox + +We build in a chroot environment, which is driven by SAL in V Crystallib. + +The Chroot environment is run like an OCI compatible container. + +## process how to build + +- build inside the sandbox +- the result sandbox gets cleaned up (remove non wanted files) +- export the sandbox to flist4 format +- store the flist4 format on chosen S3 server +- store the fileparts to chosen ZDB (future S3 server) + +Once + +## OCI = container specification + +Is a JSON spec in how to deploy a container, is low level but we have SAL for it in V. + + +!!tree.def_set name:'sandbox,build_sandbox,oci' description:'sandbox in which builds are done.' + diff --git a/book/src/tfgrid4_architecture/concepts/build_stack.md b/book/src/tfgrid4_architecture/concepts/build_stack.md new file mode 100644 index 0000000..28d0092 --- /dev/null +++ b/book/src/tfgrid4_architecture/concepts/build_stack.md @@ -0,0 +1,7 @@ + + +## installers + +- v scripts using our [System DSL's](crystallib:dsl.md) +- the v scripts run in a [sandbox](build_sandbox.md) +- V templates are used to generate the required configuration scripts. \ No newline at end of file diff --git a/book/src/tfgrid4_architecture/concepts/flist4.md b/book/src/tfgrid4_architecture/concepts/flist4.md new file mode 100644 index 0000000..728633d --- /dev/null +++ b/book/src/tfgrid4_architecture/concepts/flist4.md @@ -0,0 +1,20 @@ +# Flist 4 + +Our latest version of flists. + +The metadata of a filesystem is in SQLITE DB format and can easily be manipulated. +The data are deduped parts of files, which are stored on ZDB compatible backend (in future also S3 backends) + +There is a command line written in rust which allows us to + +- merge flist (TODO) +- import flist (dir to flist) +- export flist (flist to dir) + +remarks + +- the flist can be stored on any S3 or HTTP compatible webserver +- ZOS knows how to download the flist and mount the flist as filesystem and give to a virtual machine. + + +!!tree.def_set name:'flist4,flist' \ No newline at end of file diff --git a/book/src/tfgrid4_operations/.collection b/book/src/tfgrid4_operations/.collection new file mode 100644 index 0000000..8328b38 --- /dev/null +++ b/book/src/tfgrid4_operations/.collection @@ -0,0 +1 @@ +name:operations diff --git a/book/src/tfgrid4_operations/operations_team.md b/book/src/tfgrid4_operations/operations_team.md new file mode 100644 index 0000000..fcfa7d6 --- /dev/null +++ b/book/src/tfgrid4_operations/operations_team.md @@ -0,0 +1,31 @@ +# Operations Team + +## TFGrid Operators Manager + +The Operations manager needs to run a team of people in a global context, multi cultural skills will be important. + +Our aim is to highly automate our environments and make them as close as we can self healing. + +The team to build will consist out of + +- 4 automators (constantly advance our self healing system) +- 4 operators (monitor and learn from the systems outside, do corrective actions when needed, also add knowledge to our automation systems) + + +### skills + +We are looking for someone who has experience in running a technical IT environment. + +The system we run through is very unique and is based on very different paradigms. + +- experience with cloud or datacenters +- super organized nature +- not scare to do things in different ways +- people manager +- fast learner +- techncial knowledge + - networking principles + - security (good understanding of what can go wrong) + - how to achieve uptime of systems + - basic linux knowledge + - not afraid of reading, modifying some scripts if that would be needed (bash, ...) diff --git a/book/src/todo/.collection b/book/src/todo/.collection new file mode 100644 index 0000000..34ce289 --- /dev/null +++ b/book/src/todo/.collection @@ -0,0 +1 @@ +name:tfgrid_stories diff --git a/book/src/todo/project_3bot.md b/book/src/todo/project_3bot.md new file mode 100644 index 0000000..c2bb167 --- /dev/null +++ b/book/src/todo/project_3bot.md @@ -0,0 +1,39 @@ + +```js +!project.define + id:'***' + name:'3bot' + title:'finish our 3bot' + description: '...' + deadline:''//in ourtime format + state:'' //new,active,blocked,done,verified + +!milestone.define + id:'***' + name:'3bot_v05' + title:'version 0.5 of our 3bot' + deadline:'+30d'//means in 30 days from now + state:'active' + +!milestone.define + id:'***' + name:'3bot_v06' + title:'version 0.6 of our 3bot' + deadline:'+60d'//means in 30 days from now + state:'active' + +!milestone.define + id:'***' + name:'3bot_v07' + title:'version 0.7 of our 3bot' + deadline:'+90d'//means in 30 days from now + state:'active' + +// !team.define +// id:'***' +// name:'' +// title:'' +// description: '...' +// members: 'jan' //id's users who are part of team + +``` \ No newline at end of file diff --git a/book/src/todo/story_sandbox.md b/book/src/todo/story_sandbox.md new file mode 100644 index 0000000..f70fb9f --- /dev/null +++ b/book/src/todo/story_sandbox.md @@ -0,0 +1,75 @@ +# Build Environment In OCI Sandbox + +```js +!!tree.circle_select name:'3bot' + +!story.define + id:'***' + name:'oci_container' + project:'3bot' + title:'Build Environment In OCI Sandbox' + priority:'urgent' + deadline:'+15d' + owner:'jan' + assignment:'maxux,jan' + milestone:'3bot_v05' + description:' + see specs in info_cloud_production repo + ' +``` + +```js +!task.define id:'***' story:'oci_container' + title:'initial sandbox implementation in V' + priority:'urgent' assignment:'maxux' deadline:'+1d' effort_remaining:'10' + percent_done:'0%' state:'active' + description: ' + see V crystalib + lets proof the concept by installing e.g. caddy in such a sandbox + ' + +``` + +## Tasks + +```js +!task.define id:'***' story:'oci_container' + title:'initial sandbox implementation in V' + priority:'urgent' assignment:'maxux' deadline:'+1d' effort_remaining:'10' + percent_done:'0%' state:'active' + description:' + see V crystalib + lets proof the concept by installing e.g. caddy in such a sandbox + ' +``` + +```js +!task.define title:'export result to new style flist & run in example' + id:'***' story:'oci_container' percent_done:'0%' state:'new' + priority:'' assignment:'maxux' deadline:'' effort_remaining:'4' + description:' + - make sure we have good documentation + - can build flist tool, know where to get it + - run local zdb/webserver in tmux as backend for this flist operation + - mount the flist new sandox + - start this sandbox in tmux using V + - all as example + ' +``` + +## Requirements + +```js + +!requirement.define title:'documentation good enough, kristof signed off.' + id:'***' story:'oci_container' description: '' + +!requirement.define title:'sandboxing example in crystallib, example folder' + id:'***' story:'oci_container' description: '' + +!requirement.define title:'example how to use tmux with it' + id:'***' story:'oci_container' description: '' + + + +``` \ No newline at end of file diff --git a/book/src/todo/users.md b/book/src/todo/users.md new file mode 100644 index 0000000..11eeae6 --- /dev/null +++ b/book/src/todo/users.md @@ -0,0 +1,18 @@ +```js +!user.define + id:'***' + name: jan + alias: 'delandtj' + firstname: '' + lastname: '' + tags:'team:tftech_be' + +!contact:define + id:'***' + user:'jan' + email: 'jan@threefold.io' + 3bot: '' +``` + + +> TODO: complete \ No newline at end of file diff --git a/tfcloud/mycelium/mycelium.md b/tfcloud/mycelium/mycelium.md index c9058ca..75bdf12 100644 --- a/tfcloud/mycelium/mycelium.md +++ b/tfcloud/mycelium/mycelium.md @@ -3,17 +3,18 @@ # Mycelium: A Platform for Conscious Decentralization -> Mycelium runs on top of the TFGrid which is opensource Internet Infrastructure platform. +Mycelium has as purpose to restore the Internet to its original glory, a true Peer2Peer platform +capable to expand our collective consciousness. Mycelium allows human networks to flourish without centralized intermediaries. -Mycelium is a Internet Platform designed to become smarter and more creative together and expand collective consciousness. Built on proven technologies, Mycelium allows peer-to-peer networks to flourish without centralized intermediaries. +Mycelium uses the ThreeFold Grid as its base capacity cloud layer (storage, compute, network) which is opensource Internet Infrastructure platform. -Mycelium has no central point of control. The voluntary nodes forming the Mycelium network are operated by individuals and communities who share resources and capabilities to manifest a common vision. This structure aligns with principles of decentralization and autonomy. +Mycelium has no central point of control. Community provided nodes forming the Mycelium network are operated by individuals and communities who share resources and capabilities to manifest a common vision. Rich capabilities empower users to connect, communicate, transact and collaborate while retaining ownership of their own data and digital identities. Features like self-sovereign identity and encrypted communications enhance privacy and user agency. Incentive structures reward conscious contributions that regenerate communities and the planet. The Mycelium ecosystem cultivates an ethos of compassion, sustainability and abundance. -With Mycelium, we can rebuild critical systems - from currency to culture - in a more decentralized, conscious manner. Mycelium provides the infrastructure for a digitally-connected society aligned with our highest values. +With Mycelium, we can rebuild critical systems - from currency to culture - in a more decentralized, conscious manner. Mycelium provides a programmeable, affordable, secure and ultra scalable layer for a digitally-connected society aligned with our highest values. **A worldwide web of compassion is within reach if we take up the challenge.** diff --git a/tfcloud/mycelium/mycelium_faq.md b/tfcloud/mycelium/mycelium_faq.md index f4935e5..d7a6a13 100644 --- a/tfcloud/mycelium/mycelium_faq.md +++ b/tfcloud/mycelium/mycelium_faq.md @@ -1,6 +1,10 @@ # FAQ +## What is the link with ThreeFold. + +Mycelum is a platform connecting people and is using the TFGrid as its capacity layer (Distributed Cloud). Mycelium will have a dedicated website and dedicated team within ThreeFold Cloud or maybe later as separate company underneith the venture creator. + ## Is this a blockchain or crypto project NO, we are compatible with many blockchains but we don’t rely on them. We believe peer2peer and consensus driven decision systems with bridges to other blockchains are more powerful and way more scalable, secure and efficient. @@ -9,7 +13,9 @@ NO, we are compatible with many blockchains but we don’t rely on them. We beli We have been working on this for many many years over different of our own startups. We believe we have the pieces developed and ready to be integrated to a productized solution which can be used by many. -ThreeFold.IO can be seen as our 3e generation implementation of a peer2peer cloud. +- ThreeFold.IO can be seen as our 3e generation implementation of a peer2peer cloud. +- Mycelium with all its layers is the result of +20 years of experience +- We have working prototypes today of every required component. We are very proud of our team and grateful for the constant out of box thinking. @@ -19,8 +25,7 @@ Within 6 months is realistic, we are finalizing our next funding round and then ## How does Mycelium compare to? -* Urbit: a truly amazing project, many similar ideals and goals, we hope to be easier for people to start with and also be production quality faster. - +* Urbit: a truly amazing project, many similar ideals and goals. Urbit requires the developers to learn a lot of new concepts and development language. * Holochain: a very nice project, focussed on personal data pods as well, with intelligent decentralized peer2peer compute. Longer history than us. Good community. Maybe rather than compete we can even work together?