Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

State changes collector #5585

Merged
merged 18 commits into from
Sep 4, 2024
Merged

State changes collector #5585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
00b54ef
collect state changes
BeniaminDrasovean Sep 8, 2023
75dff98
add more unit tests
BeniaminDrasovean Sep 13, 2023
ed0f9d6
get state changes for each transaction
BeniaminDrasovean Sep 14, 2023
a05808d
pass stateChangesCollector as argument
BeniaminDrasovean Sep 15, 2023
db454f1
add flag for collectStateChanges
BeniaminDrasovean Sep 15, 2023
01d832b
add comments and unit test for stateChangesCollector
BeniaminDrasovean Sep 15, 2023
49a8c8a
Merge branch 'rc/v1.6.0' into state-changes-collector
BeniaminDrasovean Sep 18, 2023
676a23c
Merge pull request #5879 from multiversx/merge-rc/v1.7.0-in-feat/ligh…
iulianpascalau Jan 26, 2024
39eaa54
Merge remote-tracking branch 'origin/rc/v1.7.0' into state-changes-co…
BeniaminDrasovean Jan 26, 2024
7e6aa45
fix after merge
BeniaminDrasovean Jan 26, 2024
820264b
Merge branch 'feat/light-client-support' into state-changes-collector
BeniaminDrasovean Jan 26, 2024
33b6daf
correlate state changes to a tx hash
BeniaminDrasovean Jan 30, 2024
93c0e5b
Merge remote-tracking branch 'origin/rc/v1.7.0' into merge-rc/1.7.0-i…
BeniaminDrasovean Mar 18, 2024
edcbe7b
Merge pull request #6049 from multiversx/merge-rc/1.7.0-in-feat/light…
BeniaminDrasovean Mar 18, 2024
858ae75
Merge branch 'feat/light-client-support' into state-changes-collector
BeniaminDrasovean Mar 18, 2024
5331072
fix after merge
BeniaminDrasovean Mar 18, 2024
9836fde
fixes after review
BeniaminDrasovean Sep 4, 2024
5bf6ee7
Merge branch 'feat/state-changes' into state-changes-collector
BeniaminDrasovean Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cmd/node/config/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -662,6 +662,7 @@
MaxStateTrieLevelInMemory = 5
MaxPeerTrieLevelInMemory = 5
StateStatisticsEnabled = false
CollectStateChangesEnabled = false

[BlockSizeThrottleConfig]
MinSizeInBytes = 104857 # 104857 is 10% from 1MB
Expand Down
1 change: 1 addition & 0 deletions config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,7 @@ type StateTriesConfig struct {
SnapshotsEnabled bool
AccountsStatePruningEnabled bool
PeerStatePruningEnabled bool
CollectStateChangesEnabled bool
MaxStateTrieLevelInMemory uint
MaxPeerTrieLevelInMemory uint
StateStatisticsEnabled bool
Expand Down
9 changes: 9 additions & 0 deletions epochStart/bootstrap/disabled/disabledAccountsAdapter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,15 @@ func (a *accountsAdapter) GetStackDebugFirstEntry() []byte {
return nil
}

// SetTxHashForLatestStateChanges -
func (a *accountsAdapter) SetTxHashForLatestStateChanges(_ []byte) {
}

// ResetStateChangesCollector -
func (a *accountsAdapter) ResetStateChangesCollector() []state.StateChangesForTx {
return nil
}

// Close -
func (a *accountsAdapter) Close() error {
return nil
Expand Down
1 change: 1 addition & 0 deletions epochStart/metachain/systemSCs_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -763,6 +763,7 @@ func createAccountsDB(
StoragePruningManager: spm,
AddressConverter: &testscommon.PubkeyConverterMock{},
SnapshotsManager: disabledState.NewDisabledSnapshotsManager(),
StateChangesCollector: disabledState.NewDisabledStateChangesCollector(),
}
adb, _ := state.NewAccountsDB(args)
return adb
Expand Down
1 change: 1 addition & 0 deletions factory/api/apiResolverFactory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -601,6 +601,7 @@ func createNewAccountsAdapterApi(args scQueryElementArgs, chainHandler data.Chai
StoragePruningManager: storagePruning,
AddressConverter: args.coreComponents.AddressPubKeyConverter(),
SnapshotsManager: disabledState.NewDisabledSnapshotsManager(),
StateChangesCollector: disabledState.NewDisabledStateChangesCollector(),
}

provider, err := blockInfoProviders.NewCurrentBlockInfo(chainHandler)
Expand Down
1 change: 1 addition & 0 deletions factory/processing/blockProcessorCreator_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ func createAccountAdapter(
StoragePruningManager: disabled.NewDisabledStoragePruningManager(),
AddressConverter: &testscommon.PubkeyConverterMock{},
SnapshotsManager: disabledState.NewDisabledSnapshotsManager(),
StateChangesCollector: disabledState.NewDisabledStateChangesCollector(),
}
adb, err := state.NewAccountsDB(args)
if err != nil {
Expand Down
41 changes: 27 additions & 14 deletions factory/state/stateComponents.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,11 @@ func (scf *stateComponentsFactory) createAccountsAdapters(triesContainer common.
return nil, nil, nil, err
}

stateChangesCollector := disabled.NewDisabledStateChangesCollector()
if scf.config.StateTriesConfig.CollectStateChangesEnabled {
stateChangesCollector = state.NewStateChangesCollector()
}

argStateMetrics := stateMetrics.ArgsStateMetrics{
SnapshotInProgressKey: common.MetricAccountsSnapshotInProgress,
LastSnapshotDurationKey: common.MetricLastAccountsSnapshotDurationSec,
Expand All @@ -168,13 +173,14 @@ func (scf *stateComponentsFactory) createAccountsAdapters(triesContainer common.
}

argsProcessingAccountsDB := state.ArgsAccountsDB{
Trie: merkleTrie,
Hasher: scf.core.Hasher(),
Marshaller: scf.core.InternalMarshalizer(),
AccountFactory: accountFactory,
StoragePruningManager: storagePruning,
AddressConverter: scf.core.AddressPubKeyConverter(),
SnapshotsManager: snapshotsManager,
Trie: merkleTrie,
Hasher: scf.core.Hasher(),
Marshaller: scf.core.InternalMarshalizer(),
AccountFactory: accountFactory,
StoragePruningManager: storagePruning,
AddressConverter: scf.core.AddressPubKeyConverter(),
SnapshotsManager: snapshotsManager,
StateChangesCollector: stateChangesCollector,
}
accountsAdapter, err := state.NewAccountsDB(argsProcessingAccountsDB)
if err != nil {
Expand All @@ -189,6 +195,7 @@ func (scf *stateComponentsFactory) createAccountsAdapters(triesContainer common.
StoragePruningManager: storagePruning,
AddressConverter: scf.core.AddressPubKeyConverter(),
SnapshotsManager: disabled.NewDisabledSnapshotsManager(),
StateChangesCollector: disabled.NewDisabledStateChangesCollector(),
}

accountsAdapterApiOnFinal, err := factoryState.CreateAccountsAdapterAPIOnFinal(argsAPIAccountsDB, scf.chainHandler)
Expand Down Expand Up @@ -228,6 +235,11 @@ func (scf *stateComponentsFactory) createPeerAdapter(triesContainer common.Tries
return nil, err
}

stateChangesCollector := disabled.NewDisabledStateChangesCollector()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should have the state changes collected since the genesis if we only use these to export the data and index it.
Then use a flag for when we add the changes to the block.

There can also be a config flag (not enable epoch), if we want it to be exported from the node (through the outport driver) or not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was refactored in the PR from Darius

if scf.config.StateTriesConfig.CollectStateChangesEnabled {
stateChangesCollector = state.NewStateChangesCollector()
}

argStateMetrics := stateMetrics.ArgsStateMetrics{
SnapshotInProgressKey: common.MetricPeersSnapshotInProgress,
LastSnapshotDurationKey: common.MetricLastPeersSnapshotDurationSec,
Expand All @@ -244,13 +256,14 @@ func (scf *stateComponentsFactory) createPeerAdapter(triesContainer common.Tries
}

argsProcessingPeerAccountsDB := state.ArgsAccountsDB{
Trie: merkleTrie,
Hasher: scf.core.Hasher(),
Marshaller: scf.core.InternalMarshalizer(),
AccountFactory: accountFactory,
StoragePruningManager: storagePruning,
AddressConverter: scf.core.AddressPubKeyConverter(),
SnapshotsManager: snapshotManager,
Trie: merkleTrie,
Hasher: scf.core.Hasher(),
Marshaller: scf.core.InternalMarshalizer(),
AccountFactory: accountFactory,
StoragePruningManager: storagePruning,
AddressConverter: scf.core.AddressPubKeyConverter(),
SnapshotsManager: snapshotManager,
StateChangesCollector: stateChangesCollector,
}
peerAdapter, err := state.NewPeerAccountsDB(argsProcessingPeerAccountsDB)
if err != nil {
Expand Down
5 changes: 3 additions & 2 deletions genesis/mock/userAccountMock.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (

"github.com/multiversx/mx-chain-core-go/core"
"github.com/multiversx/mx-chain-go/common"
"github.com/multiversx/mx-chain-go/state"
)

// ErrNegativeValue -
Expand Down Expand Up @@ -147,8 +148,8 @@ func (uam *UserAccountMock) GetUserName() []byte {
}

// SaveDirtyData -
func (uam *UserAccountMock) SaveDirtyData(_ common.Trie) ([]core.TrieData, error) {
return nil, nil
func (uam *UserAccountMock) SaveDirtyData(_ common.Trie) ([]state.DataTrieChange, []core.TrieData, error) {
return nil, nil, nil
}

// IsGuarded -
Expand Down
1 change: 1 addition & 0 deletions genesis/process/memoryComponents.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func createAccountAdapter(
StoragePruningManager: disabled.NewDisabledStoragePruningManager(),
AddressConverter: addressConverter,
SnapshotsManager: disabledState.NewDisabledSnapshotsManager(),
StateChangesCollector: state.NewStateChangesCollector(),
}

adb, err := state.NewAccountsDB(args)
Expand Down
2 changes: 2 additions & 0 deletions integrationTests/state/stateTrie/stateTrie_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1083,6 +1083,7 @@ func createAccounts(
StoragePruningManager: spm,
AddressConverter: &testscommon.PubkeyConverterMock{},
SnapshotsManager: snapshotsManager,
StateChangesCollector: state.NewStateChangesCollector(),
}
adb, _ := state.NewAccountsDB(argsAccountsDB)

Expand Down Expand Up @@ -2757,6 +2758,7 @@ func createAccountsDBTestSetup() *state.AccountsDB {
StoragePruningManager: spm,
AddressConverter: &testscommon.PubkeyConverterMock{},
SnapshotsManager: snapshotsManager,
StateChangesCollector: state.NewStateChangesCollector(),
}
adb, _ := state.NewAccountsDB(argsAccountsDB)

Expand Down
1 change: 1 addition & 0 deletions integrationTests/testInitializer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -487,6 +487,7 @@ func CreateAccountsDBWithEnableEpochsHandler(
StoragePruningManager: spm,
AddressConverter: &testscommon.PubkeyConverterMock{},
SnapshotsManager: snapshotsManager,
StateChangesCollector: state.NewStateChangesCollector(),
}
adb, _ := state.NewAccountsDB(args)

Expand Down
2 changes: 2 additions & 0 deletions integrationTests/vm/staking/componentsHolderCreator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"github.com/multiversx/mx-chain-go/sharding"
"github.com/multiversx/mx-chain-go/sharding/nodesCoordinator"
"github.com/multiversx/mx-chain-go/state"
disabledState "github.com/multiversx/mx-chain-go/state/disabled"
stateFactory "github.com/multiversx/mx-chain-go/state/factory"
"github.com/multiversx/mx-chain-go/state/storagePruningManager"
"github.com/multiversx/mx-chain-go/state/storagePruningManager/evictionWaitingList"
Expand Down Expand Up @@ -215,6 +216,7 @@ func createAccountsDB(
StoragePruningManager: spm,
AddressConverter: coreComponents.AddressPubKeyConverter(),
SnapshotsManager: &stateTests.SnapshotsManagerStub{},
StateChangesCollector: disabledState.NewDisabledStateChangesCollector(),
}
adb, _ := state.NewAccountsDB(argsAccountsDb)
return adb
Expand Down
3 changes: 3 additions & 0 deletions process/transaction/metaProcess.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,8 @@ func (txProc *metaTxProcessor) ProcessTransaction(tx *transaction.Transaction) (
txProc.pubkeyConv,
)

defer txProc.accounts.SetTxHashForLatestStateChanges(txHash)

err = txProc.checkTxValues(tx, acntSnd, acntDst, false)
if err != nil {
if errors.Is(err, process.ErrUserNameDoesNotMatchInCrossShardTx) {
Expand All @@ -136,6 +138,7 @@ func (txProc *metaTxProcessor) ProcessTransaction(tx *transaction.Transaction) (
}

txType, _ := txProc.txTypeHandler.ComputeTransactionType(tx)

switch txType {
case process.SCDeployment:
return txProc.processSCDeployment(tx, tx.SndAddr)
Expand Down
3 changes: 3 additions & 0 deletions process/transaction/shardProcess.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,9 @@ func (txProc *txProcessor) ProcessTransaction(tx *transaction.Transaction) (vmco
txProc.pubkeyConv,
)

// TODO refactor to set the tx hash for the following state changes before the processing occurs
defer txProc.accounts.SetTxHashForLatestStateChanges(txHash)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the defer would call this method after return, and I see processing happens before, shouldn't the set happen before the processing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was to link all the changes triggered by a tx to the txHash after the tx was executed. It can be refactored as you say, but I would do that in the next PR to avoid generating more conflicts with @ssd04's PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added TODO


txType, dstShardTxType := txProc.txTypeHandler.ComputeTransactionType(tx)
err = txProc.checkTxValues(tx, acntSnd, acntDst, false)
if err != nil {
Expand Down
10 changes: 10 additions & 0 deletions process/transactionEvaluator/simulationAccountsDB.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,16 @@ func (r *simulationAccountsDB) GetStackDebugFirstEntry() []byte {
return nil
}

// SetTxHashForLatestStateChanges -
func (r *simulationAccountsDB) SetTxHashForLatestStateChanges(txHash []byte) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will not affect also the collected changes during processing right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it will not affect

r.originalAccounts.SetTxHashForLatestStateChanges(txHash)
}

// ResetStateChangesCollector -
func (r *simulationAccountsDB) ResetStateChangesCollector() []state.StateChangesForTx {
return nil
}

// Close will handle the closing of the underlying components
func (r *simulationAccountsDB) Close() error {
return nil
Expand Down
Loading
Loading