Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scylla_node: add ScyllaNode.dump_sstables() #484

Merged
merged 1 commit into from
Jul 31, 2023

Conversation

tchaikov
Copy link
Contributor

ScyllaNode.dump_sstables() is a wrapper around
ScyllaNode.run_scylla_sstable(). it provides a more user friendly interface than the latter. it is introduced so that tests can use it with less pain when migrating from
node.run_sstable2json() to node.run_scylla_sstable().

@tchaikov
Copy link
Contributor Author

with the proposed change, we can have something like

diff --git a/compaction_test.py b/compaction_test.py
index 4b4059dd..53100bca 100644
--- a/compaction_test.py
+++ b/compaction_test.py
@@ -180,19 +180,10 @@ class TestCompaction(Tester):
         node.flush()
         node.compact()
 
-        sstable_dumps = node.run_scylla_sstable('dump-data', ['--merge'],
-                                                keyspace=self.KEYSPACE_NAME)
-        numfound = 0
-        jsoninfo = []
-        for stdout, _ in sstable_dumps.values():
-            partitions = json.loads(stdout)['sstables']['anonymous']
-            jsoninfo += partitions
-            for partition in partitions:
-                if 'tombstone' in partition:
-                    numfound += 1
-
+        partitions = node.dump_sstables(keyspace=self.KEYSPACE_NAME)
+        numfound = sum('tombstone' in partition for partition in partitions)
         logger.debug(f'Number of tombstones found on node {node.name}: {numfound}')
-        return numfound, jsoninfo
+        return numfound, partitions
 
     def _test_compaction_delete_tombstone_gc(self, tombstone_gc_mode='repair', node_num: int = 2,
                                              r_factor: int = None, delete_keys: bool = True, partition_num: int = 100):

after https://github.com/scylladb/scylla-dtest/pull/3351 lands

all_partitions: List[Dict[str, Any]] = []
for stdout, _ in sstable_dumps.values():
partitions = json.loads(stdout)['sstables']['anonymous']
all_partitions += partitions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this will possibly mash together partitions from different tables. Which might be confusing. I think we should enforce that the user always passes a keyspace and exactly one table parameter. I think this is how most users will use this anyway. The rest (if any) can call run_scylla_sstable() instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed accordingly.

@tchaikov tchaikov force-pushed the scylla-dump-sstables branch 2 times, most recently from e183ab2 to c9187e8 Compare July 31, 2023 06:33
@tchaikov tchaikov requested a review from denesb July 31, 2023 06:33
Copy link
Contributor

@denesb denesb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tagline of comment of the method is outdated, LGTM otherwise.

ccmlib/scylla_node.py Outdated Show resolved Hide resolved
@tchaikov
Copy link
Contributor Author

v2

  • ks and cf are not optional anymore.
  • do not accept sstables, as no users are using this parameter so far.

@tchaikov
Copy link
Contributor Author

v3:

  • revised the comment to reflect the latest API interface.

`ScyllaNode.dump_sstables()` is a wrapper around
`ScyllaNode.run_scylla_sstable()`. it provides a more user friendly
interface than the latter. it is introduced so that tests can
use it with less pain when migrating from
`node.run_sstable2json()` to `node.run_scylla_sstable()`.

Signed-off-by: Kefu Chai <[email protected]>
Copy link
Contributor

@fruch fruch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fruch fruch merged commit e27a8f4 into scylladb:next Jul 31, 2023
3 checks passed
@tchaikov tchaikov deleted the scylla-dump-sstables branch August 1, 2023 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants