Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#155] New table: journal_persistence_ids #206

Merged
merged 34 commits into from
Aug 29, 2023

Conversation

tiagomota
Copy link
Contributor

@tiagomota tiagomota commented Oct 8, 2021

This PR introduces a new metadata table to hold journal specific information, as in issue #155.

Changelog:

  • New table journal_metadata and trigger on journal inserts that auto-populate it.
  • journal_metadata new table name and column names are configurable like the other tables.
  • Migrations scripts that take care of creating the new table and the associated triggers/functions.
  • Updates to the schemas being used on the tests, both on the core and migration artifacts.
  • Use this metadata table where the following queries were being called:
    • JournalQueries#highestSequenceNrForPersistenceId
    • ReadJounalQueries#messageQuery
  • Usage of journal metadata table is controlled through config

(These previous queries are still called in case the metadata table is empty for a specific persistence_id, acting as a fallback removing the need to actually populate the metadata beforehand in some costly migration.)

@tiagomota tiagomota self-assigned this Oct 8, 2021
@tiagomota tiagomota marked this pull request as ready for review October 13, 2021 08:02
@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch from ec8dabf to f91eab6 Compare October 13, 2021 15:19
@tiagomota
Copy link
Contributor Author

Build is failing due to the following:

  1. tests defined on PartitionedJournalSpecTestCases trait (PostgresJournalSpec.scala)
  2. tests defined on JournalSequenceActorTest abstract class (JournalSequenceActorTest.scala)

What these tests have in common is concurrent inserts on the Journal with generated sequence numbers. Then, since the inserts are not sequential, an event with smaller sequence_number will come after one with a bigger sequence_number, activating the new check introduced in this PR, that prevents inserts on the Journal if the sequence_number is smaller than the maximum that is already stored.

Not entirely sure how to solve these tests, keeping what they were initially trying to test. For instance, the failing tests defined by PartitionedJournalSpecTestCase are easily solved if we do this:

// replace this
Future
  .sequence {
    senders.map { case (sender, idx) =>
      Future {
        writeMessages((idx * batchSize) + 1, (idx + 1) * batchSize, perId, sender.ref, writerUuid)
      }
    }
  }.futureValue(Timeout(Span(1, Minute)))

// with this
Future {
  senders.map { case (sender, idx) =>
    writeMessages((idx * batchSize) + 1, (idx + 1) * batchSize, perId, sender.ref, writerUuid)
  }
}.futureValue(Timeout(Span(1, Minute)))

However, it kind of defeats the purpose of the test.

@lomigmegard
Copy link
Member

Build is failing due to the following:

1. tests defined on `PartitionedJournalSpecTestCases` trait ([PostgresJournalSpec.scala](https://github.com/SwissBorg/akka-persistence-postgres/blob/master/core/src/test/scala/akka/persistence/postgres/journal/PostgresJournalSpec.scala))

2. tests defined on `JournalSequenceActorTest` abstract class ([JournalSequenceActorTest.scala](https://github.com/SwissBorg/akka-persistence-postgres/blob/master/core/src/test/scala/akka/persistence/postgres/query/JournalSequenceActorTest.scala))

The name of the test is store events concurrently without any gaps or duplicates among ordering (offset) values. From what I understand, this suite is testing only the offset/ordering value, that is its unicity.

Without the added trigger check, the constraint was weaker : it used to verify that the pair (persistence_id, sequence_nr) was unique, but the new check also verify the order of insertion.

The test should be updated to account for that: concurrent writes across persistence_id but sequential for each one. In that case there is only one (val perId = "perId-1"), adding a second one would be interesting to not have a trivial case.

@mkubala What do you think?

@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch from f91eab6 to 9534c5c Compare October 14, 2021 07:26
@tiagomota
Copy link
Contributor Author

@lomigmegard and @mkubala 👋

Commit 9adf419 solves the tests issues and makes the build pass. Not sure if the tests objective still remains though 👐

@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch 2 times, most recently from 367650e to 4f9fc9d Compare October 25, 2021 16:00
@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch from 4f9fc9d to ed38da6 Compare November 2, 2021 14:51
@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch from e4eca16 to a32393b Compare November 10, 2021 10:37
@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch from 0e4d39b to 4943904 Compare November 11, 2021 14:23
@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch from 4943904 to ea0046f Compare July 4, 2023 17:35
@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch 2 times, most recently from fc6d492 to 2356f24 Compare July 6, 2023 15:45
With the addition of triggers that can refuse an insert on the journal, concurrent inserts on the journal for the same persistence_id can fail due to sequence number being smaller then what is already recorded. Therefore, not having gaps among ordering can only be verified across all persistence_ids and not a single one.
@tiagomota tiagomota force-pushed the 155-journal-persistence-ids-table branch from 2356f24 to 5c8970f Compare July 20, 2023 13:41
@tiagomota tiagomota merged commit 2bfafae into master Aug 29, 2023
3 checks passed
@tiagomota tiagomota deleted the 155-journal-persistence-ids-table branch August 29, 2023 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants