Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync main with production #954

Merged
merged 99 commits into from
Nov 15, 2024
Merged

Sync main with production #954

merged 99 commits into from
Nov 15, 2024

Conversation

marc-aurele-besner
Copy link
Collaborator

@marc-aurele-besner marc-aurele-besner commented Nov 15, 2024

User description

Sync main with production

Including


PR Type

Enhancement, Configuration changes, Bug fix


Description

  • Implemented a new taskboard setup using Express, Redis, and BullMQ for task management and processing.
  • Added scripts and utilities for database operations, including consensus updates and leaderboard ranking.
  • Enhanced Docker Compose configuration for dynamic port usage and improved service dependencies.
  • Introduced Slack integration for notifications and added a script to convert genesis allocation JSON to SQL seeds.
  • Refactored leaderboard mappings to use history tables and simplified account update logic.

Changes walkthrough 📝

Relevant files
Enhancement
13 files
index.ts
Implement Taskboard Setup with BullMQ and Express               

indexers/taskboard/src/index.ts

  • Added a new taskboard setup using Express and Redis.
  • Implemented authentication and session management.
  • Configured BullMQ for task processing.
  • Added routes for login and task management.
  • +216/-0 
    db.ts
    Add Database Connection and Query Utilities                           

    indexers/taskboard/src/utils/db.ts

  • Added database connection setup using pg.
  • Implemented queries for consensus and leaderboard updates.
  • Provided utility functions for database operations.
  • +158/-0 
    consensus.ts
    Implement Consensus Table Update Logic                                     

    indexers/taskboard/src/tasks/consensus.ts

  • Implemented consensus table updates.
  • Added logic for processing blocks in increments.
  • Utilized Redis for storing last processed block number.
  • +91/-0   
    updateAccount.ts
    Add Update Account Task with Polkadot API Integration       

    indexers/taskboard/src/tasks/updateAccount.ts

  • Added task for updating account balances.
  • Integrated with Polkadot API for account state retrieval.
  • Implemented database upsert for account data.
  • +70/-0   
    db.ts
    Refactor Leaderboard Mappings to Use History Tables           

    indexers/mainnet/leaderboard/src/mappings/db.ts

  • Refactored leaderboard mappings to use history tables.
  • Simplified logic for creating and updating records.
  • Removed redundant code for account updates.
  • +240/-384
    slack.ts
    Add Slack Messaging Utility                                                           

    indexers/taskboard/src/utils/slack.ts

  • Implemented utility for sending messages to Slack.
  • Supported message updates and error handling.
  • +64/-0   
    slackNotification.ts
    Implement Slack Notification Task                                               

    indexers/taskboard/src/tasks/slackNotification.ts

  • Added task for sending Slack notifications.
  • Configured message formatting and logging.
  • +80/-0   
    main.ts
    Script to Convert Genesis Allocation JSON to SQL Seeds     

    indexers/db/scripts/genisis-allocation-to-seeds/main.ts

  • Added script to convert genesis allocation JSON to SQL seeds.
  • Utilized Deno for file operations and UUID generation.
  • +54/-0   
    leaderboardSortAndRank.ts
    Implement Leaderboard Sorting and Ranking Task                     

    indexers/taskboard/src/tasks/leaderboardSortAndRank.ts

  • Implemented leaderboard sorting and ranking task.
  • Utilized database queries for updating rankings.
  • +63/-0   
    bull.ts
    Add BullMQ Utilities for Task Processing                                 

    indexers/taskboard/src/utils/bull.ts

  • Added utilities for BullMQ task processing.
  • Implemented job cleanup and processor setup.
  • +20/-14 
    store.ts
    Add Redis Connection and Utility Functions                             

    indexers/taskboard/src/utils/store.ts

  • Implemented Redis connection and utility functions.
  • Added methods for storing and retrieving values.
  • +59/-0   
    index.ts
    Define Task Handlers and Cron Jobs                                             

    indexers/taskboard/src/tasks/index.ts

  • Defined task handlers for various operations.
  • Configured cron jobs for periodic task execution.
  • +46/-0   
    cron.ts
    Add Utility for Setting Up Cron Tasks                                       

    indexers/taskboard/src/utils/cron.ts

  • Added utility for setting up cron tasks with BullMQ.
  • Configured task scheduling based on cron patterns.
  • +23/-0   
    Configuration changes
    2 files
    docker-compose.yml
    Enhance Docker Compose Configuration for Flexibility         

    docker-compose.yml

  • Updated service configurations for dynamic port usage.
  • Removed unnecessary platform and environment specifications.
  • Added health conditions for service dependencies.
  • +41/-37 
    index.ts
    Define Constants for Task Queues and Routes                           

    indexers/taskboard/src/constants/index.ts

  • Defined constants for task queues and routes.
  • Added configuration for Slack and network settings.
  • +32/-18 
    Additional files (token-limit)
    42 files
    graphql.ts
    ...                                                                                                           

    explorer/gql/graphql.ts

    ...

    +11300/-12148
    init-db.sql
    ...                                                                                                           

    indexers/db/docker-entrypoint-initdb.d/init-db.sql

    ...

    +1374/-1
    .env
    ...                                                                                                           

    .env

    ...

    +13/-7   
    package.json
    ...                                                                                                           

    indexers/taskboard/package.json

    ...

    +15/-3   
    package.json
    ...                                                                                                           

    indexers/package.json

    ...

    +2/-0     
    postgresql.conf
    ...                                                                                                           

    indexers/db/postgresql.conf

    ...

    +52/-0   
    index.ejs
    ...                                                                                                           

    indexers/taskboard/src/views/index.ejs

    ...

    +45/-0   
    actions.yaml
    ...                                                                                                           

    indexers/db/metadata/actions.yaml

    ...

    +29/-3   
    actions.graphql
    ...                                                                                                           

    indexers/db/metadata/actions.graphql

    ...

    +33/-0   
    docker-compose.prod.yml
    ...                                                                                                           

    docker-compose.prod.yml

    ...

    +6/-2     
    login.ejs
    ...                                                                                                           

    indexers/taskboard/src/views/login.ejs

    ...

    +1/-2     
    package.json
    ...                                                                                                           

    indexers/db/package.json

    ...

    +2/-1     
    tsconfig.json
    ...                                                                                                           

    indexers/taskboard/tsconfig.json

    ...

    +15/-0   
    Dockerfile
    ...                                                                                                           

    indexers/taskboard/Dockerfile

    ...

    +3/-3     
    deno.json
    ...                                                                                                           

    indexers/db/scripts/genisis-allocation-to-seeds/deno.json

    ...

    +9/-0     
    consensus_events.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/consensus_events.yaml

    ...

    +1/-1     
    README.md
    ...                                                                                                           

    indexers/db/scripts/genisis-allocation-to-seeds/README.md

    ...

    +11/-0   
    consensus_extrinsics.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/consensus_extrinsics.yaml

    ...

    +1/-1     
    README.md
    ...                                                                                                           

    indexers/db/scripts/README.md

    ...

    +6/-0     
    leaderboard_nominator_deposits_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_nominator_deposits_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_nominator_withdrawals_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_nominator_withdrawals_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_farmer_vote_and_block_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_farmer_vote_and_block_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_farmer_vote_and_block_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_farmer_vote_and_block_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_account_extrinsic_failed_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_extrinsic_failed_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_account_extrinsic_success_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_extrinsic_success_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_account_transaction_fee_paid_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_transaction_fee_paid_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_operator_deposits_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_operator_deposits_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_operator_deposits_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_operator_deposits_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_operator_withdrawals_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_operator_withdrawals_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_account_extrinsic_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_extrinsic_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_account_transfer_receiver_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_transfer_receiver_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_account_transfer_receiver_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_transfer_receiver_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_account_transfer_sender_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_transfer_sender_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_account_transfer_sender_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_transfer_sender_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_operator_total_rewards_collected_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_operator_total_rewards_collected_histories.yaml

    ...

    +3/-0     
    leaderboard_operator_total_tax_collected_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_operator_total_tax_collected_histories.yaml

    ...

    +3/-0     
    leaderboard_farmer_block_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_farmer_block_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_farmer_block_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_farmer_block_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_farmer_vote_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_farmer_vote_total_count_histories.yaml

    ...

    +3/-0     
    leaderboard_farmer_vote_total_value_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_farmer_vote_total_value_histories.yaml

    ...

    +3/-0     
    leaderboard_account_remark_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_account_remark_count_histories.yaml

    ...

    +3/-0     
    leaderboard_operator_bundle_total_count_histories.yaml
    ...                                                                                                           

    indexers/db/metadata/databases/default/tables/leaderboard_operator_bundle_total_count_histories.yaml

    ...

    +3/-0     

    💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

    marcusDesk and others added 30 commits November 7, 2024 19:40
    …te-account-balance
    
    Add Hasura action to update account balance
    …tion
    
    Fix events-extrinsics relation in hasura metadata
    …nsus-schema-at-postgres-init
    
    Seed dictionary and consensus schema at postgres init
    marcusDesk and others added 19 commits November 14, 2024 09:52
    …o-be-independant-from-indexer
    
    Improve consensus tasks to be independent from indexer
    …ers-and-tasks-similarly
    
    Improve leaderboard indexers and tasks similarly
    …h-leaderboard-historical-tables
    
    Improve Postgres init with leaderboard historical tables
    Copy link

    netlify bot commented Nov 15, 2024

    Deploy Preview for dev-astral ready!

    Name Link
    🔨 Latest commit 98ca591
    🔍 Latest deploy log https://app.netlify.com/sites/dev-astral/deploys/673748311497ca0008f95375
    😎 Deploy Preview https://deploy-preview-954--dev-astral.netlify.app
    📱 Preview on mobile
    Toggle QR Code...

    QR Code

    Use your smartphone camera to open QR code link.

    To edit notification comments on pull requests, go to your Netlify site configuration.

    Copy link

    netlify bot commented Nov 15, 2024

    Deploy Preview for astral-prod ready!

    Name Link
    🔨 Latest commit 98ca591
    🔍 Latest deploy log https://app.netlify.com/sites/astral-prod/deploys/67374831bd88aa0008656e5b
    😎 Deploy Preview https://deploy-preview-954--astral-prod.netlify.app
    📱 Preview on mobile
    Toggle QR Code...

    QR Code

    Use your smartphone camera to open QR code link.

    To edit notification comments on pull requests, go to your Netlify site configuration.

    fix port bindings and make local RPC archive node
    Copy link

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    🎫 Ticket compliance analysis 🔶

    937 - Fully compliant

    Fully compliant requirements:

    • Removed the platform tag from the node service in the docker-compose.yml file.

    938 - Fully compliant

    Fully compliant requirements:

    • Updated port mappings for Caddy service.
    • Changed ports to expose for subquery nodes.
    • Simplified the Caddyfile configuration.

    940 - Fully compliant

    Fully compliant requirements:

    • Changed expose to ports for indexer services.
    • Mapped external ports 3001, 3002, and 3003 to internal port 3000.

    939 - Fully compliant

    Fully compliant requirements:

    • Removed the --pot-external-entropy configuration line.

    934 - Fully compliant

    Fully compliant requirements:

    • Added slack task and slack Hasura action.
    • Converted taskboard to typescript.

    935 - Fully compliant

    Fully compliant requirements:

    • Added a script to convert genesis allocation JSON to SQL seed files.
    • Utilized Deno for file operations and HTTP requests.

    931 - Fully compliant

    Fully compliant requirements:

    • Fixed the relationship between events and extrinsics in Hasura metadata.
    • Updated column mappings.

    932 - Fully compliant

    Fully compliant requirements:

    • Enhanced the database initialization script to seed schemas and tables for consensus and dictionary.
    • Improved service dependency management in docker-compose.yml.
    • Simplified GraphQL schemas.
    • Removed migration scripts related to unique constraints.

    933 - Fully compliant

    Fully compliant requirements:

    • Added a new Hasura action updateAccount.
    • Implemented the updateAccount function.
    • Modified taskboard to support Hasura actions.
    • Updated Dockerfile to use Node.js version 18-alpine.
    • Added new dependencies for handling account updates.
    • Updated service port mappings and environment configurations in docker-compose.yml.

    941 - Fully compliant

    Fully compliant requirements:

    • Updated .env file to change the NODE_DOCKER_TAG version, modify RPC_URLS port, and update the DICTIONARY_URL port.
    • Enhanced docker-compose.yml by removing the --farmer option, using environment variables for DB_PORT, and updating the DICTIONARY URL to use a dynamic port.
    ⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Recommended focus areas for review

    Data Model Changes
    The PR introduces a significant change to the data model by replacing existing models with history models. This change impacts how data is stored and retrieved, which could affect performance and data integrity.

    Copy link

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Score
    Best practice
    Ensure database client is always released after use

    Ensure that the client.release() method is called in a finally block to guarantee
    that the database client is always released even if an error occurs during the
    transaction.

    indexers/taskboard/src/tasks/leaderboardSortAndRank.ts [55]

    -client.release();
    +try {
    +  ...
    +} finally {
    +  client.release();
    +}
    Suggestion importance[1-10]: 10

    Why: Ensuring the database client is released even during errors is critical for preventing resource leaks, which can lead to performance issues.

    10
    Validate database configuration to prevent runtime errors

    Validate environment variables for database configuration in connectToDB to ensure
    they meet expected formats or constraints before creating the PoolConfig.

    indexers/taskboard/src/utils/db.ts [4-10]

     const dbConfig: PoolConfig = {
       user: process.env.DB_USER || "postgres",
       host: process.env.DB_HOST || "localhost",
       database: process.env.DB_DATABASE || "postgres",
       password: process.env.DB_PASSWORD || "postgres",
       port: Number(process.env.DB_PORT) || 5432,
     };
    +if (!dbConfig.user || !dbConfig.host || !dbConfig.database || !dbConfig.password || isNaN(dbConfig.port)) {
    +  throw new Error("Invalid database configuration");
    +}
    Suggestion importance[1-10]: 7

    Why: Validating database configuration before usage helps in catching configuration errors early, preventing runtime issues and simplifying troubleshooting.

    7
    Possible issue
    Add robust error handling for network requests in sendSlackMessage

    Add error handling for the fetch call within sendSlackMessage to ensure that network
    or HTTP errors are caught and handled appropriately.

    indexers/taskboard/src/utils/slack.ts [44-53]

    -const response = await fetch(url, {
    -  method: "POST",
    -  headers: {
    -    "Content-Type": "application/json; charset=utf-8",
    -    Authorization: `Bearer ${token}`,
    -  },
    -  body: JSON.stringify(payload),
    -});
    +let response;
    +try {
    +  response = await fetch(url, {
    +    method: "POST",
    +    headers: {
    +      "Content-Type": "application/json; charset=utf-8",
    +      Authorization: `Bearer ${token}`,
    +    },
    +    body: JSON.stringify(payload),
    +  });
    +  if (!response.ok) {
    +    throw new Error(`HTTP error! status: ${response.status}`);
    +  }
    +} catch (e) {
    +  console.error("Error sending slack message", e);
    +  return undefined;
    +}
    Suggestion importance[1-10]: 9

    Why: Proper error handling for network requests is essential to manage failures gracefully, especially in communication functions like sending Slack messages.

    9
    Confirm that formatSpaceInDecimal meets the formatting needs after replacing formatSpacePledged

    Verify the replacement of formatSpacePledged with formatSpaceInDecimal to ensure it
    handles the formatting requirements correctly, as the new function might have
    different formatting behavior.

    explorer/src/app/[chain]/image/route.tsx [193]

    -{formatSpaceInDecimal(
    +{formatSpaceInDecimal( // Ensure this function formats as required
    Suggestion importance[1-10]: 3

    Why: The suggestion to verify the replacement function's behavior is valid but not directly actionable in code. It's more of a reminder to check functionality, which is useful but not a direct code improvement.

    3
    Verify that the URL generation for different networks is handled correctly

    Ensure that the EXTERNAL_ROUTES.polkadot function is properly handling different
    network IDs to generate the correct URL for each network.

    explorer/src/components/common/OutOfSyncBanner.tsx [35]

    +<Link className='ml-4' href={EXTERNAL_ROUTES.polkadot(network)} target='_blank'>
     
    -
    Suggestion importance[1-10]: 1

    Why: The suggestion is not actionable as it only asks to verify the URL generation without providing a specific issue or improvement in the code.

    1
    Enhancement
    Improve error handling for database connections

    Handle potential exceptions from connectToDB() and pool.connect() separately to
    provide more specific error messages and possibly different recovery strategies.

    indexers/taskboard/src/tasks/leaderboardSortAndRank.ts [16-24]

    -const pool: Pool = await connectToDB();
    -const client: PoolClient = await pool.connect();
    +let pool: Pool;
    +let client: PoolClient;
    +try {
    +  pool = await connectToDB();
    +  client = await pool.connect();
    +} catch (err) {
    +  console.error("Error connecting to the database:", err);
    +  throw new Error(`Database connection failed: ${err}`);
    +}
    Suggestion importance[1-10]: 8

    Why: Separating error handling for database connections provides clearer error messages and recovery strategies, improving the robustness of the database interaction.

    8
    Add error handling for Redis client to enhance server reliability

    Implement error handling for the session setup and Redis connection to prevent
    crashes and ensure the server starts correctly.

    indexers/taskboard/src/index.ts [59]

     const RedisClient = new Redis(connection);
    +RedisClient.on('error', (err) => console.error('Redis Client Error', err));
    Suggestion importance[1-10]: 7

    Why: Adding error handling for the Redis client connection is a crucial enhancement for server reliability and robustness, especially in production environments.

    7
    Implement error handling for the useChain hook to prevent potential runtime errors

    Add error handling for the useChain hook to manage cases where the chain data might
    be undefined or erroneous, which could lead to runtime errors if not checked.

    explorer/src/components/common/IndexingError.tsx [9]

    -const { network } = useChain()
    +const { network } = useChain() // Add error handling here
    Suggestion importance[1-10]: 4

    Why: Adding error handling for the useChain hook is a good practice to prevent runtime errors, especially if the hook might return undefined or erroneous data. This enhances the robustness of the component.

    4
    Possible bug
    Correct the handling of optional properties in the Incremental type to ensure proper functionality

    Ensure that the Incremental type definition correctly handles the optional
    properties for keys other than $fragmentName and __typename. The current
    implementation might inadvertently exclude other properties that should be optional.

    explorer/gql/graphql.ts [7]

    -export type Incremental<T> = T | { [P in keyof T]?: P extends ' $fragmentName' | '__typename' ? T[P] : never };
    +export type Incremental<T> = T | { [P in keyof T]?: P extends '$fragmentName' | '__typename' ? T[P] : never };
    Suggestion importance[1-10]: 5

    Why: The suggestion correctly identifies a potential issue with the handling of optional properties in the Incremental type, which could affect functionality. The improvement is moderate as it ensures better type safety and clarity.

    5
    Safeguard the BigInt operations to prevent overflows

    Ensure that the BigInt conversion and multiplication by 10^18 is safely handled to
    avoid potential overflows or precision issues.

    indexers/db/scripts/genisis-allocation-to-seeds/main.ts [30]

    -const freeBalance = (BigInt(entry[1]) * BigInt(1000000000000000000)).toString();
    +const freeBalance = (BigInt(entry[1]) * BigInt(10**18)).toString();
    Suggestion importance[1-10]: 2

    Why: The suggestion to use 10**18 instead of the literal value does not inherently prevent overflows or precision issues, it only changes the notation.

    2
    Maintainability
    Use a constant for the base value in the space formatting function

    Consider using a constant for the base value (1000) in the formatSpaceInDecimal
    function to ensure consistency and maintainability.

    explorer/src/utils/number.ts [81]

    -const k = 1000
    +const BASE = 1000
    Suggestion importance[1-10]: 5

    Why: Using a named constant improves readability and maintainability of the code. The suggestion correctly identifies the line and provides a meaningful improvement.

    5

    @marc-aurele-besner marc-aurele-besner merged commit 1837d46 into production Nov 15, 2024
    25 checks passed
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    4 participants