fix(systemtags): Add missing systemtags index

[marcelklehr]

• Resolves: Systemtag DB slowness

Summary

MariaDB [nextcloud]> explain select * from oc_systemtag_object_mapping where systemtagid = 808;
+------+-------------+-----------------------------+-------+---------------+---------+---------+------+-------+--------------------------+
| id   | select_type | table                       | type  | possible_keys | key     | key_len | ref  | rows  | Extra                    |
+------+-------------+-----------------------------+-------+---------------+---------+---------+------+-------+--------------------------+
|    1 | SIMPLE      | oc_systemtag_object_mapping | index | NULL          | PRIMARY | 524     | NULL | 38418 | Using where; Using index |
+------+-------------+-----------------------------+-------+---------------+---------+---------+------+-------+--------------------------+
1 row in set (0.006 sec)

MariaDB [nextcloud]> CREATE INDEX marcels_systag_idx on oc_systemtag_object_mapping (systemtagid, objecttype);
Query OK, 0 rows affected (0.087 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [nextcloud]> explain select * from oc_systemtag_object_mapping where systemtagid = 808;
+------+-------------+-----------------------------+------+---------------------+---------------------+---------+-------+------+-------------+
| id   | select_type | table                       | type | possible_keys       | key                 | key_len | ref   | rows | Extra       |
+------+-------------+-----------------------------+------+---------------------+---------------------+---------+-------+------+-------------+
|    1 | SIMPLE      | oc_systemtag_object_mapping | ref  | marcels_systag_idx | marcels_systag_idx | 8       | const | 626  | Using index |
+------+-------------+-----------------------------+------+---------------------+---------------------+---------+-------+------+-------------+

Before:

• rows: 38418

After:

• rows: 626

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[marcelklehr]

Can we backport this?

[nickvergessen]

Can we backport this?

I guess we should

[blizzz]

Looks promising also on postgres:

master=# explain analyze select * from oc_systemtag_object_mapping where systemtagid = 27;
                                                        QUERY PLAN                                                         
---------------------------------------------------------------------------------------------------------------------------
 Seq Scan on oc_systemtag_object_mapping  (cost=0.00..42.60 rows=169 width=21) (actual time=0.025..0.771 rows=145 loops=1)
   Filter: (systemtagid = 27)
   Rows Removed by Filter: 2063
 Planning Time: 0.265 ms
 Execution Time: 0.816 ms
(5 rows)

master=# CREATE INDEX marcels_systag_idx on oc_systemtag_object_mapping (systemtagid, objecttype);
CREATE INDEX

                                                            QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on oc_systemtag_object_mapping  (cost=5.59..22.70 rows=169 width=21) (actual time=0.174..0.303 rows=145 loops=1)
   Recheck Cond: (systemtagid = 27)
   Heap Blocks: exact=12
   ->  Bitmap Index Scan on marcels_systag_idx  (cost=0.00..5.55 rows=169 width=0) (actual time=0.143..0.144 rows=145 loops=1)
         Index Cond: (systemtagid = 27)
 Planning Time: 0.184 ms
 Execution Time: 0.380 ms
(7 rows)

[nickvergessen]

The autoloaders are not up to date
Please run: bash build/autoloaderchecker.sh
And commit the result

[marcelklehr]

Do we actually need this migration or is the missing indices stuff enough? Or should I put this into the original v13... migration that added the systemtags table?

[nickvergessen]

yeah, actually it should be in the original one, otherwise it still performs the index creation on the update and that would nullify the rest of this PR.

[marcelklehr]

/backport to stable27

[marcelklehr]

/backport to stable26

[marcelklehr]

/backport to stable25

[marcelklehr]

Backports didn't happen?

[backportbot-nextcloud]

The backport to stable27 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable27
git pull origin stable27

# Create the new backport branch
git checkout -b fix/foo-stable27

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts. Resolve them.
git cherry-pick abc123

# Push the cherry pick commit to the remote repository and open a pull request
git push origin fix/foo-stable27

More info at https://docs.nextcloud.com/server/latest/developer_manual/getting_started/development_process.html#manual-backport

[backportbot-nextcloud]

The backport to stable26 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable26
git pull origin stable26

# Create the new backport branch
git checkout -b fix/foo-stable26

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts. Resolve them.
git cherry-pick abc123

# Push the cherry pick commit to the remote repository and open a pull request
git push origin fix/foo-stable26

More info at https://docs.nextcloud.com/server/latest/developer_manual/getting_started/development_process.html#manual-backport

[backportbot-nextcloud]

The backport to stable25 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable25
git pull origin stable25

# Create the new backport branch
git checkout -b fix/foo-stable25

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts. Resolve them.
git cherry-pick abc123

# Push the cherry pick commit to the remote repository and open a pull request
git push origin fix/foo-stable25

More info at https://docs.nextcloud.com/server/latest/developer_manual/getting_started/development_process.html#manual-backport

[solracsf]

/backport to stable27

[solracsf]

/backport to stable26

[solracsf]

/backport to stable25

[backportbot-nextcloud]

The backport to stable27 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable27
git pull origin stable27

# Create the new backport branch
git checkout -b fix/foo-stable27

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts. Resolve them.
git cherry-pick abc123

# Push the cherry pick commit to the remote repository and open a pull request
git push origin fix/foo-stable27

More info at https://docs.nextcloud.com/server/latest/developer_manual/getting_started/development_process.html#manual-backport

[backportbot-nextcloud]

The backport to stable26 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable26
git pull origin stable26

# Create the new backport branch
git checkout -b fix/foo-stable26

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts. Resolve them.
git cherry-pick abc123

# Push the cherry pick commit to the remote repository and open a pull request
git push origin fix/foo-stable26

More info at https://docs.nextcloud.com/server/latest/developer_manual/getting_started/development_process.html#manual-backport

[backportbot-nextcloud]

The backport to stable25 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable25
git pull origin stable25

# Create the new backport branch
git checkout -b fix/foo-stable25

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts. Resolve them.
git cherry-pick abc123

# Push the cherry pick commit to the remote repository and open a pull request
git push origin fix/foo-stable25

More info at https://docs.nextcloud.com/server/latest/developer_manual/getting_started/development_process.html#manual-backport

[ernolf]

I don't know if this is the right place but I just updated 3 instances, two with MySQL, one with PostgreSQL to 27.0.1 RC1

The /settings/admin/overview throws this message:

but on all of the three instances

occ db:add-missing-indices

does not make any changes at all nor do i get the sql query with

occ db:add-missing-indices --dry-run

and the warning does not disapear.

MariaDB [nextcloud]> explain select * from oc_systemtag_object_mapping where systemtagid = 568;
+------+-------------+-----------------------------+-------+---------------+---------+---------+------+--------+--------------------------+
| id   | select_type | table                       | type  | possible_keys | key     | key_len | ref  | rows   | Extra                    |
+------+-------------+-----------------------------+-------+---------------+---------+---------+------+--------+--------------------------+
|    1 | SIMPLE      | oc_systemtag_object_mapping | index | NULL          | PRIMARY | 524     | NULL | 143547 | Using where; Using index |
+------+-------------+-----------------------------+-------+---------------+---------+---------+------+--------+--------------------------+
1 row in set (0.000 sec)

MariaDB [nextcloud]> DESCRIBE oc_systemtag_object_mapping;
+-------------+---------------------+------+-----+---------+-------+
| Field       | Type                | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+-------+
| objectid    | varchar(64)         | NO   | PRI |         |       |
| objecttype  | varchar(64)         | NO   | PRI |         |       |
| systemtagid | bigint(20) unsigned | NO   | PRI | 0       |       |
+-------------+---------------------+------+-----+---------+-------+
3 rows in set (0.001 sec)

MariaDB [nextcloud]> SHOW INDEXES FROM oc_systemtag_object_mapping;
+-----------------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table                       | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| oc_systemtag_object_mapping |          0 | PRIMARY  |            1 | objecttype  | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| oc_systemtag_object_mapping |          0 | PRIMARY  |            2 | objectid    | A         |      143547 |     NULL | NULL   |      | BTREE      |         |               |
| oc_systemtag_object_mapping |          0 | PRIMARY  |            3 | systemtagid | A         |      143547 |     NULL | NULL   |      | BTREE      |         |               |
+-----------------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
3 rows in set (0.001 sec)

[blizzz]

@ernolf best is to open a new issue. Turned your comment into one now. Thanks for reporting!