Problems with large results sets

[sypets]

If constraint has large result set, fetching pagetree takes much longer than without constraints and may result in errors.

I am assuming the extension may be fetching too many results. In the core - to handle the previous performance problems in the page tree - not the entire pagetree is fetched - only the expanded part. (After several changes in the pagetree mechanism).

If I fetch with this extension "pagetreefilter" using a constraint with a large result set I get one of:

• Page tree error (error message)
• A spinner making the pagetree and the text field unresponsive so you can only abort with reload (actually a problem in the TYPO3 core, I think)

I still find the extension very nice and quite useful. The editors on our site will most likely not get this problem as they see only a much smaller subset of pages (due to the mount points used).

I am not using this yet in production, it might be nice to be able to find a way to solve these problems. Will upgrade to v11 soon.

System

I am using

• TYPO3 10.4.26 (latest)
• pagetreefilter 0.2.1 (latest for v10)

site has about 40 000 pages.

select count(*) from pages where sys_language_uid=0 and not deleted;
+----------+
| count(*) |
+----------+
|    40932 |
+----------+

Reproduce

1. use constraints such as

• table=pages
• table=pages doktype=1

Watching the result times in the Network tab of developer tools, I also see long response times for the filterData requests, e.g. - 29s

error messages

Browser console:

PageTree.js?bust=1648368554:486 Uncaught SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at Object.<anonymous> (d3.js?bust=1648368554:8:28915)
    at XMLHttpRequest.c (d3.js?bust=1648368554:6:16638)

No message in TYPO3 log (log file with ERROR log level).

This may be related to these core issues:

• https://forge.typo3.org/issues/92033
• https://forge.typo3.org/issues/92036

Possible solution

It might be a possible solution to get number of results first and if too large reject the constraint and fall back to original behaviour of TYPO3 or abort the current filter with an error message ("result set too large"). (Have not really thought this through though, just tossing out ideas. Difficulty is not just detemining what kind of result set is "too large").

[christophlehmann]

I also noticed longer delays when applying filter constraints in bigger page trees.

Filtering for table=pages and then (typing) table=pages doktype=1 may result unwanted requests due to slow typing. That may be prevented in this ext, somehow. On the other hand the core may help in this situation with a (currently non-existing-existing?!) setting like "User need to press enter to search in page tree" (One has written about it in one of the issues).
The modal wizard in this ext may have a further step (or just an input field), where the user can write its complete filter query and then submit it to the pagetrees search form. That may help, even it it feels little like a workaround for a core issue, hm?

The idea about limiting results also came in my mind. Imagine you could use the currently selected page as context with the result that you only search in its subpages. From UIs point of view i think about a checkbox in the modal wizard "Search in subpages only". That may be helpful too. Question here is, does "building the uid list of subpages" maybe take longer then the actual search? :-) It really depends on the filter constraints and if constraint fields have an index. I'm open for for such context option...

Limiting the result with "give me only 20 pages" seems fiddly to implement because of editors and their mount points.

In a system with 10k pages i have seen response times of ~ 15s. IMHO this is okay. Also ~45s may be okay, but if its going to take a minutes, then something need to be changed.

Question about your 29s request resulting in an error. Is it due to max_execution_time=30s ?

[sypets]

Question about your 29s request resulting in an error. Is it due to max_execution_time=30s ?

sorry, did not see the question before.

Answer: max_execution_time is set to recommended value of 240, so no. That might have been a coincidence.

About the core feature:

On the other hand the core may help in this situation with a (currently non-existing-existing?!) setting like "User need to press enter to search in page tree"

Is there a core issue for this? (you mentioned it was mentioned in one of the issues, but it might be helpful to actually address this directly) (It always has some drawbacks to make things configurable but here, I think it would be helpful. Alternatively create an extension, which implements this or add it in this extension?).

[christophlehmann]

Is there a core issue for this?

I meaned this comment: https://forge.typo3.org/issues/92036#note-11

Would you like to beta test and check if https://github.com/christophlehmann/pagetreefilter/tree/filter-input-in-wizard fit your needs?

[sypets]

@christophlehmann

Would you like to beta test and check if https://github.com/christophlehmann/pagetreefilter/tree/filter-input-in-wizard fit your needs?

Once I apply my fix in #19, this works nicely. It also seems to handle some result sets better. However, if I have a very large result set (e.g. table=pages, doktype=1), this may still fail, resulting in result set of previous request displayed.

However, I tested with table=pages doktype=254 too which previously caused problems and now works fine with this branch.

(Also, the bevaviour in v11 is a little different than before / in TYPO3 v10, I no longer see an error message, if a request fails).