SiteCollectionManager.GetSiteCollectionsWithDetailsAsync Pagination?

[Sean-LM]

Hi! Our M365 rollout journey is in full swing, we are a very large enterprise with tens of thousands of on-prem site collections, gradually expanding into SP Online (and teams, which as you know create an SP Online site for each team!)

We have a custom application for users to manage their assets (teams, sites, etc) which we keep in sync via the graph API and utilizing the pnp core library. Our tenant site collection count is approaching 7,000 site collections, and our scanner is running out of memory as it appears we cannot paginate the SiteCollectionManager.GetSiteCollectionsWithDetailsAsync call. It just loads EVERY site collection... The rough architecture is to just scan a few hundred sites per hour, not trying to process the entire environment in one go.

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at PnP.Core.Admin.Model.SharePoint.SiteCollectionEnumerator.ProcessLoadedSitesFromTenantAdminHiddenList(IEnumerable`1 listItems, List`1 loadedSites)
   at PnP.Core.Admin.Model.SharePoint.SiteCollectionEnumerator.<>c__DisplayClass3_0.b__0(IEnumerable`1 listItems)
   at PnP.Core.Admin.Model.SharePoint.SiteCollectionEnumerator.LoadSitesViaTenantAdminHiddenListAsync(PnPContext context, String viewXml, Action`1 processResults, VanityUrlOptions vanityUrlOptions, Boolean allSites, Int32 pageSize)
   at PnP.Core.Admin.Model.SharePoint.SiteCollectionEnumerator.GetWithDetailsViaTenantAdminHiddenListAsync(PnPContext context, VanityUrlOptions vanityUrlOptions, Int32 pageSize)
   at PnP.Core.Admin.Model.SharePoint.SiteCollectionManager.GetSiteCollectionsWithDetailsAsync(VanityUrlOptions vanityUrlOptions)

I can keep cranking up the memory on my container, but this is not a sustainable path - before I start digging into the internals of the library to figure out how to do this, has anyone else encountered this, or does the library actually allow pagination on this call and I cannot figure it out?

Thank you!

[jansenbe]

@Sean-LM : this method indeed will load all site collections in memory and return them. The amount of data for 7000 site collections however should not be huge unless you keep additional data in memory for all 7000. I assume you've kept the includeSharedAndPrivateTeamChannelSites parameter set to false?

What memory do you allocate for your container?

[Sean-LM]

One of the first things the container does upon spin-up (it runs a console app) is begin gathering those sites. The out of memory exception was being thrown from within GetSiteCollectionsWithDetailsAsync, not during any post processing of the data (and thus loading additional things into memory.)

I went all the way to 4GB memory on the container but was still running out. I have since solved this issue (sorry for not coming back and updating) by implementing our own version of SiteCollectionEnumerator.GetWithDetailsViaTenantAdminHiddenListAsync which our code can call and accepts two new arguments: pageSize and nextHref.

The page size is parameterized in the CAML query: <RowLimit Paged='TRUE'>%PageSize%</RowLimit> and instead of continuing to get page after page after page, the code retrieves one page, and then returns a Tuple<List<ISiteCollectionWithDetails>, string> with the 1 page of sites, and the newNextHref, which can be stored somewhere and then used to retrieve the next page at a later time. If no nextHref is passed in, it retrieves the first page.

This has been working well, and we are able to scan 250 sites per run, and run on a schedule to keep our system in sync with the tenant.

I am not familiar enough with the internals of PnP to open a Pull Request with a more robust feature back to the library, but if any reader wishes to see what I've done, I can share the code. It was a fairly simple rough and ready solution to keep my app healthy in production.