Ensure Folder Exists based on a Specific Property

[adamfisher]

I am trying to create an extension method on IFolder that behaves somewhat similar to EnsureFolderAsync() but using properties on folders instead so I can verify if a folder exists in the directory of a given folder regardless of what a user renames the folder to.

This is the code I have so far:

public static async Task<IFolder> EnsureFolderByPropertyNameAsync(this IFolder folder, string propertyValue)
{
	var propertyKey = "OriginTemplateFolderName";
	var matchingFolder = await folder.Folders.FirstOrDefaultAsync(f => (string)f.Properties[propertyKey] == propertyValue);

	if (matchingFolder != null)
	{
		return matchingFolder;
	}

	// Folder not found so create it
	var newFolder = await folder.AddFolderAsync(propertyValue);
	
	await newFolder.EnsurePropertiesAsync(f => f.Properties);
	
	if ((string) newFolder.Properties[propertyName] != propertyValue)
	{
		newFolder.Properties[propertyName] = propertyValue;
		await newFolder.Properties.UpdateAsync();
	}
	
	return newFolder;
}

This is the error that's produced when evaluating await folder.Folders.FirstOrDefaultAsync(f => (string)f.Properties[propertyKey] == propertyValue);:

crit: Microsoft.Extensions.Hosting.Internal.Host[10]
      The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
      System.NotSupportedException: Expression f.Properties.get_Item(value(BigCorp.MyProject.FolderExtensionMethods+<>c__DisplayClass1_0).propertyKey) is not supported. Only calls to System.Collections.Generic.Dictionary`2[System.String,System.Object] indexer are supported
         at PnP.Core.QueryModel.DataModelQueryTranslator`1.GetMemberName(Expression expression, Boolean raiseError)
         at PnP.Core.QueryModel.DataModelQueryTranslator`1.GetFilterField(Expression expression)
         at PnP.Core.QueryModel.DataModelQueryTranslator`1.AddBinaryFilter(BinaryExpression expression)
         at PnP.Core.QueryModel.DataModelQueryTranslator`1.VisitFirstOrDefault(MethodCallExpression m)
         at PnP.Core.QueryModel.DataModelQueryTranslator`1.VisitMethodCall(MethodCallExpression m)
         at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
         at PnP.Core.QueryModel.DataModelQueryProvider`1.ExecuteObjectAsync(Expression expression, CancellationToken token)
         at PnP.Core.QueryModel.BaseQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
         at PnP.Core.QueryModel.QueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken cancellationToken)
         at PnP.Core.QueryModel.QueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, LambdaExpression expression, CancellationToken cancellationToken)
         at PnP.Core.QueryModel.QueryableExtensions.FirstOrDefaultAsync[TSource](IQueryable`1 source, Expression`1 predicate, CancellationToken cancellationToken)

Would there be a better/more efficient way to achieve this without the error above? I think the error has something to do with some fundamental misunderstanding I have with the way domain model child properties get loaded on the folder.

[adamfisher]

I also tried a variation like this to just manually iterate through the properties since the Linq querying doesn't seem to support queries on properties but this produces a childFolder.Properties with zero values:

foreach (var childFolder in folder.Folders)
{
	await childFolder.EnsurePropertiesAsync(f => f.Properties);
	if (childFolder.Properties.Values.TryGetValue(propertyKey, out var value) && value.Equals(propertyKey))
	{
		return childFolder;
	}
}

[jansenbe]

@adamfisher : what about using a CAML query to recursively load all folders (assuming there are thousands) and then check for the property like shown in below snippet:

var myList = context.Web.Lists.GetByTitle("Site Assets", p => p.Title,
                                     p => p.Fields.QueryProperties(p => p.InternalName,
                                                                   p => p.FieldTypeKind,
                                                                   p => p.TypeAsString,
                                                                   p => p.Title));

int pageSize = 500;
// Filter on files and load 'system' properties like FileRef
string viewXml = @$"<View Scope='RecursiveAll'>
                        <ViewFields>
                          <FieldRef Name='FileRef' />
                          <FieldRef Name='FileLeafRef' />
                          <FieldRef Name='FSObjType'/>
                        </ViewFields>
                        <Query>
                            <Where>
                            <Eq>
                                <FieldRef Name='FSObjType'/>
                                <Value Type='Integer'>1</Value>
                            </Eq>
                            </Where>
                        </Query>
                        <OrderBy Override='TRUE'><FieldRef Name= 'ID' Ascending= 'FALSE' /></OrderBy>
                        <RowLimit>{pageSize}</RowLimit>
                        </View>";

// Load all the needed data using paged requests
bool paging = true;
string nextPage = null;
int pages = 0;
while (paging)
{
    // Execute the query
    await myList.LoadItemsByCamlQueryAsync(new CamlQueryOptions()
    {
        ViewXml = viewXml,
        DatesInUtc = true,
        PagingInfo = nextPage
    },
    // Load the Folder information
    p => p.Folder.QueryProperties(p=>p.Properties));

    pages++;

    if (myList.Items.Length == pages * pageSize)
    {
        nextPage = $"Paged=TRUE&p_ID={myList.Items.AsRequested().Last().Id}";
    }
    else
    {
        paging = false;
    }
}

// Iterate over ALL the retrieved list items
foreach (var folderListItem in myList.Items.AsRequested())
{
    //var fileSize = folder.FieldValuesAsText["File_x005f_x0020_x005f_Size"];
    if (folderListItem.Folder.Properties.Values.ContainsKey("vti_level"))
    {
        Debug.WriteLine($"Folder {folderListItem.Values["FileRef"]} contains property vti_level with value {folderListItem.Folder.Properties.Values["vti_level"]}");
    }               
}

Once you've a list of "missing" folders you can load the ParentFolder (e.g. via the EnsureFolder methods) and add the missing one + set the property bag.

[adamfisher]

That's not exactly what I was looking for but I may be able to repurpose that code and use it for when I have to iterate through and find specific files in an existing directory with a specific property value set and find/replace values inside of each of those files. So I thank you for that.

However my original method above was for making sure a child folder exists inside of the directory of a given IFolder instance passed into that extension method. I need to create a hierarchy structure initially and I am just walking down the directory path about three levels and at each level I need to make sure a folder got created with a specific name. Since users are free to change folder names, I am adding a property on each folder to ensure I am grabbing the correct folder regardless of what they rename it to. So I will end up using it like this:

var customerFolder = await EnsureFolderByPropertyNameAsync(rootFolder, customerName);
var locationFolder = await EnsureFolderByPropertyNameAsync(customerFolder, locationName);
var projectFolder = await EnsureFolderByPropertyNameAsync(locationFolder, projectName);

The issue I am facing with this method currently is that I can't read the Properties on the folder which is why it errors out on that line in the method.

[adamfisher]

For whatever reason the child folders on a folder.Folders call does not have the properties loaded. That collection always comes back zero items which is the source of my frustration. I'm sure it has to do with how the framework loads items but I would have expected one of these approaches to actually load them:

await childFolder.EnsurePropertiesAsync(f => f.Properties); // Doesn't work - zero properties on the folder
await childFolder.LoadAsync(f => f.Properties); // Doesn't work - zero properties on the folder
childFolder = await childFolder.GetAsync(f => f.Properties); // Doesn't work - zero properties on the folder

I have to manually call back out to the server to get it to load the properties. This worked:

public static async Task<IFolder> EnsureFolderByTemplateFolderPropertyAsync(this IFolder folder, string propertyValue)
{
	var propertyKey = "BigCorp_OriginTemplateFolderName";

	foreach (var childFolder in folder.Folders)
	{
		var currentFolder = await folder.PnPContext.Web.GetFolderByServerRelativeUrlAsync(childFolder.ServerRelativeUrl, f => f.Properties);
		if (currentFolder.Properties.Values.TryGetValue(propertyKey, out var value) && value.Equals(propertyValue))
		{
			return currentFolder;
		}
	}

	// Folder not found so create it
	var newFolder = await folder.AddFolderAsync(propertyValue);
	await newFolder.EnsureFolderHasPropertyAsync(propertyKey, propertyValue);
	return newFolder;
}

It feels like I shouldn't have to do that var currentFolder = await folder.PnPContext... line though. I'd be interested if there is a way to get one of the official load methods I mentioned above working. If not, I guess this is the next best approach ☹️

[jansenbe]

@adamfisher : that's indeed weird. We do a "get" to load folder properties in this test (https://github.com/pnp/pnpcore/blob/dev/src/sdk/PnP.Core.Test/SharePoint/FoldersTests.cs#L185-L194) and it works.

[adamfisher]

The proverbial "it works on my machine" 😄. Thanks for all your help Bert. There must be something I'm missing with loading domain model objects because that has been the main thing slowing me down using the SDK even though I've tried all three approaches as stated above and mentioned in the docs. I'll call folder.LoadAsync(p => p.Name); and depending where I'm calling it I get mixed results on whether that folder will have a name. I'm at a point where I'd really just like everything to load as I access a property even if I'm pulling back more than I need. But at least I got this functionality working now.