is there a change to what the create_project method returns -- I'm getting None back for the project dictionary.

[hschellman]

I'm running my test code again after a month or so and I find that this code:

try:
print ("try to make a project")
proj_dict = dd_client.create_project(files=query_files, query=thequery, idle_timeout=259201)
print ("project creation succeeded",query_files,thequery)
except:

print ("project creation failed")

if debug: print("project dictionary",proj_dict)

succeeds but prints out:

create_project with 10 files
try to make a project
project creation succeeded 10 files from schellma:rawPhysics_5141_5143 ordered skip 10 limit 10
project dictionary None

so valid list of files and query, no exception but no response.

THanks, Heidi

[ivmfnal]

I think I fixed it.
Please try and let me know

[hschellman]

Works now… On Sep 22, 2023, at 5:21 AM, Igor Mandrichenko ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] I think I fixed it. Please try and let me know — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA37DIBIEK7NL4BACXHMSTX3V7C3ANCNFSM6AAAAAA5CIBDG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[hschellman]

I did get a strange error at the end of a project which I think was trying to read files frmo tape but that did fix the original issue. More checks for me to do. On Sep 22, 2023, at 7:09 AM, Igor Mandrichenko ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] Closed #22<#22> as completed. — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA37DIT3NI7CONWZJAL4W3X3WL25ANCNFSM6AAAAAA5CIBDG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ivmfnal]

What was the error ?

[hschellman]

Interesting behavior. 1st time I ran it, no replicas found. Ran it again and now finding replicas at FNAL. My job was only actually using 1 replica. You can see the behavior change from #25 to #29 at https://metacat.fnal.gov:9443/dune/dd/gui/P/projects I am now running with a request for 3 files that actually uses 3 files. This looks as if there was some response failure from rucio or there may actually only be 1 copy of those files and dcache is staging on request. On Sep 22, 2023, at 8:08 AM, Igor Mandrichenko ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] Reopened #22<#22>. — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA37DMVYVKP6YSNDB4LZILX3WSWRANCNFSM6AAAAAA5CIBDG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ivmfnal]

Replica availability information is not updated in real time. So initially replicas are always not found until DD polls Rucio next time

[hschellman]

Any way to get it to behave better, like wait? having the job check, fail to find anything and then fail to do anything seems non-optimal. On Sep 22, 2023, at 8:26 AM, Igor Mandrichenko ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] Replica availability information is not updated in real time. So initially replicas are always not found until DD polls Rucio next time — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA37DPW55HRYS757743LX3X3WU3BANCNFSM6AAAAAA5CIBDG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ivmfnal]

I am not sure what you mean.
The worker is supposed to issue "next file", which will block until a replica to become available or the project ends.
Are you doing something else ?

[hschellman]

May be missing that step. this is based on Jake’s work almost a year ago. right now it is asking for list of replicas and due to the design of larsoft was passing the list of locations to the executable. On Sep 22, 2023, at 8:53 AM, Igor Mandrichenko ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] I am not sure what you mean. The worker is supposed to issue "next file" and wait for a replica to become available. Are you doing something else ? — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA37DJCJSAJITEHYNSBLL3X3WYAFANCNFSM6AAAAAA5CIBDG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ivmfnal]

List of replicas ?
List of replicas for the next file or list of all replicas for all files in the project ?