Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dig into release failures from DevOps timeouts/uploading duplicated artifacts #2180

Closed
danieljurek opened this issue Apr 11, 2022 · 3 comments
Closed

Dig into release failures from DevOps timeouts/uploading duplicated artifacts #2180

danieljurek opened this issue Apr 11, 2022 · 3 comments
Assignees
@danieljurek
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Central-EngSys This issue is owned by the Engineering System team. EngSys This issue is impacting the engineering system.

Comments

@danieljurek
Copy link
Member

danieljurek commented Apr 11, 2022 •
edited
Loading

How do we ensure that the pipeline succeeds when retrying just the job.

See: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1484310&view=results
@danieljurek danieljurek added the Central-EngSys This issue is owned by the Engineering System team. label Apr 11, 2022
@ghost ghost added the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Apr 11, 2022
@danieljurek danieljurek transferred this issue from Azure/azure-sdk-for-cpp Apr 11, 2022
@ahsonkhan
Copy link
Member

Example pipeline that failed after timeout with this error, during Publish bom_manifest_$(BomArtifactName) Artifacts step:

##[error]Artifact bom_manifest_LinuxARM_Release_Logging already exists for build 1484310.

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1484310&view=results

[](https://vsblobprodcus3.vsblob.visualstudio.com/A0fb41ef4-5012-48a9-bf39-4ee3de03ee35/%22,%22X_TFS_Session%22:%22ee63dc5c-11ab-43a1-a39b-076feb3ade7c%22,%22DeploymentEnvironment%22:%22PRODUCTION%22,%22DeploymentEnvironmentIsProduction%22:%22True%22,%22VSOAccount%22:%22vsblobprodcus3%22,%22OSName%22:%22Linux%22,%22OSVersion%22:%222022%22,%22FrameworkDescription%22:%22.NET)Verbose, ApplicationInsightsTelemetrySender waiting for 0 of 2 TrackEvent operations to complete...
Verbose, ApplicationInsightsTelemetrySender sent DedupUpload.UploadAsync telemetry
Verbose, ApplicationInsightsTelemetrySender operations completed in 338 ms
Information, ApplicationInsightsTelemetrySender correlated 2 events with X-TFS-Session ee63dc5c-11ab-43a1-a39b-076feb3ade7c
Verbose, Stopped ITelemetrySender BlobStoreApplicationInsightsTelemetrySender.
##[error]Artifact bom_manifest_LinuxARM_Release_Logging already exists for build 1484310.
##[debug]Processed: ##vso[task.logissue type=error;]Artifact bom_manifest_LinuxARM_Release_Logging already exists for build 1484310.
##[debug]Processed: ##vso[task.complete result=Failed;]
##[debug]   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.TeamFoundation.Build.WebApi.BuildHttpClientCompatBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken, Func`3 processResponse)
   at Microsoft.TeamFoundation.Build.WebApi.BuildHttpClientCompatBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken, Func`3 processResponse)
   at Agent.Plugins.BuildServer.AssociateArtifactAsync(Guid projectId, Int32 pipelineId, String name, String jobId, String type, String data, Dictionary`2 propertiesDictionary, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineArtifactServer.<>c__DisplayClass2_1.<<UploadAsync>b__5>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Content.Common.ExceptionExtensions.ReThrow(Exception ex)
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
   at Agent.Plugins.PipelineArtifactServer.UploadAsync(AgentTaskPluginExecutionContext context, Guid projectId, Int32 pipelineId, String name, String source, IDictionary`2 properties, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineArtifact.PublishPipelineArtifactTaskV1.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)
Finishing: Publish bom_manifest_$(BomArtifactName) Artifacts

@RickWinter RickWinter added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Apr 12, 2022
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Apr 12, 2022
@RickWinter RickWinter added the EngSys This issue is impacting the engineering system. label Apr 12, 2022
@ahsonkhan
Copy link
Member

This failure might be related, on MacOS:
https://dev.azure.com/azure-sdk/public/_build/results?buildId=1678165&view=logs&j=e3a64eae-3ca2-515e-3e44-1207c46c3ba5&t=1b9ced73-614e-5671-7440-5a208be067b9&s=6884a131-87da-5381-61f3-d7acc3b91d76

2022-06-30T19:03:17.5733870Z ##[section]Starting: Publish failed bom_manifest_$(BomArtifactName) Artifacts
2022-06-30T19:03:17.5749340Z ==============================================================================
2022-06-30T19:03:17.5750030Z Task : Publish Pipeline Artifacts
2022-06-30T19:03:17.5750990Z Description : Publish (upload) a file or directory as a named artifact for the current run
2022-06-30T19:03:17.5751560Z Version : 1.199.0
2022-06-30T19:03:17.5751970Z Author : Microsoft Corporation
2022-06-30T19:03:17.5752560Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/publish-pipeline-artifact
2022-06-30T19:03:17.5753640Z ==============================================================================
2022-06-30T19:03:18.1304210Z Artifact name input: bom_manifest_$(BomArtifactName)-FailedAttempt1
2022-06-30T19:03:18.1568400Z Uploading pipeline artifact from /Users/runner/work/1/a/_manifest for build #1678165
2022-06-30T19:03:18.9496850Z Using default max parallelism.
2022-06-30T19:03:18.9560630Z Max dedup parallelism: 192
2022-06-30T19:03:19.8868400Z ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session ee2f3f1c-7e77-4e0f-9060-f918036595cf
2022-06-30T19:03:19.9252760Z DedupManifestArtifactClient will correlate http requests with X-TFS-Session ee2f3f1c-7e77-4e0f-9060-f918036595cf
2022-06-30T19:03:21.0987740Z 4 files processed.
2022-06-30T19:03:21.0997520Z Processed 4 files from /Users/runner/work/1/a/manifest successfully.
2022-06-30T19:03:25.2491820Z Uploading 4 files from directory /Users/runner/work/1/a/manifest.
2022-06-30T19:03:25.2556530Z Uploaded 0 out of 11,442 bytes.
2022-06-30T19:03:28.5020890Z Uploaded 12,026 out of 12,026 bytes
2022-06-30T19:03:28.5022890Z Content upload is done!
2022-06-30T19:03:28.5040420Z
2022-06-30T19:03:28.5041090Z Content upload statistics:
2022-06-30T19:03:28.5041930Z Total Content: 23.5 KB
2022-06-30T19:03:28.5042450Z Physical Content Uploaded: 3.5 KB
2022-06-30T19:03:28.5043020Z Logical Content Uploaded: 12.0 KB
2022-06-30T19:03:28.5043570Z Compression Saved: 8.5 KB
2022-06-30T19:03:28.5044110Z Deduplication Saved: 11.4 KB
2022-06-30T19:03:28.5044950Z Number of Chunks Uploaded: 5
2022-06-30T19:03:28.5045500Z Total Number of Chunks: 9
2022-06-30T19:03:28.5046420Z
2022-06-30T19:03:29.5381990Z ApplicationInsightsTelemetrySender correlated 2 events with X-TFS-Session ee2f3f1c-7e77-4e0f-9060-f918036595cf
2022-06-30T19:03:29.5391340Z ##[error]Artifact bom_manifest$(BomArtifactName)-FailedAttempt1 already exists for build 1678165.
2022-06-30T19:03:29.5523710Z ##[section]Finishing: Publish failed bom_manifest$(BomArtifactName) Artifacts

@danieljurek
Copy link
Member Author

This issue was fixed a while back. No more collisions on bom manifest uploads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danieljurek danieljurek

Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Central-EngSys This issue is owned by the Engineering System team. EngSys This issue is impacting the engineering system.
Projects
Azure SDK EngSys 📈
Status: 🎊 Closed
Milestone
No milestone
Development

No branches or pull requests
3 participants
@danieljurek @RickWinter @ahsonkhan