Fix extraction of Xamarin.Build.Download #1048

maonaoda · 2024-11-16T15:51:45Z

Before fix:

2024-11-16.23.39.10.mov

After fix:

moljac · 2024-11-17T08:49:30Z

util/Xamarin.Build.Download/source/Xamarin.Build.Download.Tests/Test.cs

+
+			prel.AddItem (
+				"XamarinBuildDownload" , "GAppM-10.28.0" , new Dictionary<string , string> {
+					{ "Url", "https://dl.google.com/firebase/ios/analytics/4d5ec9a36b6d4fd4/GoogleAppMeasurement-10.28.0.tar.gz" },


Maybe we'll need to move this test data somewhere else.

moljac

LGTM with comment to move test data somewhere else

jpobst

I feel like we're missing the root cause of the issue here, and this is just a workaround.

The reported issue is that the C:\Users\wangyk\AppData\Local\XamarinBuildDownloadCache\GAppM-10.28.0\GAppM-10.28.0.tar file already exists.

If the file already exists then we should not be downloading it again in the first place, so the question is "why are we trying to download this?".

My guess is that maybe a previous extraction failed before the GAppM-10.28.0.unpacked file got written.

If you delete your C:\Users\wangyk\AppData\Local\XamarinBuildDownloadCache\GAppM-10.28.0 directory and then try again, do you still see this error? If so, did the GAppM-10.28.0.unpacked file get written?

We need to figure out if:

This was a one time bad state and clearing the directory fixes it.
There is an issue writing the .tar, extracting the .tar file, or writing the .unpacked file.
Everything is extracted correctly and we aren't detecting the existing files and are trying to run the process a second time when it isn't needed.

I tried running the unit test locally without the overwrite change and it succeeded. I'm not sure how the test could fail because it always tries writing to a fresh Path.GetTempFileName (), so there should never be an existing file that needs to be overwritten. If the test is failing for you locally, hopefully debugging it will help determine which of the above scenarios is occuring.

maonaoda · 2024-11-21T09:38:33Z

Yes, a new folder will be created every time, but I will definitely reproduce this

maonaoda · 2024-11-21T09:43:41Z

It has been staying in Extracting

テストプロジェクトのビルド
========== テストの実行を開始しています ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET 9.0.0)
[xUnit.net 00:00:00.46] Starting: Xamarin.Build.Download.Tests
Build started.
Project "project.csproj" (_XamarinBuildDownload target(s)):
Target "_XamarinBuildAddDownloadedItems" in file "D:\practice\android-libraries\util\Xamarin.Build.Download\source\Xamarin.Build.Download.Tests\bin\Release\net9.0\Xamarin.Build.Download.targets" from project "C:\Users\wangyk\AppData\Local\Temp\tmpavnnef.tmp\project.csproj" (target "_XamarinBuildDownload" depends on it):
Done building target "_XamarinBuildAddDownloadedItems" in project "project.csproj".
Target "_XamarinBuildDownloadCore" in file "D:\practice\android-libraries\util\Xamarin.Build.Download\source\Xamarin.Build.Download.Tests\bin\Release\net9.0\Xamarin.Build.Download.targets" from project "C:\Users\wangyk\AppData\Local\Temp\tmpavnnef.tmp\project.csproj" (target "_XamarinBuildDownload" depends on it):
The ".overridetasks" files could not be successfully loaded from their expected location "D:\practice\android-libraries\util\Xamarin.Build.Download\source\Xamarin.Build.Download.Tests\bin\Release\net9.0". Default tasks will not be overridden.
The ".tasks" files could not be successfully loaded from their expected location "D:\practice\android-libraries\util\Xamarin.Build.Download\source\Xamarin.Build.Download.Tests\bin\Release\net9.0". Default tasks will not be available.
Using "XamarinDownloadArchives" task from assembly "D:\practice\android-libraries\util\Xamarin.Build.Download\source\Xamarin.Build.Download.Tests\bin\Release\net9.0\Xamarin.Build.Download.dll".
Task "XamarinDownloadArchives"
D:\Program Files\7-Zip
C:\Program Files\Microsoft Visual Studio\2022\Enterprise
Downloading https://dl.google.com/firebase/ios/analytics/4d5ec9a36b6d4fd4/GoogleAppMeasurement-10.28.0.tar.gz to C:\Users\wangyk\AppData\Local\Temp\tmpavnnef.tmp\unpacked\GAppM-10.28.0.tgz
(2271/10038515b), total 0.0%
(1004512/10038515b), total 10.0%
(2015232/10038515b), total 20.0%
(3014640/10038515b), total 30.0%
(4030448/10038515b), total 40.0%
(5029856/10038515b), total 50.0%
(6029280/10038515b), total 60.0%
(7028688/10038515b), total 70.0%
(8044496/10038515b), total 80.0%
(9043904/10038515b), total 90.0%
(10038515/10038515b), total 100.0%
Downloading Complete
Extracting C:\Users\wangyk\AppData\Local\Temp\tmpavnnef.tmp\unpacked\GAppM-10.28.0.tgz to C:\Users\wangyk\AppData\Local\Temp\tmpavnnef.tmp\unpacked\GAppM-10.28.0

The only difference is that I set the compilation environment to net9

maonaoda · 2024-11-21T09:48:54Z

maonaoda · 2024-11-21T15:01:48Z

It is not a download, but a decompression problem of GAppM-10.28.0

jpobst · 2024-11-21T18:44:18Z

It is not a download, but a decompression problem of GAppM-10.28.0

Sure, but I still don't understand what the root issue is.

When we decompress GAppM-10.28.0, why do the file(s) already exist?

If this is the first time it is running, then the folder should be empty and there won't be any existing files.
If this is the second time it is running, it should have detected that the file(s) already exist and it shouldn't be downloading or extracting them again.

Perhaps another way to explore this:

Does the TestGMapsDownload test fail for you as well without this change?
If not, why does it succeed but TestGoogleAppMeasurementDownload hangs? They are exactly the same test but with a different .tar.gz file, so what makes GoogleAppMeasurement-10.28.0.tar.gz hang while c0e534927c0c955e-GoogleMaps-1.11.1.tar.gz succeeds?

maonaoda · 2024-11-22T11:35:15Z

In fact, tar.gz performs two extractions. The first extraction is successful, but during the second extraction, a confirmation about the existence of the file will appear.

maonaoda · 2024-11-22T13:25:49Z

The first extraction:

psi

	名前	値	種類
◢	psi	FileName = "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Xamarin.VisualStudio\7-Zip\7z.exe",Arguments = "e "-oC:\Users\wangyk\AppData\Local\Temp\tmpp0glvd.tmp\unpacked\GAppM-10.28.0" "C:\Users\wangyk\AppData\Local\Temp\tmpp0glvd.tmp\unpacked\GAppM-10.28.0.tgz"", WorkingDirectory = ""	System.Diagnostics.ProcessStartInfo

The next extraction:

psi

	名前	値	種類
◢	psi	FileName = "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Xamarin.VisualStudio\7-Zip\7z.exe", Arguments = "x -snl- "-oC:\Users\wangyk\AppData\Local\Temp\tmpp0glvd.tmp\unpacked\GAppM-10.28.0" "C:\Users\wangyk\AppData\Local\Temp\tmpp0glvd.tmp\unpacked\GAppM-10.28.0\GAppM-10.28.0.tar"", WorkingDirectory = ""	System.Diagnostics.ProcessStartInfo

maonaoda · 2024-11-22T13:42:17Z

Now that we have obtained the execution parameters for the two decompressions, try to execute them in PowerShell

& "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Xamarin.VisualStudio\7-Zip\7z.exe" e "-oD:\Test\GAppM-10.28.0" "D:\Test\GAppM-10.28.0.tgz"

PS D:\Test> & "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Xamarin.VisualStudio\7-Zip\7z.exe" e "-oD:\Test\GAppM-10.28.0" "D:\Test\GAppM-10.28.0.tgz"

7-Zip 18.05 (x86) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30

Scanning the drive for archives:
1 file, 10038515 bytes (9804 KiB)

Extracting archive: D:\Test\GAppM-10.28.0.tgz
--
Path = D:\Test\GAppM-10.28.0.tgz
Type = gzip
Headers Size = 10

Everything is Ok

Size:       40711680
Compressed: 10038515

& "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Xamarin.VisualStudio\7-Zip\7z.exe" x -snl- "-oD:\Test\GAppM-10.28.0" "D:\Test\GAppM-10.28.0\GAppM-10.28.0.tar"

PS D:\Test> & "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Xamarin.VisualStudio\7-Zip\7z.exe" x -snl- "-oD:\Test\GAppM-10.28.0" "D:\Test\GAppM-10.28.0\GAppM-10.28.0.tar"

7-Zip 18.05 (x86) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30

Scanning the drive for archives:
1 file, 40711680 bytes (39 MiB)

Extracting archive: D:\Test\GAppM-10.28.0\GAppM-10.28.0.tar

WARNINGS:
Headers Error

--
Path = D:\Test\GAppM-10.28.0\GAppM-10.28.0.tar
Type = tar
WARNINGS:
Headers Error
Physical Size = 40711680
Headers Size = 124416
Code Page = UTF-8


Would you like to replace the existing file:
  Path:     D:\Test\GAppM-10.28.0\GoogleAppMeasurement-10.28.0\Frameworks\GoogleAppMeasurementIdentitySupport.xcframework\ios-arm64_x86_64-maccatalyst\GoogleAppMeasurementIdentitySupport.framework\PaxHeader\GoogleAppMeasurementIdentitySupport
  Size:     44 bytes (1 KiB)
  Modified: 2024-06-11 03:14:30
with the file from archive:
  Path:     .\GoogleAppMeasurement-10.28.0\Frameworks\GoogleAppMeasurementIdentitySupport.xcframework\ios-arm64_x86_64-maccatalyst\GoogleAppMeasurementIdentitySupport.framework\PaxHeader\GoogleAppMeasurementIdentitySupport
  Size:     14 bytes (1 KiB)
  Modified: 2024-06-11 03:11:10
? (Y)es / (N)o / (A)lways / (S)kip all / A(u)to rename all / (Q)uit?

maonaoda · 2024-11-22T13:50:26Z

When extracting GAppM-10.28.0.tgz and GAppM-10.28.0.tar, it prompts whether to overwrite because they contain files with the same name.

Is this a problem unique to GAppM-10.28.0?

maonaoda added 2 commits November 16, 2024 23:22

fix: fix extraction

e7d07f2

test: add test

6b6ccd4

maonaoda changed the title ~~fix: fix extraction~~ Fix extraction of Xamarin.Build.Download Nov 16, 2024

dellis1972 requested a review from jpobst November 16, 2024 16:48

moljac reviewed Nov 17, 2024

View reviewed changes

moljac approved these changes Nov 17, 2024

View reviewed changes

jpobst reviewed Nov 20, 2024

View reviewed changes

maonaoda mentioned this pull request Nov 22, 2024

Build stuck on Extracting Extracting ...\XamarinBuildDownloadCache\GMps-9.1.1.tgz to ...\XamarinBuildDownloadCache\GMps-9.1.1 dotnet/maui#26045

Closed

LeoJHarris mentioned this pull request Dec 3, 2024

App will not install on device with Sign in with Apple capability in the provisioning profile and com.apple.developer.applesignin key is in Entitlements.plist after publishing. xamarin/xamarin-macios#21645

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix extraction of Xamarin.Build.Download #1048

Fix extraction of Xamarin.Build.Download #1048

maonaoda commented Nov 16, 2024 •

edited

Loading

moljac Nov 17, 2024

moljac left a comment

jpobst left a comment

maonaoda commented Nov 21, 2024

maonaoda commented Nov 21, 2024

maonaoda commented Nov 21, 2024

maonaoda commented Nov 21, 2024

jpobst commented Nov 21, 2024

maonaoda commented Nov 22, 2024

maonaoda commented Nov 22, 2024

maonaoda commented Nov 22, 2024 •

edited

Loading

maonaoda commented Nov 22, 2024

Fix extraction of Xamarin.Build.Download #1048

Are you sure you want to change the base?

Fix extraction of Xamarin.Build.Download #1048

Conversation

maonaoda commented Nov 16, 2024 • edited Loading

moljac Nov 17, 2024

Choose a reason for hiding this comment

moljac left a comment

Choose a reason for hiding this comment

jpobst left a comment

Choose a reason for hiding this comment

maonaoda commented Nov 21, 2024

maonaoda commented Nov 21, 2024

maonaoda commented Nov 21, 2024

maonaoda commented Nov 21, 2024

jpobst commented Nov 21, 2024

maonaoda commented Nov 22, 2024

maonaoda commented Nov 22, 2024

maonaoda commented Nov 22, 2024 • edited Loading

maonaoda commented Nov 22, 2024

maonaoda commented Nov 16, 2024 •

edited

Loading

maonaoda commented Nov 22, 2024 •

edited

Loading