Oh, and the files I'm uploading here to test with are 3 to 5 MB. Certainly not Killer 136 - 200 MB files like I sometimes have. 
Normal end upload "saving..." pause is 1-2 seconds at most, not the 30 seconds give or take for my "production" rPi units.
Last point of interest - possibly.
The three units with the delay have 64GB cards, the Spare #4 without the delay is only (grin) a 32GB card.
Analysis indeed happens in a different thread than the upload. It only gets enqueued then, that's all. It cannot hold the upload from completing. What happens between 100% and saving is moving the file to its final location. Upload happens to a temporary file on disk. That path then gets handed over to the upload handler, which moves the file to its final destination and collects some metadata (on a clean/safe mode OctoPrint that is only a file hash).
I see two possible reasons for a slow down here. One is that the move is not an atomic move but instead has to be done as a copy and delete, which would happen if your temporary directory is located on a different partition or drive than the final upload folder. The other is there being a whole TON of files in the target folder, meaning the metadata.json file that has to be first read, then updated, then written back to disk takes a lot of time to process.
I have subfolders on each device that drills down to no more than 40 files in a folder (and usually between 1 and 10 per folder). I can certainly try backing up one of the .octoprint/uploads folder on a slow unit to my Spare and see if replicating that structure causes the issue to appear.
I thought the metadata.json might be an issue, so I had started with a newly created folder. Problem occurs with a newly created folder, too. That should not have a metadata.json in it until I upload a .gcode. I verified that to be true with WinSCP.
There are only two partitions on the OctoPi. the FAT32 pre-boot partition and the ext4 of the Linux system.
Does the temporary directory (unsure where that is) contain a metadata.json?
Interesting. While watching the folder with WinSCP during the save, and refreshing I noticed that the metadata.json and the uploaded .gcode gets written to the target folder nearly immediately (one to two seconds max) - then the hang starts. And if I try to refresh my browser, It hangs until that ~30 seconds pass. Replicated this with Firefox and Chrome.
Good news, the delay is down to 24 seconds now.
@foosel - As I have three units all doing the same thing, I can try reimaging one with stock OctoPi 0.17, and upgrading to 1.4.2 (no python upgrade) and see if this issue reproduces itself, then do the Python 3 update and repeat tests. If this sounds like a valid test, I will make that so.
Original upgrade path was OctoPi 0.17 with it's OctoPrint 1.3x, upgraded to 1.40 (and in use for months with no issue), then 1.42 upgrade last week, followed closely by Python 3. I did not upload to my production Pi's on 1.42 before upgrading Python. Thoughts?
Certainly sounds like a good test to do, please make it so 
OctoPi included OctoPrint 1.3.12 - uploads instant. "Saving" bar doesn't even show 100% before file is listed.
Upgraded to 1.4.2 - Same excellent results.
Restore my back up (no .gcode's) - Same excellent results.
Restore my .gcodes (8 Gigs, 720 Files, 154 folders) - slowness pops up. File uploads take 15-20 seconds to complete once the bar hits 100%. Also, creating a folder takes a few seconds now.
Python isn't upgraded yet, but I don't think we need to look at that yet.
So, it would look like perhaps there is a difference in file saving/handling & folder creation between 1.4.0 and 1.4.2 affecting large upload collections?
I've just pushed a change to the maintenance branch that should hopefully mitigate that. I've added more granular and aggressive caching right into the local storage manager, on a folder base. If you now upload to a folder, only the data for that folder will be invalidated, everything else can still be served from the cache instead of having to be rescanned again from the file system. Greatly reduces the time needed to produce a file list in such cases. Before, it meant the whole recursive scan needed to be done again since caching only happened at storage level.
Will be part of 1.5.0.
I'll still look though if I can find anything that changed between 1.4.0 and 1.4.1/2 that would explain worse performance here in the first place.
Thank you @foosel - that does explain why larger collections could experience the delay to begin with. And likely would not change results if I swapped rPi boards around.
10 existing files in the folder, I upload one, the scan hits all 11 files to generate a fresh .metadata.json file?
I'm still on release/stable - and tried again (before trying to switch over to maintenance branch) on rPi #1 through #4 - mostly to confirm status of the issue.
I'm now down to a 6-8 second delay on my #1 re-imaged, 1.4.2/Python2 device. I have not printed (maybe once) or rebooted that device since I last posted. No updates, no plug-in changes, nothing.
Units #2 and #3 (1.4.2/Python3) are at about a 25 second delay.
Unit #4 - my Spare that had no issues before, now has a 5-6 second delay. That is with two files in the upload root, and a folder. Created a folder, and uploaded to that folder, and now there's a delay. (#4 is OctoPrint 1.4.2, Python 3.7.3, OctoPi 0.17.0, same as Unit #2 and #3)
I have no idea why #1 is gaining speed, while #2 and #3 are the same speed, and #4 is slowing down. Same model rPi, and quality, fast Samsung SD cards.
Curious.
And I'm sorry this doesn't seem to help much.
My next test will be take my #4, load 1 GB of uploads, and retest. Then go maintenance release, load the entire 4.8 GB upload folder to it and perform the "create folder, upload file" test.
Question: does the .metadata.json file in each upload folder tie in any way back to the original rPi, in that the hash value is based in part on the salt, or is it more independent like a CRC32 of the file? i.e. should I copy the .metadata.json in each upload folder between units, or should I not copy it?
The hash is just a hash of the file itself (md5sum if I remember correctly, so nothing fancy), so definitely not device dependent. It's weird that you get so different results. Just to rule out any third party plugins interfering here - best make any kind of tests in safe mode if you don't already. Plugins that process the file during upload (there are hooks for that) COULD cause this, completely unrelated to the library size.
I've also spent some hours yesterday going over the code and trying to figure out if there are any specific changes between 1.4.0 and 1.4.2 that could cause a slowdown here but I haven't found anything. I'm wondering though if I should add a new status to that progress bar to at least give us a bit more to work with in the future - "Saving..." will be displayed while the file is still being written to the backend, but also until the fresh file list has been fetched from the server. So it's currently not possible to say if it staying on "saving" for a long time means it's being slow to write or slow to refresh the list. I'll see that I can make this more transparent so that we get more options for debugging such situations in the future.
@foosel I don't know if this is suitable for a cross platform application like OctoPrint but on Linux one can use inotify to keep the cached version of a directory tree in sync with the filesystem. Maybe there are similar functionalities available on Windows and Mac.
For Python on Linux I have found PyInotify
I have used this for a c++ application and it was moderately easy.
I see little to no difference in safe mode on any of the units. An upload may take a couple seconds less to upload in safe mode, but it's the difference between a 30 second pause and a 28 second pause.
Same thing on my mostly empty #4 Spare - it goes from 5 seconds to, well, 4.5 seconds.
Certainly having a Saving.../Scanning.../Refreshing List... status on an upload may clue in further where that delay is. I like that idea! 
With the slight variations in time (30 second delay one day, 25 the next) It's almost like there's mandatory wear leveling forced on the microSD card or the volume with 1.4.2 that wasn't in place with 1.4.0.
If I were to add a USB 3.0 thumbdrive, and moved the uploads to it, I wonder what would happen...
I hate to moldypost like this, but... yes, the additional status change was noticed and it sits at the "Refreshing List" message point since I last posted.
Continued to update, now running 1.5.3, I still get the delay at the "Refreshing List" message.
BUT! I noticed that if I upload a file, wait for the "Refreshing List" delay to clear, then start a print job, I can immediately upload further files with no delay. Uploads take seconds like expected.
Now at: OctoPrint 1.5.3 Python 3.7.3 OctoPi 0.17.0
Next test will be with a OctoPi 0.18 image once I backup/image/restore.
Now at OctoPrint 1.5.3 Python 3.7.3 OctoPi 0.18.0
Fresh image of octopi-buster-armhf-lite-0.18.0 onto a 64GB Samsung TF card
About a 20-25 second delay on the upload.
looking at the log:
2021-03-12 03:04:32,836 - octoprint.filemanager.analysis - INFO - Starting analysis of local:Test/Folder/Testfile.gcode
2021-03-12 03:04:32,837 - octoprint.filemanager.analysis - INFO - Invoking analysis command: /home/pi/oprint/bin/python3 -m octoprint analysis gcode --speed-x=6000 --speed-y=6000 --max-t=10 --throttle=0.0 --throttle-lines=100 /home/pi/.octoprint/uploads/Test/Folder/Testfile.gcode
2021-03-12 03:04:54,677 - octoprint.filemanager.analysis - INFO - Analysis of entry local:Test/Folder/Testfile.gcode finished, needed 21.84s
so appears to be waiting for the analysis to complete.
Analysis should be happening asynchronously and not make the upload wait. But I can reproduce. So that is a bug and needs to be investigated.
ETA It doesn't do it always though... my current guess is that there's a race condition in there somewhere. A lock that sometimes gets taken by the API and sometimes by the analysis would explain this behaviour, but right now I don't see something like that.
@m.hutchinson Can you do me a favor and instead of uploading a file simply click the refresh icon on the list and share how long that takes? Should be visible in the log or your browser's network console. After further testing here I cannot confirm that it waits for the analysis, I explicitly set that to wait 20s more and the request still isn't as fast as I wish it was, but definitely finishes before the analysis finishes.
Refreshes show about 2.2 - 2.4 seconds in the Firefox network console. This is testing with a few folders, with between 1 and 10 files, and 1-35 folders in each.
I did it from the root, and with probably 200 folders and an average of 6 .gcodes per folder, it took about 2.3 sec. Each file in the log took around 6-50ms each, ending in:
2021-03-12 15:02:23,399 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726cd370>,'/home/pi/.octoprint/uploads',base='',force_refresh=True) took 1730.04ms
2021-03-12 15:02:48,722 - octoprint.server.util.sockjs - INFO - Client connection closed: ::ffff: (removed IP)
THis is really weird. I just tried this on a fresh install on a Pi3+, definitely no waiting for the analysis. Then I transferred my testing library with 450MB of various files, folders and whatnot over, tried again, and now I seem to see a wait. It doesn't make any sense at all so far.
Case in point, fresh:
2021-03-12 14:42:20,508 - octoprint.filemanager.analysis - INFO - Starting analysis of local:trispinner_prusa.gcode
2021-03-12 14:42:20,510 - octoprint.filemanager.analysis - INFO - Invoking analysis command: /home/pi/oprint/bin/python3 -m octoprint analysis gcode --speed-x=6000 --speed-y=6000 --max-t=10 --throttle=0.0 --throttle-lines=100 /home/pi/.octoprint/uploads/trispinner_prusa.gcode
2021-03-12 14:42:20,537 - tornado.access - INFO - 201 POST /api/files/local (::ffff:192.168.1.3) 1526.08ms
2021-03-12 14:42:20,551 - tornado.access - INFO - 200 POST /api/files/local (::ffff:192.168.1.3) 1540.22ms
2021-03-12 14:42:20,641 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads',base='',force_refresh=False) took 29.56ms
2021-03-12 14:42:20,674 - octoprint.server.api.files.timings - DEBUG - octoprint.server.api.files._getFileList('local',filter=False,recursive=True,allow_from_cache=True) took 63.53ms
2021-03-12 14:42:20,674 - octoprint.server.api.files.timings - DEBUG - octoprint.server.api.files._getFileList('sdcard',) took 0.03ms
2021-03-12 14:42:20,741 - tornado.access - INFO - 200 GET /api/files?recursive=true (::ffff:192.168.1.3) 149.74ms
2021-03-12 14:42:31,361 - octoprint.filemanager.analysis - INFO - Analysis of entry local:trispinner_prusa.gcode finished, needed 10.85s
2021-03-12 14:42:31,397 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads',base='',force_refresh=False) took 4.85ms
2021-03-12 14:42:31,400 - octoprint.server.api.files.timings - DEBUG - octoprint.server.api.files._getFileList('local',filter=False,recursive=True,allow_from_cache=True) took 8.85ms
2021-03-12 14:42:31,406 - octoprint.server.api.files.timings - DEBUG - octoprint.server.api.files._getFileList('sdcard',) took 0.02ms
2021-03-12 14:42:31,415 - tornado.access - INFO - 200 GET /api/files?recursive=true (::ffff:192.168.1.3) 31.23ms
And with 450MB library:
2021-03-12 15:00:19,293 - tornado.access - INFO - 201 POST /api/files/local (::ffff:192.168.1.3) 3297.66ms
2021-03-12 15:00:19,313 - tornado.access - INFO - 200 POST /api/files/local (::ffff:192.168.1.3) 3318.07ms
2021-03-12 15:00:30,723 - octoprint.filemanager.analysis - INFO - Analysis of entry local:pia/trispinner_prusa.gcode finished, needed 11.49s
2021-03-12 15:00:31,373 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/pia',base='pia/',force_refresh=False) took 1.89ms
2021-03-12 15:00:31,376 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/biqu_bx',base='biqu_bx/',force_refresh=False) took 2.06ms
2021-03-12 15:00:31,381 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_2773',base='test/issue_2773/',force_refresh=False) took 0.24ms
2021-03-12 15:00:31,384 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_2629',base='test/issue_2629/',force_refresh=False) took 1.87ms
2021-03-12 15:00:31,387 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/gcodeviewer_lag',base='test/gcodeviewer_lag/',force_refresh=False) took 1.94ms
2021-03-12 15:00:31,389 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3930',base='test/issue_3930/',force_refresh=False) took 1.95ms
2021-03-12 15:00:31,392 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3387',base='test/issue_3387/',force_refresh=False) took 1.92ms
2021-03-12 15:00:31,394 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3437',base='test/issue_3437/',force_refresh=False) took 1.94ms
2021-03-12 15:00:31,397 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/foo',base='test/foo/',force_refresh=False) took 1.90ms
2021-03-12 15:00:31,401 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3541/Folder_1/Sub_Folder_1',base='test/issue_3541/Folder_1/Sub_Folder_1/',force_refresh=False) took 0.20ms
2021-03-12 15:00:31,403 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3541/Folder_1/Sub_Folder_2',base='test/issue_3541/Folder_1/Sub_Folder_2/',force_refresh=False) took 1.87ms
2021-03-12 15:00:31,404 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3541/Folder_1',base='test/issue_3541/Folder_1/',force_refresh=False) took 3.84ms
2021-03-12 15:00:31,404 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3541',base='test/issue_3541/',force_refresh=False) took 6.73ms
2021-03-12 15:00:31,409 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test/issue_3057',base='test/issue_3057/',force_refresh=False) took 4.46ms
2021-03-12 15:00:31,409 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/test',base='test/',force_refresh=False) took 32.71ms
2021-03-12 15:00:31,412 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/h2',base='h2/',force_refresh=False) took 1.99ms
2021-03-12 15:00:31,414 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/foo',base='foo/',force_refresh=False) took 1.74ms
2021-03-12 15:00:31,418 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/blurb',base='haussge/blurb/',force_refresh=False) took 0.18ms
2021-03-12 15:00:31,422 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/test/wolf_face',base='haussge/test/wolf_face/',force_refresh=False) took 0.18ms
2021-03-12 15:00:31,424 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/test/blorb',base='haussge/test/blorb/',force_refresh=False) took 1.83ms
2021-03-12 15:00:31,426 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/test/bloorb',base='haussge/test/bloorb/',force_refresh=False) took 1.77ms
2021-03-12 15:00:31,429 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/test/dog_face',base='haussge/test/dog_face/',force_refresh=False) took 1.63ms
2021-03-12 15:00:31,431 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/test/test',base='haussge/test/test/',force_refresh=False) took 1.68ms
2021-03-12 15:00:31,433 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/test/blarb',base='haussge/test/blarb/',force_refresh=False) took 1.79ms
2021-03-12 15:00:31,434 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge/test',base='haussge/test/',force_refresh=False) took 15.03ms
2021-03-12 15:00:31,434 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/haussge',base='haussge/',force_refresh=False) took 19.28ms
2021-03-12 15:00:31,439 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads/another_test',base='another_test/',force_refresh=False) took 4.14ms
2021-03-12 15:00:31,443 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x726fe030>,'/home/pi/.octoprint/uploads',base='',force_refresh=False) took 10403.85ms
2021-03-12 15:00:31,505 - octoprint.server.api.files.timings - DEBUG - octoprint.server.api.files._getFileList('local',filter=False,recursive=True,allow_from_cache=True) took 11162.68ms
2021-03-12 15:00:31,506 - octoprint.server.api.files.timings - DEBUG - octoprint.server.api.files._getFileList('sdcard',) took 0.03ms
2021-03-12 15:00:31,535 - tornado.access - INFO - 200 GET /api/files?recursive=true (::ffff:192.168.1.3) 12196.06ms
Interesting. Where you get:
/home/pi/.octoprint/uploads',base='',force_refresh=False) took 10403.85ms
On a folder refresh, mine shows:
force_refresh=True) took 1816.68ms
and on a reboot:
force_refresh=False) took 14236.09ms
and I just spotted this earlier in the scan:
2021-03-12 17:41:12,694 - octoprint.filemanager.storage.timings - DEBUG - octoprint.filemanager.storage._list_folder(<octoprint.filemanager.storage.LocalFileStorage object at 0x728a4650>,'/home/pi/.octoprint/uploads/FDG',base='FDG/',force_refresh=False) took 14033.18ms
FDG is my largest folder (contents are around 5.5 Gigs), it is in the root, and it's got a ton of sub-folders and files. Probably 80% of my storage is this folder. Understandable to take a long time to scan when asked for, but should not be included in the scan when I upload a file.
So, happy to see it's reproducible for you at least.
The problem at its core is probably that right now OctoPrint doesn't just refresh the folder you uploaded to, it always refreshes the whole tree. I use some heavy internal caching here to make sure it only actually has to crawl all the files from the changed subtree again, but that still means it'll have to walk the full tree.
We might simply see the limitations of that, though I'm not entirely sure yet.
I'll try to continue to dig, but it might just need a change in the refresh approach altogether after all.
Danke, Gina. I'll consider it a known issue then, and continue to live with the delay for now.
I suppose it's a good thing I don't have it pointed at my NAS folder where I save all my .gcodes to before uploading to each Pi. 
