How to screen a 20TB library for corrupted files?
How to screen a 20TB library for corrupted files?
Hi All,
About a year ago I transferred all my files to a new drive. I used filzezilla which did mostly ok-ish, but I didn't notice that some of the video files were corrupted. Random files will have a green tinge to them (like someone put a green filter over the lens).
It seems random, although if it's a series it's usually the whole series.
I've been replacing them as they come up, but I was wondering if anyone had any bright ideas to expedite the process.
Thanks for any help!
green tinge to them
Are you sure it's corrupted and not an unsupported HDR color format?
Corrupted files usually skip frames or get blocking and ghosting errors.
I certainly am not sure lol. I did try disabling the HDR tone mapping to no affect. It's possible this is the issue as when I transferred the library, it was to new hardware with a different GPU.
Is there a way to tell the color format from the file info?
Thank you!
Edit: I wanted to add context, as I think this may be the culprit. I initially transferred the files from one machine to another via filezilla. About a week after, we had a power outage, which screwed up the SSD that had the operating system (lesson learned about surge protectors). To get the Plex back and running quickly, I simply pulled the physical hard-drive, and popped it into a 3rd machine. So it does make sense to me that the file itself may be fine.
edit2: @[email protected] you are definitely onto it! I just downloaded the file to another machine, and it played with no color issues. So my guess is it's something to do with the GPU on the machine hosting plex?
Navigate to a problematic movie or episode -> 3 dots -> Get info -> right column -> Color space
Usually SDR is bt709 and HDR is bt2020. In older codecs (h264) 10 or 12 bit color depth can also cause issues.
I'm not sure how tone mapping works on plex since I don't have a pass but with Jellyfin you need to setup OpenCL which is run on the GPU so your guess that a hardware change can break it is plausible.
Tdarr and its Health Checks is the first thing that comes to mind:
this looks like it may be helpful, but I'm not super clear on what it does exactly. It looks like it does conditional transcoding, but I'm not clear on what that means. Would this screw with plexs own traditional transcoding? Or does it attempt to repair a file by transcoding.
Thank you for the input!
The health check that Tdarr can do depend on what health check you select, the "quick" health check will only check the file headers. This probably won't find any video corruption unless the file headers are corrupted or invalid.
The thourough health check will basically do a transcode of the input file without an output. This is being used to run and check each single frame of the file and "process" it without actually producing an output. https://trac.ffmpeg.org/wiki/Null
Would this screw with plexs own traditional transcoding?
No, Plex and Tdarr are two completely different things.
Transcoding is not some universal thing but rather a term to describe what is happening. In general "transcoding" means that you convert something from one format into another.
When you use the processing pipeline through Tdarr, it would detect a file in your library and run it through your specified pipeline. Depending on what you want to do with the file and how the pipeline is configured, it might or might not do anything with that specific file. When you, for example, configure that you want any file that has a video stream that isn't encoded with HEVC to be HEVC then a H.264 file would then be transcoded.
This is on a "replacement" bases, so any file you run through this would replace the existing file in your library. Meaning: you have a H.264 file before tdarrs processing and after it is finished, you have the file with HEVC as video stream.
With the health check, as explained above, this is a bit different since you don't have an ouput file, there is also no replacement happening. It is just a "check" that the file can be transcoded in its entirety (with a thourough health check).
When Plex transcodes something, this is done "on the fly". This means that it only transcodes the current file in your library.
Technically, tdarr could "screw" with the Plex transcoder when Tdarr processes a file while you yourself watch the same file that is then transcoded by plex. When Tdarr is finished, it would replace the file which then could throw off the Plex transcoder. But this is very rare or non existent and you could even configure tdarr to do those things only at a certain time and it wouldn't happen with health checks. as someone who runs both, I have yet to come across something like this.
Or does it attempt to repair a file by transcoding
As the name suggests, health checks are only "checking" if the file is healthy, there is nothing being repaired here. While you might be able to repair the file header, a corrupted video or audio is not something you can repair because of missing information. But that is nothing that Tdarr will do anyway, it just verifies/checks if it is healthy or not.
Tdarr has many usages revolving around transcoding. I use it for space optimization mainly but you can build your own workflow/pipeline to only do integrity checks… It will check that the file is eligible prior to attempting to transcode and check new files added to your library as Tdarr is designed to run continuously, checking and transcoding as configured. I’m not sure it is relevant to your specific issue but you could try the handbrake/ffmpeg health checks commands outside of Tdarr first on a file you know has the problem and if it works, automate the thing through Tdarr to do the same check on the entire folder/library. This would only help you in flagging them. I’m not sure about repairing though and usually I would look for a new version in case of issues. The question on the transcoding behavior is complex to answer as it depends on hardware on both server and client but you could run Tdarr without changing anything on this side. Please do go through the Tdarr documentation before actually running it across your entire library. My concern is that on my library none came back with a failed health check while a few do fail during transcode to x265 so again I don’t know if that will work for you.
Making a toplevel comment to say thank you! Especially to @[email protected] , @[email protected] and @[email protected] .
- The files are not corrupted.
- Disabling hardware acceleration removed the issue entirely. (edit: "use hardware acceleration when available" is unchecked, "use hardware-accelerated video encoding" remains active"
So something in the way plex works with my GPU (Ellesmere Radeon Pro WX 5100) for hardware acceleration was going askew.
Thanks for educating me on some new tools! My nieces also thank you. Bluey should be blue, not green.
I usually keep the torrent file around for this purpose. It might be a bit cumbersome to find the source for 20 TB of media but it might help you the next time. Generally if you don't store the hash then you can't check if the file is good.
I actually do have the torrent, files for a lot of them, but I've moved folders and I'm not really clear how that might affect things.
By my guess it's probably about 10% of the library that's corrupted, so re-downloading them wouldn't be the worst (I've already been doing it piece wise as they come up).
While I'm not great with system level and IT stuff, I'm OK with coding. I'm debating writing a python script to get the average color of each video file, I'd bet there's some libraries out there to help with that.
This is going to be very unhelpful in your circumstance, but a filesystem like ZFS can scrub the entire pool and find/repair any corrupted blocks. It will report how many bad blocks it found during the scan.