For home movies and videos, make sure the library is something like Photos, and turn off all the metadata. I think this is quite an annoying part of jellyfin. I think a good project would be to add support for simply showing a simple library with no metadata or logic. Currently even for home movies, jellyfin tries to logic some information from the folders and file names.
I also had issues with memory usage. My server is janky and so it only has 16 gigs of ram and with all the other stuff running on it, jellyfin ended up being really slow and my server itself started slowing down.
I turned off jellyfin but I'm a big fan of it overall!