Note: This is not a tutorial, it’s an explanation of testing and research. Future posts will follow with tutorials and guides of each component.
During the past month or so we have been experimenting with building a Plex Media Server using cloud based personal storage and FUSE to mount the the object storage on a low cost virtual machine, with the goal of building a low cost but extremely high storage media centre.
The first thing to search for was cloud storage which would be suitable for a media centre, bearing in mind that this is not what personal cloud storage was designed for. The main requirements are:
- Throughput which would exceed media bit-rate.
- Low cost.
- A FUSE filesystem which could be used.
If you’re not familiar with FUSE, FUSE is a tool for developing filesystems easily. The goal isn’t to create amazingly efficient or scalable filesystems such as EXT4 or ZFS, but to be able to create a filesystem easily in a language such as Python which can be made by most developers and to be able to mount them as a non-root user.
Cloud Storage Providers
Prices are being driven down by heavy competition in the person cloud storage space. The cloud storage provider we’re all familiar with is of course Dropbox. Although Dropbox could probably meet the technical requirements of a media file system, there’s simply much better value providers around with less polished software (which is fine as we will FUSE mount).
Recently OVH (A well known French service provider) dropped the price of their personal cloud storage service, hubiC. hubiC now only charges €5.00 per month for 10 TB of storage which is absolutely fantastically priced, better yet they give you an extra 500 GB for each referral.
But there’s a catch. hubiC is limited to 10 Mbps. It would take you 92 days to download or upload the full 10 TB capacity, which is completely ridiculous. It in no way would be sufficient for a reliable media server.
Not only that, but the only FUSE filesystem we could come across for hubiC is completely unsuitable for purpose.
If you have Google Drive with a Google Apps account, it’s likely that you have a vast amount of storage. A lot of companies and educational institutions assign their users Google Apps account so this could save you a lot of money.
The issue with Google Drive is that despite no clear definition on throughput, in our tests we were unable to upload or download at a good rate. Tests averaged at about 19 Mbps uploading to Google Drive and the download average far below even 1 Mbps, no where near enough.
That said, a lot of people use Google Drive on FUSE, and the Google Drive filesystems out there are rather mature and functional. It may also be entirely possible to get a better throughput from different server providers, the ones we tested were not showing hope though.
Amazon Cloud Drive
Leave the best for last, right? Amazon Cloud Drive meets all the requirements we would need for a Plex Media Server’s media storage.
Amazon recently announced that the cost of unlimited storage is only $59.99 per year. A fantastic price that matches (or beats, depending if you think you can exceed 10 TB) the price of hubiC, without the throughput limitation which hubiC enforces.
There are a couple of issues with Amazon Cloud Drive however which should maybe be addressed:
- Amazon do not define unlimited, it’s much nicer to have a clear line like hubiC. Even if that limit was 5 TB, it is still great value.
- Amazon state that ‘Unlimited Everything’ is only usable in the US, which suggest is may be wise to use a proxy if you or your media centre are located elsewhere. Without a proxy you could also end up with a suspended account, however we are unsure how well Amazon enforce this term.
The last question that comes up when trying to match the requirements is regarding the throughput of Amazon Cloud Drive. During the testing there was mixed results, however when testing from the US the throughput was consistently impressive peaking at 600 Mbps reading and writing (Typically around 200 Mbps). This is plenty to concurrently stream plenty (providing the Media Server’s network connection is sufficient).
Needless to say Amazon Cloud Drive was the best of a bad selection, and without any obvious flaws.
The FUSE Filesystem
Proceeding with Amazon Cloud Drive, the main thing to look into was the FUSE filesystem that could communicate with Amazon. Firstly a quick Google came up with
acd_fuse. It became quickly obvious that
acd_fuse was very poor, it frequently encountered issues and communicated with Amazon by being a wrapper around the web interface, something which doesn’t make much sense when there is an Amazon Cloud Drive API.
After a bit more searching for an alternative, we found
acd_cli, which is absolutely fantastic.
acd_cli communicates with Amazon using their API, it took a while to come across this project because the name was a little deceiving. Regardless of the name,
acd_cli does provide a read-only FUSE filesystem and resolving a number of bugs works extremely well.
At this point the next struggle is writing to Amazon Cloud Drive, but that’s not too much of an issue as you can do it with the command line interface which is the whole point of the project in the first place.
acd_cli had a few issues which made it unstable for purpose however we have worked these out and the filesystem now works great. Scans are fast, file reads are quick and media can be streamed with no real issue.
A major problem with storing media on personal cloud storage is ensuring that no one can ever read your files. This can be an issue for a number of reasons and it’s best to simply encrypt your files before uploading them.
Since we are using FUSE for the Amazon Cloud Drive mount, it makes sense to simply layer another filesystem on top to handle encryption and decryption of the media that you do not want others to be able to read.
There’s a FUSE filesystem called
encfs is quite simple, it mounts a human readable directory to another directory where the encrypted files are stored, ultimately creating a proxy style directory of your files. Files you read/write to the
encfs mountpoint are decrypted, and then will on-the-fly be encrypted to another directory you choose, for example:
/home/plex/.acd-sorted/could be an
acd_climountpoint which resembles the files on your Amazon Cloud Drive.
/home/plex/acd-sorted/could be an
encfsmountpoint which reads encrypted files from
Providing you do not upload your keys to Amazon Cloud Drive, there’s no possible way that anyone (including Amazon) could read your data or have any indication of what it is.
encfs together can still be used to stream the media stored on Amazon Cloud Drive still with almost zero performance impact.
In most cases the most reads of media you will have are to new episodes you have on your setup. It makes sense to keep these locally as they will not use much space and then to remove them when they become older (and ergo less read). This is where another FUSE filesystem comes in.
UnionFS-FUSE is a filesystem that “unites” directories (or in our case, multiple filesystems). What we want
UnionFS-FUSE to do is display all the media that is stored locally and on Amazon Cloud Drive, and to read from the local media where possible - when we write, we also want to write locally as Amazon Cloud Drive does not support FUSE writes.
Every night it makes sense to upload
.local-sorted/, and keep your Amazon Cloud Drive up to date and after some time delete the files locally. The UnionFS mount will remain readable for your locally deleted media as it persists on the other mount (
encfs over Amazon Cloud Drive).
Things to keep in mind:
- Plex will not be notified that the filesystem has changed, so it’s auto scan on change will not work.
- The directory sturcture of the two directories you unite with
UnionFS-FUSEshould be the same.
Both of these issues can be addressed by FileBot. When a new file is added, it’s possible to make your download manager trigger a FileBot scan to move your media into
sorted/ (which will ultimately write to
.local-sorted/) and once that has completed, FileBot can tell Plex to issue a scan. The directory stucture will also remain pretty consistent.
This setup may seem like quite a lot of “string and glue” however ultimately the result is fantastic. Reading from Amazon Cloud Drive is quick,
encfs doesn’t hit performance hard, and
UnionFS-FUSE makes it possible to keep as much media local as you need. You can of course tweak what you store locally to work better for you.
It’s important to keep in mind that Amazon Cloud Drive is “personal storage”. You should not try and go around sharing a Plex library with 100 people using Amazon backed storage, that breaks their TOC and we can almost guarantee will end up in you losing all your data due to service suspension.
My testing has lasted 2 months, a number of
acd_cli bugs have been addressed during that time and we are comfortable enough to now remove my local storage for this option.
If you are not sold on this, Amazon Cloud Drive has a 3 month trial and you can obtain OpenVZ VMs which support FUSE for next to no money at all - try it out.