Converting to External Container Storage

Questions
1

Hello friends

Can anyone point me to a white paper, blog post, thread, etc. that describes the benefits of switching to external container storage?

Also, is there any way of doing the conversion purely server-side? My client has just shy of 100K records, approximately 70% have 2 - 16 images embedded. For a file size approaching 60GB.

Thanks!

2

shows how you can automate it

As to the benefits:

  1. Remote Containers are newer than embedded as a feature and it is more efficient at retrieving the container content; it's faster.
  2. it has tremendous benefit for keeping the FM file size down and making backups and overall disk usage A LOT more efficient. That's because of the hard-linking it does under the hood. If you are not familiar with that, ask away.
9 Likes
3

Hi Wim, thank you so much!

Much as I like your process, it is not quite what I need. I'd like to convert the containers to external storage, without monopolizing the system, maxing out the network, or locking up my own computer.

I see a 2 alternatives:

  1. Bite the bullet and just do it.
  2. Create new, externally stored containers then move the images from one to the other. (This could be done on the server asynchronously/over night.)

To be honest, I'm leaning toward option 2 if only because it means that I can re-vamp the structure (i.e. instead of having 16 container fields on each record, create a related table of records).

I'd value any all feedback or wisdom you (or anyone else) would like to share!

4

instead of having 16 container fields on each record, create a related table of records

One bit of wisdom that I've encountered as it relates to this is to dynamically set the external container folder path by date, so that the external container folder doesn't get too full of loose container files, which makes it slow to view and browse on the server machine's file explorer. Rather, it will produce a folder hierarchy like year/month/day.

Something like this:

Screen Shot 2020-07-22 at 2.31.41 PM
Screen Shot 2020-07-22 at 2.31.41 PM1394×631 161 KB

2 Likes
5

Appreciate the tip! Definitely will take advantage of it; It doesn't seem necessary to keep it all secured.

6

instead of having 16 container fields on each record, create a related table of records

Do that.

@jwilling's suggestion to pigeon hole the content by date is good, though I haven't had a situation where it was necessary to go down to day. Year/month has been sufficient.

The trick for reliable transfer has always been to give the server time to do its own thing. If you allow the script to run unimpeded it can choke your resources and you'll get copy failures. You want to ensure that all the file i/o succeeds (and there can be a lot of system activity associated with all this).

The last time I had to do this we set up a workstation to run the process and left it alone. We had created a table for logging the process and installed a script timer to provide a pause between each run of the script. There were tens of thousands of records, several GB of data. It took forever but it worked.

2 Likes
7

The first part of my report into researching how best to use container fields for image storage on Medium is here

2 Likes
8

A great read, thank you. Interesting to see the advantages of FileMaker being explained for an audience that may not be familiar with it.

1 Like