FMDataMigration Tool

Resources
1

It is slowly dawning on me how much we reply on the FMDataMigration tool nowadays, which has now joined 2empowerFM Developer Assistant and Perception as our ‘can’t do without’ utilities.

Although we use both the Mac and Windows versions, it is used on Windows most frequently, and even though most of our solutions are separated, it has significantly improved productivity.

Mostly we keep the syntax needed as text within secure storage (stored locally, not on any server) with predefined paths to our working folder.

Our standard procedure is:

Create the clones on our development server using a dedicated backup in FMS Admin Console

Transfer to the live server and store in a ‘Clones’ folder then remove the ‘clone’ suffix(es)

Stop FMS and copy the live files to a ‘Data’ or ‘Originals’ folder

Run the command line as administrator and navigate to the FMDataMigration folder within this

Copy and paste the syntax for each file, which creates the new files in a ‘New’ folder (there are tools out there to remove the need to learn or setup the syntax)

Copy the new files to the FMS data folder (no permissions issues in Windows, or set permissions if a Mac)

Archive the original files

Delete the clones

Start FMS

The tool reduces the need to attempt to keep the field creation in sync between the live and dev versions and can allow full updates to be completed in minutes. For the introduction of a new system, we may be doing a number of these each week.

Thank you Claris for this wonderful utility and, if any developer out there hasn’t used it, I strongly suggest you try it. This has massively increased the value of a FileMaker Developer Subscription.

Regards

Andy

6 Likes
2

Andy, how would you compare the FMDMT to 360WorksDeploy? We use it. My colleague was involved in the setup. What I see is all the code from our DEV server magically up onto our PROD server each Saturday morning on an unattended schedule. Totally agree about FMPerception and DevAssistant.

1 Like
3

Hi Stephen

I'm afraid I have no experience of 360Works Deploy. My only knowledge is that it is an automation that uses the FMDataMigration tool. 360Works supply excellent products with good support, so I can only assume it will work extremely well, which is what you've indicated.

We support roughly 50 client systems both on premises and within our own streamed cloud infrastructure, hence each update has to be timed and controlled very carefully. Versioning is key to our operations.

I'm really glad Deploy works so well for you.

All the best and kindest regards

Andy

1 Like
4

Same here, I couldn't do without the FMDMT anymore. All of my customers run their servers on Windows, but the Migrations I do mostly on my Mac.

Transfering the files to, from and back from my Mac is only a matter of a minute (even with 10Gb files) because the Migration runs eassier on my Mac.

I use just one folder containing 3 folders (Clones/New/Old) and a migrationscript:

##########################################################################################
#
# This script helps to easily migrate data using FileMaker's FMDM tool.
#
# Place this script in a directory with 3 subdirectories:
# Old (this will contain the old files from which the data must be imported)
# New (this dir must be left empty, the script will place the migrated files here)
# Clones (this will contain the clones, they must have exactly the same names as those
# in "old", but then supplemented with " clone")
#
# The script assumes that all files have the extension ".fmp12". You have to
# adjust script accordingly.
#
# This script applies 4 options by default:
# -force
# -ignore_fonts
# -ignore_valuelists
# -v
# So any existing files in "New" will be overwritten without warning.
# Nothing is done with any loaded fonts
# Valuelists are not transferred
# The logging is "verbose", so extended logging is saved in the file Conversionlog.txt
# written off.
# All accounts will be transferred
# Known tables will not rebuild indexes and no
# calculations re-evaluated. This does not apply to new tables and new fields.
#
# If you want to change this behaviour, you can:
# https://help.claris.com/en/data-migration-tool-guide/content/migrate-data.html
# read which options you can adjust.
# 
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program. If not, see https://www.gnu.org/licenses/
# 
# © 2022 - https://www.filemakertips.nl - Menno van Beek
# 
########################################################################################## 

#!/bin/bash

echo -p "Enter the username:" 
read accnt
echo -p "Enter the password" 
read -s pswrd
SRCACC="$accnt"
SRCPWD="$pswrd"
echo -p "Use the same account for the clones (y/n)?" 
read answer
case ${answer:0:1} in 
	y|Y|j|J|1 )
		CLNACC="$accnt"
		CLNPWD="$pswrd"
	;;
	n|N|0 )
		echo -p "Enter the username for the clones:" 
		read accntc
		echo -p "Enter the password for the clones" 
		read -s pswrdc
		CLNACC="$accntc"
		CLNPWD="$pswrdc"
	;;
	* )
	exit
	;;
esac
D0=Old
D1=Clones
D2=New
logfile=Conversionlog.txt
SOURCE="${BASH_SOURCE[0]}"
DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null && pwd )"
cd $DIR
OLDIFS=$IFS
IFS=$'\n' 
cd $D0
theList=($(find . -type f -name "*.fmp12" | sed 's|^./||'))
cd ..
IFS=$OLDIFS
echo $PWD > $logfile
tLen=${#theList[@]}
for (( i=0; i<${tLen}; i++ ));
do
	y="${theList[$i]}"
	theFile=${y%.fmp12}
	FMDataMigration -src_path "$D0"/"$theFile".fmp12 -clone_path "$D1"/"$theFile"\ Clone.fmp12 -target_path "$D2"/"$theFile".fmp12 -src_account $SRCACC -src_pwd $SRCPWD -clone_account $CLNACC -clone_pwd $CLNPWD -force -ignore_fonts -ignore_valuelists -v  >> $logfile
done 
exit

And run the migration from the commandline:
sh ./Migration.sh
The migrated/new files can then be copied from the "New" folder and the conversion can be checked in the "Conversion.log" file.

4 Likes
5

Hi Andy,

we only use the FMDMT in rare cases, where the structure of source and target files are guaranteed to be identical.

We have an updater tool that we have developed alongside our product, which works using the old Import Records by matching field names, ... and it has logic in it to deal with various restructurings that have occurred over the years (...er...decades) of development, so some tables have moved, indeed some complete files have been integrated elsewhere and dropped.

That's because the solution started back in FM5 days where there was a limit on 50 tables/files, so various tricks were used back then to cram more data into one table...which at some point has been normalised, one table splitting into two or three or more tables, ...etc.

Above all this has allowed us to update customers from VERY old versions of our software. In other words KEEP THEM! :smiley:

The FMDMT is fantastic, I would be interested, however, to know how you + other FM-devs support :

  • renaming fields
  • initialising of new fields
  • correction of old data to fit new corrected logic
  • and larger changes to the DB structure?

Do you have Pre- or Post- Update scripts?

...and if anybody has a solution to the following problems, I would be VERY grateful for any ideas and tips!

  • how I can rename hundreds of fields with a dot in their name (so that we can use the Data-API in the future)
  • whilst not breaking the update process
    • in other words after renaming the fields, how can I continue to update customers from old field names to new field names

Thanks

MrWatson

6

The DMT is awesome. It's the only way to copy accounts from one file to another without having to reset user passwords each time, for instance. But it has some major drawbacks and gotchas that you have to be aware of, which makes it a not-so-perfect update tool in all circumstances.

I'll try to comment more tonight, but specifically regarding "renamed fields," the DMT automatically takes care of this using FM's internal IDs. It determines the best match in this order:

  1. both names and IDs match
  2. only names match
  3. only IDs match

If no match is found, then the data is not migrated. So even if you rename a field, the internal ID would be the same, so it would transfer (unless you intentionally created a different field and gave it that original name).

1 Like
7

Pre-DMT, I used the 100% manually created, scripted import. It was flawless, but it took so much time and effort to keep track of everything and maintain, especially as development moved into a multi-developer environment. The old-school scripted method was so straight-forward, though, that it was easy enough for clients to do themselves.

I was ecstatic about the DMT when it was first released and looked forward to abandoning those old scripts, but I quickly ran into its limitations and realized it wasn't possible to ditch them completely.

We obviously try to avoid any major structural changes, but it's simply not always possible. Likewise, even while maintaining the exact same structure, we always need to update certain globals. As such, yes, I now utilize a pre-update & prep script, DMT transfer script, and then a post-migration globals update & clean-up script. Unfortunately, I have not yet been able to build an idiot-proof version of this and thus don't trust clients to do it themselves :slightly_frowning_face:

I've been tempted to revert back to the all imports method, but the transfer of data from one file to another using DMT is so dramatically faster (AKA instantaneous), and the transfer of user accounts isn't otherwise possible, so it's still worth including in the process, at least for now.

Of all the great things DMT can do, the wishlist of features still includes:

  • ability to work on non-clone files and/or have something like an -ignore_table parameter
  • ability to ignore all data and only transfer accounts
  • ability to set "semi-verbose" mode for the fmmigration PS (which is forced quiet) and should not see everything in verbose
  • ability to ignore local file settings
  • ability to migrate from an unencrypted file into an encrypted file
  • < insert_idea_here >
1 Like
8

I've been using the Lesterius FM wrapper for DMT for some time now, and it is very click stuff. Makes the management of DMT much easier.

1 Like