Performance core principles: 10. Scripting

[FileKraft]

I think it would make sense always to track platform/OS as well as version. Many things are changing quickly on each rev. Thanks !

[bdbd]

@nicklightbody

You could well be correct but the fact that it appears to be several times faster suggests to me that it is checking for “no more records” at a more direct level than the calc engine?

I am making an educated guess here… but I don't think there is a verification on every loop iteration. GoToRecord produces an error code during every iteration, calculation or not. FileMaker processes additional code when non-zero errors come up, again calculation or not. I suspect the exit loop condition is evaluated only when there is a non-zero error. In a 100 record set that starts at record 1, this approach reduces the need to check the exit condition 99 times.

[Bobino]

Thanks @bdbd for rewording what I was trying to say. @nicklightbody, @bdbd and I are just guessing, this would need to be tested, but the pattern we allude to suggests the loop repeats itself without checking for anything, with an "onError" trigger that would exit the loop if error is 100.

I think this can be mimicked to some degree in how a loop is scripted by a developer and testing how that specific variation performs against the ones already mentioned would give more clues about what happens "under the hood".

[jwilling]

I wanted to expand on @mrwatson-de 's fantastic summary. I'll preface by saying these are just my observations (well actually jointly reached with my friend Dan Smith, if he'll allow me to throw him under the bus too).

@mrwatson-de

As far as I know the internal JSON cacheing has been present from the start

My experiments suggested caching arrived in v18 (at least with the current behavior).

@mrwatson-de

• FileMaker keeps a single copy of the internal objects

That's right. Anytime you call any JSON* function, FM compares the input string to the previous input. If they match exactly (no extra whitespace etc), FM will reuse the cached object. If not, FM overwrites the cache with the new object (this is what you want to avoid doing too much). One nuance here is if you use JSONSetElement, the output becomes the new input. So in the following example, $json is modified, and the cache is now based on the modified $json string.

// $json contains the new input string
Set Variable [ $json ; JSONSetElement ( $json ; "height": "6ft" ; JSONString ) ]

@mrwatson-de

• build smaller JSON trees before adding them to a larger tree, rather than incrementally adding to a huge tree

Yep. This part is fun. Since the huge JSON object is cached, shouldn't it be fast to keep appending it? Not quite: the fact remains, every invocation of a JSON* function requires FM to read the whole input string just to decide if it can use the cached object (or maybe it reads only up to the first difference). So in this case, it's not the object reading/modifying that's slow, it's the input string comparison.

@mrwatson-de

Prefer a single JSONSetElement call with multiple parameters over multiple single JSONSetElements calls

While true, one of the nice implications of the current caching behavior is that as long as you only work with one JSON object at a time, it can be quite performant to "build up" the JSON in multiple steps because it's not parsing the object each time. E.g. this is pretty chill:

Set Variable [ $json ; JSONSetElement ( "{}" ; "first" ; "Josh" ; JSONString ) ]
Set Variable [ $json ; JSONSetElement ( $json ; "middle" ; "Willing" ; JSONString ) ]
Set Variable [ $json ; JSONSetElement ( $json ; "last" ; "Halpern" ; JSONString ) ]

^ The first step above triggers a parse/cache. The second two steps merely check if $json is the same input (it is, despite the new properties), and reuse the cache.

Personally I'd favor separate JSONSetElement steps in case of optional properties. But be cognizant the input string length, and avoid doing this too much if the JSON was gigantic.

Bonus

JSONGetElement counts as a JSON* function. So if you read from a second object in between modifying a JSON object, FM will have to re-parse and cache the first object when it gets back to it:

Set Variable [ $name ; JSONGetElement ( $json ; "name" ) ]
// now $json is parsed
Set variable [ $country ; JSONGetElement ( $json ; "country" ) ]

Set variable [ $overwritesCache ; JSONGetElement ( $json2 ; "prop" ) ]
// now $json2 is cached instead (even though we only did "Get")

// $json will have to be re-parsed/cached
Set Variable [ $age ; JSONGetElement ( $json ; "age" ) ]

The cached object is scoped to the FM application, not scripts, calculations, windows, or files.

• an object cached in a parent script is available to subscripts (if fed the same input string)
• an object cached in a subscript is available to parents
• the cache outlives the script callstack! So if the input json is the same as the last script called 10 minutes ago it's already cached.
• the cache even survives closing and reopening files. It therefore appears to be FileMaker-scoped.
• the cache doesn't care about your input string source. If you have $json cached and you copy that variable to $json2, then JSONGetElement ( $json2 ; ... ) will use the cache. You can also read the value from a field with the same effect.

One sneaky gotcha here is if you are setting one json object using properties from another, this can be cause a lot of re-parsing if done in, say, a loop, because you keep overwriting the cache with each step:

Set Variable [ $value ; JSONGetElement ( $source ; "name" ) ]
Set Variable [ $target ; JSONSetElement ( $target ; "name" ; $value ; JSONString ) ]
Set Variable [ $value ; JSONGetElement ( $source ; "dob" ) ]
Set Variable [ $target ; JSONSetElement ( $target ; "dob" ; $value ; JSONString ) ]

Whew, hope you don't mind an impromptu essay!
J

[Malcolm]

Here's a simple test file that you can use to run tests and get results. GoToRecordTest.fmp12.zip (9.0 MB)