JSONFormatElements Performance

[anon45965781]

Take a look at the short video above and see how your technique compares. I'm not trying to change your mind about your approach or to critique it in any way. Really just curious about performance. (I personally keep all JSON functionality out of FileMaker.)

You can get the JSON data for testing, here:
https://api.covidtracking.com/v1/states/daily.json

The JSON data is over 642,000 lines when formatted. Using the micro-service, it takes less than one second to parse and return to FMP (including the POST to the service itself). The micro-service method I wrote also handles JSONObjects and JSONArrays equally well.

Interesting discussion!!!

[jormond]

@AndyHibbs

Josh’s example seems OK, but we’ve experimented with other variations and it is possible to receive false positive or negative results depending on how the check is being made.

If you can come up with some, I'd like to see examples of false positives here. My custom function, which I believe I pulled from Karbon ( Todd Geist and Dave Graham's functions ), I've never seen give me an incorrect result, and I have thousands of unit tests for it. If it is valid JSON, the answer is usually null/empty, if it is not valid JSON, you get the error message back that starts with "? *".

Everything that I've tested, I've also validated against jsonlint.com. The result has been the same in every tested case.

Like @Bobino mentioned, this is just to see if the JSON is valid, nothing more. From there, if I need to know, I can see what is wrong, the error tells me where the JSON is broken.

[jwilling]

Regarding JSON validation, my favorite approach is simply to confirm the JSON string is a valid Object or Array explicitly with JSONFormatElements:

Object:
Left ( JSONFormatElements ( $json ) ; 1 ) = "{"

Array:
Left ( JSONFormatElements ( $json ) ; 1 ) = "["

The benefit of this approach, vs looking for "?" is that it validates the JSON and the type simultaneously. Since I tend to pass script params as objects, I use the first one a lot.

[BenMiller]

This very interesting - yes! Can you post more details. Also, with the new FMP19 JS functions, does it make sense to do something like within FileMaker this using a Web Viewer to run a kind of "micro service" using JS?

[anon45965781]

That does seem to work, but you only get a 0 or 1 as a return. Plus, you have to include both tests in code. Of course, yes, having both tests is needed since you don't know what you'll get (object or array) ahead of time (I had to do the same thing in the micro-service using local error handling).

In the screenshot below, I removed a ":" from the JSON on purpose to see what would happen.

Your CF correctly returned "0".

However, I still prefer getting the information about what is actually wrong using a JSON API designed for it:

---

The actual Java code that does this attempts to create a JSONObject from the passed JSON. If that fails, the Java code then tries to create a JSONArray from the passed JSON. If that fails, you get an actual message from the API (see above example) showing what's wrong.

---

I'm guessing MBS also gives detailed JSON errors, but I haven't tried it.

[anon45965781]

You'd have to test that, but a micro-service is external by definition. By being external, a micro-service (REST Service, Web Service, whatever you want to call it) is not dependent on any program.

I avoid JSON at all costs in FMP due to performance issues, FMP's JSON implementation which has problems with certain characters, no Boolean support, and other issues as reported over and over on the forums.

Try this JSON data for testing, here:
https://api.covidtracking.com/v1/states/daily.json

See what kind of performance you get in FMP.

Using an external micro-service, it validated this 642,000+ JSON array in less than one second.

See how FileMaker does and post back.

[anon45965781]

How does your CF work with the JSON Array I posted above?

Here it is again: https://api.covidtracking.com/v1/states/daily.json

Does your CF complete?

If so, how long did your CF take (seconds)?

Interested to hearing objective benchmark results so we can compare.

Thanks J.

[AndyHibbs]

Hi Josh

I wasn’t doubting your approach, I said it looks OK, but as Josh Willing pointed out, there are various methods for checking the error and my point is that it would be very easy to put in a check, which would appear to work in principal, but due to the nature of JSON, could result in a false negative or positive.

Kind regards
Andy

[anon45965781]

True dat.

[AndyHibbs]

@anon45965781

Interested to hearing objective benchmark results so we can compare

I think this does sum up that everyone has their own approach and there is no right or wrong. I believe you earn income from writing micro services, hence this is your usual response (understandably). Most of our income comes from streaming FileMaker to any computer device, which is centrally managed. It would be very easy for us to recommend running FileMaker this way rather than installing on a computer device and connecting to a cloud based FileMaker Server. However, we respect all approaches and have clients that do both.

The majority of people here, with some very knowledgeable exceptions, specialise in FileMaker and, if they can keep within their own comfort zone, then will use FileMaker whenever possible. This is also why Claris Connect has been released.

We’ve just embarked on our first FMP 19/Javascript project, which is taking us out of our own comfort zone and expanding our use of JSON beyond where we have before.

This post is resulting in some very, very useful information, so I encourage everyone to continue with their input to continue this process and respect each person’s approach.

Thanks everyone

Andy

[anon45965781]

Thanks Andy.

Actually, the majority of what I do currently is Machine Learning in Python. I'm currently doing a dimensionality reduction (Principal Component Analysis) of a 10,000+ feature data set. Lots of math as you can imagine.

Having been a Java developer for a long time, I started doing micro-services for FileMaker specifically when I realized it couldn't do regular expressions and lots of things I needed years back. It's all kind of taken off since then.

My postings are often enthusiastic, but I hope nobody confuses that with anything other than my love of technology, coding, and sharing!

[ajmchambers]

Hey all, so some quick testing out of the native fm approaches.

My initial approach using JSONFormatElements looked like this:

Let ([
	~rawParameter = Get ( ScriptParameter );
	~isJSON = not Exact ( Left ( JSONFormatElements ( ~rawParameter ) ; 1 ) ; "?" )
];
	~isJSON
)

Adjusted approach using JSONGetElement seems to be a little bit faster:

Let ([
	~rawParameter = Get ( ScriptParameter );
	~isJSON = not Exact ( Left ( JSONGetElement ( ~rawParameter ; "" ) ; 1 ) ; "?" )
];
	~isJSON
)

You can modify it to check explicitly for objects:

~isObject = Exact ( Left ( JSONGetElement ( ~rawParameter ; "" ) ; 1 ) ; "{" )

or arrays:

~isArray = Exact ( Left ( JSONGetElement ( ~rawParameter ; "" ) ; 1 ) ; "[" )

I made a quick test file: JSONTest.fmp12 (224 KB) .

Comparing the two using the previously mentioned daily covid-19 json I get:

• JSONFormatElements (1285ms)
• JSONGetElement (1102ms)

Though the results can vary a bit (possibly due to caching or something) making the difference larger.

[anon45965781]

Awesome!

[jormond]

No worries. I am more concerned with finding any potential fails than anything else. I understood what you were saying and was just hoping you had some solid examples of where that gives you a false positive. I don’t see how it can.

[jwilling]

I still prefer getting the information about what is actually wrong using a JSON API designed for it:

My approach can just as easily do that too. Simply break the test up into two steps, one to format, one to see if the leftmost character is as expected:

You'll notice the data viewer tells me exactly where to look and what for, in this case.

[BenMiller]

@anon45965781 - great, thank you.

[jwilling]

using a JSON API designed for it:

FWIW, FM uses RapidJSON and JsonCpp, both open source libraries "designed for it", under the hood, as suggested by their acknowledgments page.

[anon45965781]

Agreed ... That's good for small JSON's using an interactive approach.

Try pasting a 640,000 line JSON document into the data viewer.

Does that still work OK?

(I haven't tried it.)

[jwilling]

Interactive? No, reading the left character is how your script checks validity, then, and only then, does your script log an error (to wherever you want), which is now contained in the $json variable. You never have to look at the data viewer at all.

[anon45965781]

Here's a JS example showing how to handle periods in object names. I wonder if this will work using FMP's native JSON functions (haven't tried it). This period in object name issue comes up a lot on the FMP forums.

Java (not JavaScript) micro-service handles periods in object names automatically, but this workaround for JavaScript is not bad at all.