FMPA memory uptake and file size ballooning due to missing explicite type conversion in script

[jormond]

Well, we have 1 of 2 options.

1. Report and work with them to resolve these issues.
2. Suffer with them.

It doesn’t even matter how they respond, or how we respond. It’s important, in my mind, that even when the communication is challenging, that we keep trying to work it out. I know, for me, sometimes a user can’t effectively describe an issue in a way that I understand the larger problem. It’s often later that I better understand that particular issue and thus can fix it.

This may be the case with this issue. It may not. Either way, I’ll keep trying to push it forward.

[Torsten]

Follow the boss, cheers!

[Torsten]

Thank you Joshua. If you feel motivated, go ahead. I will certainly not use Claris’ back-channel.

I’d like to add: yes, sometimes a user can’t accurately describe the issue. But if the user offers evidence, I will certainly accept it and examine it. Never would I dare telling a user ‘Not thanks, keep your stuff and get along with it’.

[jormond]

I’ve told users that. If it’s something that 999 times out of a 1000, it was user error, I would totally tell them that. For your post, you had more detailed evidence of what was happening. I get it.

I would like to see if it is indeed expected behavior or a misunderstanding of what the problem you are describing is.

[Torsten]

When an issue is reported, I start with collecting evidence, not by refusing it. I start with searching the root cause. If it is a user error, I will inform the user and explain to him how he could do it better. Starting with the assumption that it is a user error is genuinely wrong because it means that the investigator is biased. Not helpful.

[jormond]

With some users I 100% agree. With other users I 100% disagree. LOL

Really, though, I do very much filter the requests. It got so bad, we basically had to cut off access from our users to our development team. 80% of the calls were solved by restarting FileMaker, or something simple like looking at the actual button they are clicking.

If we answered every call that came in, and spent time investigating every report, we would get nothing else done. I tend to lean on the side of being helpful, but even I have a limit. We read every report, but there are some reports that simply get placed on hold. If multiple of the same report comes in from the users, we go straight to the department to investigate what’s actually happening. In comes the other problem with some user reports… the submitted report is not accurate to the actual problem they are having.

[Torsten]

Joshua, I am a developer (and Claris’ customer - and I bring in customers through my work). But yes, my request has been handled the way you described. Well observed!

[jormond]

My apologizes, don’t read my comment as acceptance of how it was handled.

The reality for me is this:

1. Do I want it fixed?
2. Did they fix it?
3. If not, what can I do to prompt them to fix it?

For this case, ‘Yes’ ‘No’ ‘Keep describing the problem until someone either gets it, or explains sufficiently that it is expected and normal’. Now in this case, there isn’t much you can say to me that would accept that as normal behavior. The work around is easy-ish, but should be unnecessary. Thus, the need to keep pushing the issue.

This use-case doesn’t warrant “riot in the streets”. But it does need some “hey, can we go out for a drink, I want to talk about an issue I can’t seem to explain on the forums”. If the end result is an issue getting fixed, it’s a win-win.

[Torsten]

No problem, if someone wants to get in touch with me, that‘s easy. He/she can also suggest having a drink together and talk.

[Torsten]

It would be worth to check if the while function has an impact here.

[jormond]

That was a thought I had also. The file size is the weird part. I’ll spend some time this week and this weekend peeling it back, and then send in to get some eyes on it.

[AndyHibbs]

I was demonstrating this to our team here this morning so that they are aware of it. Out of interest, we saved a compacted copy of the ‘ballooned’ file after the non-explicit script was run. In the screen image below, the bottom file was after running (and subsequently quitting FMPA18) the non-explicit script, the middle file after running the explicit script and the top is a compacted copy of the bottom file.

[Torsten]

That you Andy. The fact that an explicit conversion solves the problem trigged some more thinking in my head. In a copy of the non-explicit script, I added a filter ( value ; “0123456789”) function, just to see if something sneaked into the stack list that should not be there. Using filter() yields the same result as GetAsNumber ().
What I found: in $ID_Stack, sometimes a space (char(32) ) is added, sometimes not (see screenshot). This brings us to the custom function with a while loop.
What I don’t get is how the space characters got there. The values in $ID_Stack all originate from the ID field which is validated as a numeric only, value auto-generated.
Did I insert some silly mistake into the code somewhere? Did I overlook something?

[Torsten]

Caught it!
It is a typo in the custom function (I’ll eat my hat for that one). There is a space before the pilcrow (see red pointer). Once the space is deleted, the script without conversion yields the expected result.
Conclusion: the cause is a typo that introduces a space character in a VL line. Copying that value to a number field breaks the file. In debug mode, no warning or error message is shown that FM expects a validated number value.

Thank you to everyone who contributed to the hunt!

[Malcolm]

That is a really dramatic example of how things can go wrong. I’m impressed by your troubleshooting efforts @Torsten, well done.

Two things that stand out for me in your custom function which might help others with less experience

1. There is no need to put the pilcrow into quotes.
2. The List() function could have been used instead of concatenating strings

[Torsten]

Thanks @Malcolm, I will modify the cf accordingly. Will be cleaner and easier to read.

[steve_ssh]

+1 to catching the added space being a great catch/good eye.

When I saw that post, one of the first concerns that I had was:

How many times is the custom function being called, and how many total extra chars are being added to the data being set in the fields?

I did my best to grok the script to get a sense of this, but I could not, so I decided to run a test, using a variation of the script.

The script variation was as follows:

1. Define some extra variables in the script to capture some metrics:

2. In each place where the script would have set a field (regardless of by name or not), comment out the setting of the field action, and instead, add a line that captures some metrics about the size of the data that would have been set in the field:

This is what the calculation looked like that was used to capture stats:

3. Periodically display stats so that I could observe any obvious trends:

Having set up the above, I grabbed the following stats upon running the script:

1. After processing 10K records:

2. After processing 20K records:

3. After processing 30K records:

4. The final batch of stats:

Comments:

The first thing I’d like to say is that it would be great if someone could check my work for sanity.

My FMP skills are rather rusty these days, and I don’t doubt that I could have slipped up somewhere in my methodology. Additional sets of eyes and brain cells would be good to have before we accept what I’ve posted above.

If the methodology and stats do seem accurate, then I think an important aspect of this thread that I did not see mentioned, is that there is a rather non-trivial amount of extra data that the original script tries to set into the records (versus a script which first scrubs that data by use of GetAsNumber, or anything else that will remove the added space chars).

I’m deliberately staying away from passing judgement or making comments about what the behavior should be when one tries to store long string values into a FMP number field. My intent is simply to bring awareness to what I am observing, which is that this does not appear to be a matter of a small number extra space characters causing the slowdown. Barring an error on how I set up the metrics (and again, I hope someone will check my work), it appears as though a very non-trivial number of unwanted characters are being handled by the script.

I hope that this helps in some way – either in better understanding of the nature of the symptoms observed, or any other way.

Kind regards & HTH,

-steve

I’ll attach a zip of the file as I modified it.

FMPA Type Conversion Isssue_MOD.fmp12.zip (525.2 KB)

[Torsten]

@steve_ssh, Thank you for providing additional insight and a good method for debugging loop routines. Your analysis is accurate. The faulty routine built up more spaces in the stack entries each time the custom function was called. The single typo has a considerable effect. It always humbles me when I learn how easy things can be overlooked in development. Type casting is a safeguard we can apply but does not replace fault-free routines.

[FileKraft]

great catch @steve_ssh !! thanks for your analysis!

conclusion - not a bug!

[Torsten]

@steve_ssh’s observation is to the point. My custom function had a major glitch.

Even when not qualified as a bug, the fact that a number field with validation ‘numeric only’ on, ‘allow user override’ off and ‘validate always’ on accepts non-numeric input should be investigated.