FMPA memory uptake and file size ballooning due to missing explicite type conversion in script

When looping in a script over a large data set (38’000 records), FMPA goes off the rails and leaks memory.

The concerned layout (table mode) shows a calculated field (result type text, store calculated result).
The script does not alter the two fields used in calculation. No recursions involved.

Script structure:
Loop

#Do some stuff

Go To Next Record (Exit after last)
End Loop

Expected behaviour:
FMPA slower than without the calculated field on layout.

Observed behaviour:
FMPA slows down to a crawl and eats up most of the available memory (see screenshot). After ending of the script, FMPA became unresponsive at attempts of scrolling in layout and ate more memory. Required termination of FMPA and restart.
Taking the calculated field off the layout or adding a 0.1 sec Pause Script step before ‘Go To Next Record’ solved the issue.

FMPA 18.0.3 on MacBook OS v10.14.6

3 Likes

did you add a FREEZE WINDOW script step before the loop?

No, I didn‘t.

if you loop in table mode it’s almost a requirement - you should see major speed improvements …

Yes, I expected a slowdown. What I did not expect is a memory management meltdown.

Just tested with ‘Freeze Window’ added. The memory leak does not occur. This hints at FM’s window drawing routine as the bug location.

4 Likes

I would still report it as a severe bug (if you haven’t yet), it should work and handle memory without issues at any setup.
Great catch!

1 Like

The last time I reported a severe bug, it took over 2 years to get that fixed - and there was no workaround and no information provided if it will be fixed at all. For this one there are several workarounds. No hurry to wasting time on typing a bug report.

1 Like

Would still reach out and report the bug - maybe times change with the new CEO and they improve QA.

Just curios what bug did you report?

The latest improvement under new management was the new forum software :grimacing:

This was my bug report:
Text cursor position delta

3 Likes

What a mess the forum change was and still is // it’s dead Jim!

Getting Bugs across is tedious and frustrating but it has to be done just for the sake of informing others to save time and confirm that there might be an issue with the product (and not with the developer).

// thanks for fighting for a better product and being upfront and honest.

1 Like

Hi Torsten

I know it is frustrating, but unless we all report back with these type of problems or feature requests, we’re never going to make progress.

The whole Window handling/memory issue does appear to be a mess, I’ve referred to this in other posts.

However, we’d all really appreciate it if you can take the time to report this through the official channels.

Many thanks

Andy

More interestingly, I observed the following today:

  • open file in FMPA, layout ‘Import’, table view
  • selected row is #1
  • click ‘Records’ slider in toolbar to rightmost position (select record #38145)
  • FMPA starts spinballing for minutes and balloons to over 10GB ram, keeping over 9 GB ram when finally spinballing stops and selected record is shown at the bottom of the screen
  • try scrolling up from there, FMPA starts spinballing for a while, whait for that to stop…
  • click ‘Records’ slider to leftmost position (select record #1), wait until spinballing stops and record #1 is shown on top of list. FMPA down to 1 GB ram
  • scroll down from there, works just fine

This cycle can be repeated at leasure. I ran the file through recovery, no errors found.
File has text and number fields, no calc fields. 38145 records, 1.9 GB.

@AndyHibbs, @FileKraft,
did the reporting: spinballing-and-memory-leak-when-moving-from-record-1-to-last-record-of-a-large-table

2 Likes

great thanks @Torsten good job!!

Curious if you see the same issue in form view? Clay has spoken often about looping in Form view vs list or table view. Not sure the impact with the bug you are describing, but curious.

We’re making more and more use of totally blank form layouts for script use nowadays.

2 Likes

Do the same for script runs. For development I need to see these tables and according to FM’s tech spec, this should not happen. Meanwhile, I emptied and re-imported data and re-run the subsequent script in order to exclude a foul index. Makes no difference.
Other observation:
with active record #1, ram usage is 0.9 GB, at record #20’000 ram use is 4.99 GB and at record 48’000 ram use is 8 GB. So this is fairly linear behaviour. When scrolling up from record 48’000 FM starts spinballing immediately, this becoming useless for viewing the table.

I also found this post about large data sets and index issues:¨Script control over field indexing.

I switched off all indexes except for the ID field and it does not change FMPA behaviour. Still spinballing and memory gobbling.

Have you been able to replicate that in a new file? Claris can find great use in that if you can get them a file that clearly shows the issue.