Parallel network transfers with MBS Plugin

Products, Plugins & FM hosting/cloud services MBS Plugins
, ,
1

You may know how to download a file with either our CURL functions or Insert From URL in FileMaker. But have you ever tried to download multiple files in parallel?

Let us show you three variants. First using FileMaker's Insert from URL, then CURL sequential and finally CURL parallel. The FileMaker one may quite common, but you can speed up your processing with parallel requests and run a lot of them at the same time. Let's say you have a job to download 4 files together and you like to do it faster. This can of course also be changed to do multiple uploads and of course you could use index variables and loops to code it for a variable number of requests.

First variant: Insert From URL

Using Insert From URL may be familiar, but isn't as flexible as our CURL functions:

Set Error Capture [ On ]
# 
# first one
Insert from URL [ Select ; With dialog: Off ; Target: CURL Test::HTML1 ; CURL Test::URL1 ; cURL options: " --trace $$debug1 " ] 
Set Variable [ $error1 ; Value: Get(LastError) ] 
If [ $error1 ] 
	Set Field [ CURL Test::debug ; $$debug1 ] 
End If
# 
# second download
Insert from URL [ Select ; With dialog: Off ; Target: CURL Test::HTML2 ; CURL Test::URL2 ; cURL options: " --trace $$debug2 " ] 
Set Variable [ $error2 ; Value: Get(LastError) ] 
If [ $error2 ] 
	Set Field [ CURL Test::debug ; $$debug2 ] 
End If
# 
# third download
Insert from URL [ Select ; With dialog: Off ; Target: CURL Test::HTML3 ; CURL Test::URL3 ; cURL options: " --trace $$debug3 " ] 
Set Variable [ $error3 ; Value: Get(LastError) ] 
If [ $error3 ] 
	Set Field [ CURL Test::debug ; $$debug3 ] 
End If
# 
# forth download
Insert from URL [ Select ; With dialog: Off ; Target: CURL Test::HTML4 ; CURL Test::URL4 ; cURL options: " --trace $$debug4 " ] 
Set Variable [ $error4 ; Value: Get(LastError) ] 
If [ $error4 ] 
	Set Field [ CURL Test::debug ; $$debug4 ] 
End If

Second variant: CURL sequential

Next we do the same with MBS FileMaker Plugin and just download all files one after one. We can reuse the $curl, so we may save some time to create and destroy curl objects.

# Start new session
Set Variable [ $curl ; Value: MBS("CURL.New") ] 
# 
# first one
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl; CURL Test::URL1) ] 
Set Variable [ $result1 ; Value: MBS("CURL.Perform"; $curl) ] 
If [ $result1 = "OK" ] 
	Set Field [ CURL Test::HTML1 ; MBS("CURL.GetResultAsText"; $curl; "UTF8") ] 
Else
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl) ] 
End If
# 
# second download
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl; CURL Test::URL2) ] 
Set Variable [ $result2 ; Value: MBS("CURL.Perform"; $curl) ] 
If [ $result2 = "OK" ] 
	Set Field [ CURL Test::HTML2 ; MBS("CURL.GetResultAsText"; $curl; "UTF8") ] 
Else
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl) ] 
End If
# 
# third download
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl; CURL Test::URL3) ] 
Set Variable [ $result3 ; Value: MBS("CURL.Perform"; $curl) ] 
If [ $result3 = "OK" ] 
	Set Field [ CURL Test::HTML3 ; MBS("CURL.GetResultAsText"; $curl; "UTF8") ] 
Else
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl) ] 
End If
# 
# forth download
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl; CURL Test::URL4) ] 
Set Variable [ $result4 ; Value: MBS("CURL.Perform"; $curl) ] 
If [ $result4 = "OK" ] 
	Set Field [ CURL Test::HTML4 ; MBS("CURL.GetResultAsText"; $curl; "UTF8") ] 
Else
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl) ] 
End If
# 
# Cleanup
Set Variable [ $result ; Value: MBS("CURL.Release"; $curl) ]

Third variant: CURL parallel

Finally we can show you running these four requests in parallel.

# Start new sessions
Set Variable [ $curl1 ; Value: MBS("CURL.New") ] 
Set Variable [ $curl2 ; Value: MBS("CURL.New") ] 
Set Variable [ $curl3 ; Value: MBS("CURL.New") ] 
Set Variable [ $curl4 ; Value: MBS("CURL.New") ] 
# 
# set all the URLs
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl1; CURL Test::URL1) ] 
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl2; CURL Test::URL2) ] 
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl3; CURL Test::URL3) ] 
Set Variable [ $r ; Value: MBS("CURL.SetOptionURL"; $curl4; CURL Test::URL4) ] 
# 
# launch the transfers
Set Variable [ $r ; Value: MBS("CURL.PerformAsync"; $curl1) ] 
Set Variable [ $r ; Value: MBS("CURL.PerformAsync"; $curl2) ] 
Set Variable [ $r ; Value: MBS("CURL.PerformAsync"; $curl3) ] 
Set Variable [ $r ; Value: MBS("CURL.PerformAsync"; $curl4) ] 
# 
# now wait for all to finish
Loop [ Flush: Always ]
	Pause/Resume Script [ Duration (seconds): ,01 ] 
	Set Variable [ $Running ; Value: MBS("CURL.IsRunning"; $curl1) or MBS("CURL.IsRunning"; $curl2) or MBS("CURL.IsRunning"; $curl3) or MBS("CURL.IsRunning"; $curl4) ] 
	Exit Loop If [ not $Running ] 
End Loop
# 
# check results
Set Variable [ $result1 ; Value: MBS("CURL.ErrorCode"; $curl1) ] 
Set Variable [ $result2 ; Value: MBS("CURL.ErrorCode"; $curl2) ] 
Set Variable [ $result3 ; Value: MBS("CURL.ErrorCode"; $curl3) ] 
Set Variable [ $result4 ; Value: MBS("CURL.ErrorCode"; $curl4) ] 
# 
# get the downloaded html
Set Field [ CURL Test::HTML1 ; MBS("CURL.GetResultAsText"; $curl1; "UTF8") ] 
Set Field [ CURL Test::HTML2 ; MBS("CURL.GetResultAsText"; $curl2; "UTF8") ] 
Set Field [ CURL Test::HTML3 ; MBS("CURL.GetResultAsText"; $curl3; "UTF8") ] 
Set Field [ CURL Test::HTML4 ; MBS("CURL.GetResultAsText"; $curl4; "UTF8") ] 
# 
# if one of them goes wrong, show log in a field
If [ $result1 ≠ 0 ] 
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl1) ] 
End If
If [ $result2 ≠ 0 ] 
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl2) ] 
End If
If [ $result3 ≠ 0 ] 
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl3) ] 
End If
If [ $result4 ≠ 0 ] 
	Set Field [ CURL Test::debug ; MBS("CURL.GetDebugMessages"; $curl4) ] 
End If
# 
# Cleanup
Set Variable [ $result ; Value: MBS("CURL.Release"; $curl1) ] 
Set Variable [ $result ; Value: MBS("CURL.Release"; $curl2) ] 
Set Variable [ $result ; Value: MBS("CURL.Release"; $curl3) ] 
Set Variable [ $result ; Value: MBS("CURL.Release"; $curl4) ]

If you run all three variants, you would see, that the last variant is faster. Your milage may vary depending on how quick servers react, the network connection and how big files are. There is time needed to connect, to initialize transport encryption, to send the request and wait for the answer to arrive. While this all happens for one request, you can also do the same for the next request. Bandwidth is shared, but for a lot of time of the script running, there is no packet transferred as one waits for the other.

Our CURL.PerformAsync function is great to perform things in background while your script runs with lots of parallel requests. The CURL.PerformInBackground function can run transfers on separate threads, so they won't stop even when FileMaker shows a dialog. And you can use it server side or in your FileMaker iOS SDK application, too.

Please try and let us know if you have questions.

1 Like