Listing import timeout and memory

General Import
1

Hi All,

I’m having issues with creating listings using the import due to memory usage. I fixed the timeout issue by increasing the php timeout settings, but now it’s the image processing that is causing my server to run out of memory. Any chance you have a snippet I can use to maybe include a pause between processsing files, or if it’s processing many simultaneously a snippet to process them individually? The files are already webp format and in an appropriate directory on the server. Thanks.

3

Hi,

Please send us a screenshot of the error or the text itself so that we can review it in more detail. Also, make sure that you do not have a memory limit from your hosting provider.

4

Hi Andrii,

Thanks for the reply, but we have gone for a major server upgrade that we needed anyway! However now when I’m running the import I notice it’s only utilizing one core - Is there any plans (or a snippet I can use now) to chunk the import into multiple requests so we can do some parallel processing on big imports?

6

Hi,

Unfortunately there’s no easy way to do this in the current version, the import process is not chunked. We plan to resolve the max_execution_time limit by adding a JS script that would re-send the import request once the max_execution_time is reached, and start from the last imported CSV row. Multiple cores may be useful for user activity on your website, when the website content is served to multiple users simultaneously.

Hope this helps

1 Like
7

Hi Ihor,

Thanks for the reply. That would be good, although I’m guessing most people just raise their php timeout setting so the import continues to run. Even with the JS fix you’d still get Cloudflare timeouts from the page, even though the process continues to run in the backround right? We are currently running imports of less than 200 listings at a time with images already uploaded via sftp, but with only one php worker processing it’s very slow.

Do you plan to update the listing in future to break down the import into x number of listings and assign a worker to them - by each block of listings starting a seperate request (if thats even possible)? Potentialy you could speed up import by x times the number of workers people have set.

9

Thanks for your feedback, we plan to break down the process by max_execution_time, but we’ll also research if using parallel processes is possible.

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.