Hello,
the php bridge and FTP connection are good, but please consider that many clients could be on ADSL connection where the upload speed is just 1 Mbps, as my connection is at the moment.
My server, instead, besides being very powerful with 8 cores + 32GB RAM... also has 200-500 Mbps speed.
In my opinion obviosly you could keep FTP work by client as option, but the other option should be the Server job.
so, when using Icecat integration:
1) FTP to send just a file (or a few files) to the server (as you do with the PHP bridge)
2) in this file there should be any task which the Server should run in sequence by PHP
3) so... images URL to download from Icecat, routine for thumbnails generation and correct path where images must be placed
4) prepare a log file on the server, so you can check if everything was done fine (in order to verify)
5) if not repeat the task where it failed (eg. failed image download from Icecat)
New
1 vote
Vote
Reply
23 Answers
Feb 21, 2016
Alessiowrote
"In my opinion obviously..."
you need to consider that there are customers on dedicated servers, so they should have the option to exploit their servers, otherwise they are really underused with no reason.
then, also...
FTP connection by client could be unreliable (often it is)... if you have many images to upload you can't rely on ADSL provider...while the server connection is UP 99,99%
I have many issues with images at the moment.
thanks
Feb 22, 2016
Alex Pryshlivskiyagentwrote
You are right, this is interesting suggestion.
Right now you'll be able to increase image upload with help of bridge connector - http://addons.prestashop.com/en/fast-mass-updates/20242-bridge-connector-for-store-manager.html
Install connector manually on your PrestaShop Store.
So next time Store Manager will upload 1 image to Connector as a large picture instead of 7-8 images of 1 picture. All another generation of images will be done on your server by Bridge connector.
Feb 22, 2016
Alessiowrote
isn't that compatible with PS 1.4? why if store manager starts from 1.3 ?
Feb 22, 2016
Alex Pryshlivskiyagentwrote
What version of PrestaShop do you have? Bridge Connector works with PrestaShop ver.1.5 and higher. It's not reasonable to add this option for old versions of store.
Feb 24, 2016
Alessiowrote
well, I tried your bridge connector on PS 1.6.1.4 latest version
result is:
eg. about 4300-4500 products
about 12000 downloaded images (by client)... about 2 hours at 2MB/s
then it did more than 20.000 tasks on the queue
at an average of 1 per second... my upload speed is 1 Mbps
are you able to understand what this means for a multistore installation where the products are 60-100.000 for any store?
It means that the first import of a product list is a total nightmare.
Are you able to understand that your method is wrong (unfortunately) for a professional management and you need to take measures as soon as possible? to let the server do the complete image management (from download to anything else...) at least as an option.
About 8 hours (2+6) for just 4.000 products... what this means for 60-100.000 products and 6-7 stores (100.000 * 7) ?
Want to do a calculation yourself about the needed time? and not only... this maybe with someone who always keeps attention to your software, to repeat actions since the 30.000 batch on the Icecat window...strongly hoping that it doesn't become unstable at the end (already happened sometimes)...
and then, how to change the image folder? I don't want all those images on C:\... how to change local drive for the images when using the bridge connector?
thanks
Feb 25, 2016
Alessiowrote
UPDATE:
"then it did more than 20.000 tasks on the queue
at an average of 1 per second..."
no, that's even wrong...
process started at night about 00.30
now is 09:30 (exactly 9 hours) and it did about +7000 tasks on a total of 20000
so the average is 7000/9h = 777/h = 0.2/s !!!
Is it now clear that is impossible to manage and you need to add the option for image management by SERVER speed?
So, you need to do the exact opposite:
1) FIRST you let the server do its job for images... with the help of log files, maybe adding some options into the bridgeconnector
2) THEN you just download data from server to understand if images are in place, in order to sync client/server, retry if something's wrong JUST with the images not correctly set
ok...
thanks
Feb 25, 2016
Alex Pryshlivskiyagentwrote
Thanks for details, before I'd like how to speed up work with ICECat image.
If you want to speed up in 6 times you need to have:
1. Store Manager with PHP Bridge Connection
2. PrestaShop ver.1.6.x
Here it is simple scheme of work.
By steps:
1. Addon requests 4000 products with image - so we have 4000 images.
2. Addon will transfer 4 000 images to Store Manager
3. Stora Manager will upload 4000 images by FTP to PHP bridge connector.
4. PHP Bridge connector will generate images according to PrestaShop logics
So we have 4000 x 6 = 24 000 images on server.
In this case you don't need to transfer 24 thousands by FTP.
I told about this variant for ver. 1.6
Please note - right now we don't transfer URL only.
In your case there are 2 variants:
1 - You don't have brdige connection
OR
2. You have 4 000 products with Product Gallery - where did you get 20 000 tasks for 4300 products in this case ?
It seems you have direct connection in this case.
Could you clarify your connection details ? Could you provide FTP queue screenshot ? do you have FTP or HTTP threads ?
option for servers: shared and dedicated. You first try with PHP memory 512/1024MB execution time at least for one hour for dedicated (you can request this as requirement), do your test... so you can understand how large may be every batch for dedicated... according to that you proceed for shared (maybe starting from 32/64MB memory) and lower time consuming for any batch...
at the end of any batch you help yourself with the log files, even if errors you can easily fix them...
you can register any download result and any copy process to the correct folder for images/thumbnails...
and/or let the users choose if they want to retry where it failed...
Feb 25, 2016
Alex Pryshlivskiyagentwrote
Thanks for your comments.
This improvement was added by me to our list of future tasks.
Feb 26, 2016
Alessiowrote
it's HTTP for sure with bridge connector for sure... indeed I get many "Connection reset by peer" and it's EXPECTED, about timeout ;-) you can't set an unlimited timeout, 'cause it should be crazy
and yes, you are right when you talk about Product Gallery (or better, High Res images)
eg. a specific product on Icecat has:
6 images (high res only)
7.74 MB | 9.53 MB | 17.8 MB | 9.38 MB | 9.53 MB | 9.38 MB
Total 63.36 MB just for ONE product! ;-)
So, put a 1 Mbit/s upload speed.
What happens?
Put even you are able to upload 7 MB/minute .... it takes at least 10 minutes for ONE product
then the server must do the other work... and this is the BEST scenario, in practice this often doesn't happen
and also consider the timeout issues...
Instead, the dedicated servers currently usually download at a range of 25MB/s - 125MB/s, then may do the rest...
it means that the image management finally becomes a "child's play" ;-)
not only, by client you could download data in realtime and see if product images are in place without consuming client upload bandwidth... (which in many cases, eg. ADSL like mine at the moment, could be really limited)
Feb 26, 2016
Alessiowrote
then, as already said, you can even let the users choose if they want to download one image kind...
eg. download by server only the medium pics (if available) and let the server do the thumbnail regeneration
IF the medium pic is not there on Icecat for a specific image, download the High version...
If medium and high for one image of the product are not available, break the foreach loop and go to the other images of the product... and so on...
Feb 26, 2016
Alex Pryshlivskiyagentwrote
Good morning Alessio,
it's clear for me. You don't need to explain in details.
You are right, this is great suggestion.
I've got a meeting with our dev. team according to your situation.
Right now I'll not be able to change current development plan.
This task was added by me to our To-Do list.
From my side I want to invite you to be as a Beta tester when new improvement will be ready for testing. :-)
Feb 26, 2016
Alessiowrote
you could do this:
eg.
- select products (or even all products, or 30.000 products) from Store Manager client...
- choose to find matching products on Icecat
- when it's all about text (product name, description, features...) you can even keep the current method Icecat -> Client -> Server
- BUT if the user chooses to update images, instead, and the options on the Icecat plugin to work the images by Server are selected, you download only the images URLs as text (http://..........), then send a tasks file to the server and let the server do all the needed...
best regards
Feb 26, 2016
Alex Pryshlivskiyagentwrote
Sure, there are a lot of variants are possible for this idea. Let's begin from images transfer (because of Presta images generation behaviour) - a good beginning is a half of the battle :-)
Feb 27, 2016
Alessiowrote
;-) sure, I will be glad to test if you like ...even 'cause the default Presta csv import from backoffice currently seems to have issues when the images are from external source (eg. Icecat as in this case)... I'm developing my personal script to first download images, verify... and then assign the local images paths to a .csv
anyway, at the moment I don't know if local paths have issues too...
and if Store Manager will be able to easily do it... obviously better ;-)
and also, even better if Store Manager was able to do an automated import for Icecat
eg.
you auto import new products, after that daily routine has ended... you are able to auto import images from Icecat if found for the new added products
thanks
Feb 28, 2016
Alessiowrote
Update:
after my personal testing, my server was able to search by EAN about 7000 products, download any mid or high image (if mid, download mid, if not mid download high), writes the images local URLs in a .csv in about 37 minutes ;-)
then the default CSV import works fine when the images are on local path
with the format:
http://mydomain.com/path/to/file.jpg
then it takes about 3 minutes for any 1000 products import batch...
the 37 minutes period can even be improved 'cause into my script at the moment it also gets other infos I don't need...
I will retry setting to get images URLs only.
Feb 28, 2016
Alessiowrote
Test update:
52050 product rows by SKU and BRAND
just 1 hour and 44 minutes ;-) for searching and downloading images...
Max memory used by php process 384MB
Feb 28, 2016
Alessiowrote
Memory... (VIRT)
Feb 29, 2016
Alex Pryshlivskiyagentwrote
Are you talking about PrestaShop import products OR your custom PHP script operation ?
------------
"then the default CSV import works fine when the images are on local path
with the format:
http://mydomain.com/path/to/file.jpg
then it takes about 3 minutes for any 1000 products import batch..."
-------------
Feb 29, 2016
Alessiowrote
that's the final default Prestashop import...
before that, by PHP I had to search and download images, create a proper .csv
and It's really fast
p.s.: if users create the brands names accordingly to Icecat (and usually suppliers provide better SKUs compared to EANs, that is SKUs are 99% correct while EANs often are missing or bad)
searching on Icecat by SKU + correct BRAND name is really good and efficient
Feb 29, 2016
Alessiowrote
eg. if your brand is "ASUSTEK" better if you call it "ASUS"
if "BRAND MANAGEMENT GROUP" you need to rename it as "HP"
if "BUFFALO TECHNOLOGY" , "BUFFALO" is better...
otherwise results could be not accurate
and so on...
Feb 29, 2016
Alessiowrote
hey, forgot ;-) another option you could offer...
when user chooses to change product name if found on Icecat...
you could do a simple function to append the brand name to the product (if the user wants...)
that is, often when you update product name from Icecat it only writes a model name without brand... and it's not really good about searches on Google
so, let the users choose if they want to append the brand name before the product name...(if not already there in the name provided by Icecat) ;-)
Mar 01, 2016
Alex Pryshlivskiyagentwrote
Thanks for comment, Alessio!
From my side I'll try to add your suggestion to the description of task.
Let the server download images and put them in place
Let the server download images and put them in place