Hello,
I’m involved in running non-profit grocery store, and would like to automate part of the work involved in selecting products from a supplier online catalog/shop (which is based on prestashop).
Ideally, I’d need to fetch information about a subset of products (name, packaging, price), and then integrate with some polling system to collectively decide which ones we order.
Currently the scraping is done manually with copy and past to a google sheet, and the polling is done in whatsapp (with copy and paste again).
My first attempt was to improve the google sheet : it store the URLs of some products, and the product name and packaging are scraped with IMPORTXML, using its xpath capabilities.
It works, but
1/ it is not very flexible (I’d better use JS or python than sheet formulas for filtering)
2/ IMPORTXML cannot handle authentication when scraping, and thus cannot see the products prices, which are not public.
I think the next step in google ecosystem is using apps script which would enable authentication and advanced filtering (but does not have xpath out of the box).
But I wonder what would be my options in grist (hosted or self-hosted) ecosystem? Ideally using open-source/no lock-in tools.
I think I could write the python or JS for the following features:
- given a reference number, find the corresponding product URL
- given an EAN number, find the corresponding product URL
- given a product url, fetch name, packaging, price
But I don’t know where to run them nor how to integrate them in grist.
(This topic gave some hints : Extracting title from a webpage . But it is quite old.)
What would be the big picture of such integration?
Note: maybe I should instead use some browser extension for the web scraping part, and just copy the result in grist for the poll. I’m currently ignorant about those extension, an I’m unsure they can be trusted.