Skip to main content

Product data scraping

How Bambuser Live Shopping product data scraping works

This document provides a more detailed view of how the Bambuser Live Shopping Product data scraping works.

Test our product data scraper

If you do not have access to our dashboard, you can still test how our product scraper will behave with your product URLs. Visit this page and enter your product URL to test.


When Caller enters the Cobrowsing mode, some basic product details are scraped from the content of the given URL they are on (e.g. That information will be displayed to the Agent so he/she can better steer the conversation during the call.

The following properties will be extracted from the page:

  • a product name or title
  • an image URL (product thumbnail)
  • a brand name (not displayed anywhere)
  • a product reference (not displayed anywhere)

How does the product scraper work?

The scraper looks for different kinds of structured product data and metadata, using the following priority order:

  1. markup
    1. JSON-LD
    2. Microdata
  2. OpenGraph meta-tags (og:)
  3. Generic HTML tags

If the scraper is not able to find a product reference (SKU) it will use the provided product URL as a reference.

1. markup


Google's testing tool can be used to see if your site supports this:


<script type="application/ld+json">
"@type": "Product",
"@context": "",
"name": "My Product Name",
"description": "My Description",
"brand": { "@type": "Thing", "name": "My Brand Name" },
"image": "",
"sku": "product-sku-12345"

Microdata Beta


<div itemscope itemtype="">
<span itemprop="name">My Product Name</span>
<span itemprop="brand">My Brand Name</span><br>
<img itemprop="image" src=""><br>
<span itemprop="description">Some optional description</span><br>
Product number: <span itemprop="sku" content="product-sku-12345"></span><br>

2. OpenGraph meta-tags (og:)



<meta property="og:type" content="og:product" />
<meta property="og:title" content="My Product Name" />
<meta property="product:brand" content="My Brand Name" />
<meta property="og:image" content="http://path-to-thumbnail" />
<meta property="og:description" content="Some optional description!" />
<meta property="product:retailer_item_id" content="product-sku-12345" />

3. Generic meta tags

If the aforementioned structured product data are not found, the product scraper looks for generic information found on most websites such as the title element, images.

<title> My Product Page Name </title>
<img src="">


The Bambuser Product Scraper server is located in the US. Your assets need to be accessible from US-based IP addresses. Otherwise, you need to whitelist our product scraper as described in the following.

Whitelist the scraper

An example use case for when you need to whitelist our scraper is when you intend to add products from your staging/test environment that is not publicly accessible. You can make an exemption for our scraper user-agent or whitelist static IP address.

By User-agent:

The scraper will identify itself with the following user-agent: BambuserLiveShopping/1.0. You can make an exception for requests made by this user-agent.
Once whitelisted the user-agent, it should start working right away.

By Static IP address:

You can also whitelist our scraper through the static IP address:

  • Global server:
  • EU server:
Static IP Activation

Beside whitelisting our static IP address from your side, you also need to inform Bambuser staff to enable 'Static IP proxy' for your organization.

Whitelist Bambuser Image Transformer

If your products are scraped correctly, but the images/thumbnails are not shown properly, you may need to whitelist our image transformer.
This can be due to restrictions from your CDN that blocks requests from our cloud-based image transformer.

This issue is also common if you are using Akamai CDN with strict rules.

Follow the same process as for whitelisting the scraper on your CDN rules.
For best performance, we recommend that you only whitelist Bambuser Image Transformer by user-agent (BambuserLiveShopping/1.0) as this option does not require an additional proxy stage.

Product data scraping FAQ

We highly recommend you to use the JSON-LD format of the

  • Ensure your URL is also accessible from the US regions since our scraper is located in the United States. If you are outside of the US, you can test this using a VPN.

  • You can check the forwarded error response in the network tab of your browser dev tool.

For the scraper to initialize the product template fields with correct values, there must be valid structured data available and accessible on your product page.

  • Ensure that the structured data are available on the page load response and not loaded and rendered after the main page request.

How to check that?

Navigate to your product page (same URL you are trying to add), right-click, and select the "View Page Source" to open the response of the request to the product URL. Then look into the source code and double-check if the structured data exist, is valid, and has correct values.

There might be errors in JSON-LD data that Google Structured data tool does not complain about. Use to check the validity of your JSON-LD data.

If everything still looks fine on your side, contact the support department at