Process Feed

This document is in Beta stage and we have try to cover all topics in right hierarchy yet indepth. To make this document better, we need your help. Please give your feedback by rating the stars at the end of each topic. For more updates join slack channel #unbxd-knowledgebase

Use this document for internal purpose only. It is not suppose to be shared with customers


After we are done with feed review, we are ready to process the feed in the format in which it can be consumed by the Unbxd system. Having feed indexing done, covers the significant progress in the integration process. Once feed is indexed, rest of the integration can by continued as search and category APIs become available.

Once we get a feed from the customer and feed review process is completed then we are ready to work on a feed and make it in the format which can be ingested in the Unbxd database system.

As you already know the final feed structure which can be ingested in Unbxd system should look like this

 {
  "feed": {
    "catalog": {
      "add": {
        "items": [
          {
            "uniqueId": "ss10010",
            "title": "Short Sleeve Shirt",
            "description": "Get the perfect look for the summer.",
            "variants" : [
              {
                "variantId":"ff10010",
                "var_size" : "black",
                "var_title" : "Black short sleeve shirt"
              }
            ]
          },
          {
            "uniqueId":"10101",
            "title": "another product title",
            "another_product_field" : "another_key_value"
          },
          {
            "uniqueId":"20345",
            "title": "another product title",
            "another_product_field" : "another_key_value"
          }
        ]
      }
    }
  }
}

Feed Processing Stages

Source to Target Mapping

Source to target mapping is the table which maps how attributes in the given feed are mapped to unbxd attributes and what type of transformation we need to apply.

Consider this sample schema

productId,brand, instock, categoryIds, size,title,description,image,product_url,price,discount,parentId
"4321ANC","nike","1","home,womens,tops|home,womens","small","Pink Tshirt","Pink tshirt made up of cotton", "https://dummyurl.com/pink-t-shirt/img.jpg","https://dummyurl.com/pink-t-shirt",100,20, NULL
"4321DBC","adidas","1","home,womens,tops|home,womens","medium","Green Tshirt","Pink tshirt made up of cotton", "https://dummyurl.com/green-t-shirt/img.jpg","https://dummyurl.com/green-t-shirt",100,20, 4321ANC
"4321FGH","HnM","1","home,womens,tops|home,womens","small","red Tshirt","Red tshirt made up of cotton", "https://dummyurl.com/red-t-shirt/img.jpg","https://dummyurl.com/red-t-shirt",100,20, 4321ANC
"098ABD","Calvin","1","home,womens,tops|home,womens,tops,tshirt|home,new|home|sale","Black Tshirt","Black tshirt made up of cotton", "https://dummyurl.com/black-t-shirt/img.jpg","https://dummyurl.com/black-t-shirt",100,20, NULL

Once we are done with all the stages of feed review, we should start creating a source to target mapping. Source to target mapping is a document in which we maps feed attributes to the Unbxd attributes and while mapping what transformations need to apply.

There are three purpose of Source to Target mapping

  1. Understanding which feed attributes could be mapped to which Unbxd attributes.
  2. Understanding what and how transformations need to apply on the feed attributes so that it can be fed in the Unbxd systems
  3. Managing a holistic document which can then refer back later when there is a need to change the logic of any feed attribute.

Lets take an example to how to create Source to Target Mapping

Feed FieldFeed DatatypeTransformationUnbxd Feed Field
productIdtextDirectuniqueId
brandtextDirectbrand
instocktextrename to "availability" & transform value 1 to "True" and 0 to "False"availability
categoriestextTransformation Steps 1. Split categories string based on "|" to find multiple categories a product belong to 2. For each category split them on "," and then join with ">" 3. Append all categories in a listcategoryPath
sizetextDirectsize
titletextDirecttitle
descriptiontextDirectdescription
imagetext1. Rename image to imageURL 2. split image on "," and append each image in a listimageUrl
product_urltextRename product_url to productUrlproductUrl
pricetextTransform datatype from string to integerprice
discounttextTransform datatype from string to integerdiscount
parentIdtext1. It is auxillary field and its sole purpose it too determine variants 2. If parentId is NULL then it is main product 3. If parentId is NOT NULL then it is vairant of the product having uniqueId as the parentId of this variant
textDirectparentId

In Source to Target mapping, make sure all feed attributes are properly mapped to the Unbxd attributes.

Processing Script

Having source to target mapping in hand, we can start with developing feed processing script which will have primary three functions

  1. Transform the attributes from feed to Unbxd attributes as given in source to target mapping
  2. Implement the variant logic based on parentId
  3. Directly pass other feed attributes which may not be using in Unbxd system but useful for Implementing business logic on customer side.

So lets get started with creating a processing script. Script could be developed in any language of your choice. Here we will develop the processing script in python

Flow of Processing script

  1. Loop through each product in the feed
  2. Transform attributes as given in the source to target mapping and create a json of it
  3. Based on the key parentId , if a product is variant then push it as a part of variants in the main product
  4. Finally push products in the global list of products

<< Summary video on how to create feed >>

Once we have processed feed, we need to create the schema even before we can index the feed.


Did you find it useful? Rate it

← Feed Logic ReviewProcess Schema →