{"version":"https://jsonfeed.org/version/1.1","title":"Jaryl Chng's Knowledge Base","home_page_url":"https://kb.jarylchng.com","feed_url":"https://kb-jarylchng-com.pages.dev/json/","description":"

Welcome to the index page of my knowledge base, if you haven't done so, do visit my website at https://jarylchng.com.

I will mainly use this site to document stuff, most of which will likely be in the public domain.

","icon":"https://kb-static.jarylchng.com/kb-jarylchng-com/production/images/channel-c68f1f55f856ab833b4365991609dbec.png","favicon":"https://kb-static.jarylchng.com/kb-jarylchng-com/production/images/favicon-b94914f57599a477f9f72dab6bc71001.png","authors":[{"name":"Jaryl Chng"}],"language":"en-us","items":[{"id":"k95O_GJ6Sdf","title":"N8N - Carousell.sg - Automatic Reply Bot","url":"https://kb.jarylchng.com/i/n8n-carousellsg-automatic-reply-bot-k95O_GJ6Sdf/","content_html":"

This has been superceded by Carousell GoBot project.

Context

I was looking for ideas to try out n8n.io's capabilities and picked this as the first big idea to try out.

Basically, I would like to n8n.io to handle chat messages and offers from Carousell, a Singaporean marketplace platform, similar to Facebook Marketplace.

Right now I would only like it to:

The problem was that Carousell does not have any public documentation on their APIs and I have to resort to inspecting the network traffic manually to achieve my goals.

Docker image: jarylc/n8n

Process

Automatic Triggers

There are 3 ways to this workflow is automatically triggered:

-> Webhook - Using Tasker on my Android device, it will send a GET request to this webhook everytime a notification from Carousell app arrives.

-> IMAP - Reads a dedicated inbox for e-mails from Carousell and trigger if so, acts as fallback for the above.

-> Cron - Every 30 minutes as fallback in case all of the above doesn't work. Note that there is a better to do this if standalone, you could just connect to the chat WebSocket and wait for messages.

Retrieve Chat Data and Preliminary Checks

When meddling around https://www.carousell.sg/inbox/received/ and inspecting the network trace, I found out that Carousell has an undocumented API to retreive that I could tap on.

This API call allows me to retrieve the list of messages and offers that I have recieved as a seller which I put into good use as the first step after the trigger.

It seems that only the Cookie header is required to authenticate myself.

The preliminary checks consists of checking if the chat list is empty, or the very last message contains my message signature.

Now this comes the hard part for this workflow, upon inspecting, Carousell uses SendBird for their chat platform which uses WebSockets instead of API calls.

I had to customize my n8n.io installation to include ws Node dependency and add it to NODE_FUNCTION_ALLOW_EXTERNAL environment variable to be used in the vm2 environment. This was made easy with my own Docker image on Docker hub using the environment variable ADDITIONAL_MODULES, jarylc/n8n.

After which, I had to mimic the WebSocket calls when chatting and came up with this final JavaScript node.

Custom function codes

Check & Split Checks

const fs = require(\"fs\")\n\nconst seen = JSON.parse(fs.readFileSync('/data/carousell.json', 'utf8'))\n\nconst split = []\nlet save = {}\nfor (const item of $node[\"HTTP Request\"].json[\"data\"][\"offers\"]) {\n  if (item['state'] === 'A')\n    continue\n\n  let old_price = seen[item['id']] || 0.0\n  let latest_price = parseFloat(item['latest_price'].replace(',', ''))\n\n  let message_price_search = item['latest_price_message'].match(/^(\\d{1,5}\\.?\\d{0,2})$|(\\d+\\.?\\d{0,2}((?<=(\\$|offer|quote|can|please|pls|quick|fast|sell).*)|(?=.*(\\$|offer|quote|can|please|pls|quick|fast|bucks|ok|\\?))))/gi)\n  if (message_price_search != null) {\n    let message_price = parseFloat(message_price_search[message_price_search.length - 1])\n    if (message_price > old_price && message_price > latest_price) {\n      latest_price = message_price\n      item['latest_price_formatted'] = (''+latest_price.toFixed(2)).replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\")\n    }\n  }\n\n  save[item['id']] = latest_price\n  if (item['id'] in seen && old_price >= latest_price)\n    continue\n\n  item['price_changed'] = old_price !== 0.0\n  item['low_balled'] = latest_price !== 0 && latest_price <= parseFloat(item['product']['price']) * 0.85\n\n  split.push({json: item})\n}\n\nif (split.length > 0) {\n  const fs = require(\"fs\")\n  fs.writeFileSync('/data/carousell.json', JSON.stringify(save))\n}\n\nreturn split\n

Send Reply

const WebSocket = require('ws')\n\nreturn await new Promise((resolve, reject) => {\n  let sent = false\n  const sendbird_subdomain = items[0].json['channel_url'].slice(0, items[0].json['channel_url'].indexOf('-carousell')).toLowerCase()\n  const client = new WebSocket('wss://ws-' + sendbird_subdomain + '.sendbird.com/?p=JS&pv=Mozilla%2F5.0%20(X11%3B%20Linux%20x86_64%3B%20rv%3A90.0)%20Gecko%2F20100101%20Firefox%2F90.0&sv=3.0.149&ai=F3CB6187-CB42-4CD1-95FC-1C46F8856006&user_id=344194&access_token=' + $node['Get Chat Token'].json['data']['token'] + '&active=1&SB-User-Agent=JS%2Fc3.0.149%2F%2F&Request-Sent-Timestamp=' + Date.now() + '&include_extra_data=premium_feature_list%2Cfile_upload_size_limit%2Capplication_attributes%2Cemoji_hash', {\n    perMessageDeflate: true\n  })\n  client.on('message', (data) => {\n    if (data.startsWith('LOGI') && !sent) {\n      sent = true\n      for (const i in items) {\n        try {\n          const item = items[i].json\n          let text = 'Hello @' + item['user']['username'] + '!\\n\\n' +\n            'Thank you for your '\n          if (item['latest_price_formatted'] !== '0') {\n            text += (item['price_changed'] ? 'new ' : '') + 'offer of ' + item['currency_symbol'] + item['latest_price_formatted'] + ' on'\n          } else {\n            text += 'interest in'\n          }\n          text += ' my item: ' + item['product']['title'] + '.'\n          if (!item['is_product_sold'] && item['product']['status'] !== 'R') {\n            if (item['low_balled'])\n              text += '\\n\\nWARNING: Offer price is more than 15% below listing price, it is too low and may not get a future reply unless you increase it!'\n\n            if (!item['price_changed']) {\n              text += '\\n\\nFAQ:\\n' +\n                '» Where do I normally deal?\\n' +\n                'Choa Chu Kang or Bukit Panjang area if I am not in office, Bencoolen area if I am.\\n' +\n                '» What payment methods do I accept?\\n' +\n                'In order of preference: Google Pay, PayLah, PayNow, Cash, CarouPay, Bank Transfer\\n' +\n                '» What happens if I did not reply?\\n' +\n                'Very likely you offered too low-ball of a price' + (item['low_balled'] ? ' (which you probably did)' : '') + '. If not, feel free to message me again.'\n            }\n          } else {\n            text += '\\n\\nHowever, this listing has already been ' + (item['is_product_sold'] ? 'sold' : 'reserved') + ' and not available anymore.'\n          }\n          text += '\\n\\n- @jarylc'\n\n          const msg = {\n            channel_url: '' + item['channel_url'],\n            message: text,\n            data: JSON.stringify({\n              offer_id: '' + item['id'],\n              source: 'web'\n            }),\n            mention_type: 'users',\n            mentioned_user_ids: [],\n            custom_type: 'MESSAGE',\n            req_id: Date.now()\n          }\n\n          client.send('MESG' + JSON.stringify(msg) + '\\n')\n        } catch\n          (err) {\n          reject(err)\n        }\n      }\n    } else if (data.startsWith('READ')) {\n      client.terminate()\n      resolve(true)\n    }\n  })\n  client.on('error', (err) => {\n    reject(err)\n  })\n  client.on('open', () => {\n    setTimeout(() => {\n      client.terminate()\n      resolve(true)\n    }, 10000)\n  })\n}).then(_ => {\n  return []\n}).catch(err => {\n  throw err\n})\n

Telegram Notify Flow

The other branch just forwards this to my personal Telegram bot so that I can be notified via another channel. Right now it will also tell me if the offer was a low-ball or not.

","content_text":"[https://kb-static.jarylchng.com/kb-jarylchng-com/production/media/rich-editor/items/k95O_GJ6Sdf/image-11ef9b231e071e02933af945d3bd0829.png]\n\n\nTHIS HAS BEEN SUPERCEDED BY CAROUSELL GOBOT PROJECT.\n\n\nCONTEXT\n\nI was looking for ideas to try out n8n.io's capabilities and picked this as the\nfirst big idea to try out.\n\nBasically, I would like to n8n.io to handle chat messages and offers from\nCarousell, a Singaporean marketplace platform, similar to Facebook Marketplace.\n\nRight now I would only like it to:\n\n * Reply a standard response template on new messages and offers\n * Forward to my personal Telegram bot to notify me via one additional channel\n * Detect any low-balling (offering a price much lower than the listed price)\n\nThe problem was that Carousell does not have any public documentation on their\nAPIs and I have to resort to inspecting the network traffic manually to achieve\nmy goals.\n\nDocker image: jarylc/n8n\n\n\nPROCESS\n\n\nAUTOMATIC TRIGGERS\n\nThere are 3 ways to this workflow is automatically triggered:\n\n-> Webhook - Using Tasker on my Android device, it will send a GET request to\nthis webhook everytime a notification from Carousell app arrives.\n\n-> IMAP - Reads a dedicated inbox for e-mails from Carousell and trigger if so,\nacts as fallback for the above.\n\n-> Cron - Every 30 minutes as fallback in case all of the above doesn't work.\nNote that there is a better to do this if standalone, you could just connect to\nthe chat WebSocket and wait for messages.\n\n\nRETRIEVE CHAT DATA AND PRELIMINARY CHECKS\n\nWhen meddling around https://www.carousell.sg/inbox/received/ and inspecting the\nnetwork trace, I found out that Carousell has an undocumented API to retreive\nthat I could tap on.\n\nThis API call allows me to retrieve the list of messages and offers that I have\nrecieved as a seller which I put into good use as the first step after the\ntrigger.\n\nIt seems that only the Cookie header is required to authenticate myself.\n\nThe preliminary checks consists of checking if the chat list is empty, or the\nvery last message contains my message signature.\n\nNow this comes the hard part for this workflow, upon inspecting, Carousell uses\nSendBird for their chat platform which uses WebSockets instead of API calls.\n\nI had to customize my n8n.io installation to include ws Node dependency and add\nit to NODE_FUNCTION_ALLOW_EXTERNAL environment variable to be used in the vm2\nenvironment. This was made easy with my own Docker image on Docker hub using the\nenvironment variable ADDITIONAL_MODULES, jarylc/n8n.\n\nAfter which, I had to mimic the WebSocket calls when chatting and came up with\nthis final JavaScript node.\n\n\nCUSTOM FUNCTION CODES\n\nCheck & Split Checks\n\nconst fs = require(\"fs\")\n\nconst seen = JSON.parse(fs.readFileSync('/data/carousell.json', 'utf8'))\n\nconst split = []\nlet save = {}\nfor (const item of $node[\"HTTP Request\"].json[\"data\"][\"offers\"]) {\n if (item['state'] === 'A')\n continue\n\n let old_price = seen[item['id']] || 0.0\n let latest_price = parseFloat(item['latest_price'].replace(',', ''))\n\n let message_price_search = item['latest_price_message'].match(/^(\\d{1,5}\\.?\\d{0,2})$|(\\d+\\.?\\d{0,2}((?<=(\\$|offer|quote|can|please|pls|quick|fast|sell).*)|(?=.*(\\$|offer|quote|can|please|pls|quick|fast|bucks|ok|\\?))))/gi)\n if (message_price_search != null) {\n let message_price = parseFloat(message_price_search[message_price_search.length - 1])\n if (message_price > old_price && message_price > latest_price) {\n latest_price = message_price\n item['latest_price_formatted'] = (''+latest_price.toFixed(2)).replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\")\n }\n }\n\n save[item['id']] = latest_price\n if (item['id'] in seen && old_price >= latest_price)\n continue\n\n item['price_changed'] = old_price !== 0.0\n item['low_balled'] = latest_price !== 0 && latest_price <= parseFloat(item['product']['price']) * 0.85\n\n split.push({json: item})\n}\n\nif (split.length > 0) {\n const fs = require(\"fs\")\n fs.writeFileSync('/data/carousell.json', JSON.stringify(save))\n}\n\nreturn split\n\n\nSend Reply\n\nconst WebSocket = require('ws')\n\nreturn await new Promise((resolve, reject) => {\n let sent = false\n const sendbird_subdomain = items[0].json['channel_url'].slice(0, items[0].json['channel_url'].indexOf('-carousell')).toLowerCase()\n const client = new WebSocket('wss://ws-' + sendbird_subdomain + '.sendbird.com/?p=JS&pv=Mozilla%2F5.0%20(X11%3B%20Linux%20x86_64%3B%20rv%3A90.0)%20Gecko%2F20100101%20Firefox%2F90.0&sv=3.0.149&ai=F3CB6187-CB42-4CD1-95FC-1C46F8856006&user_id=344194&access_token=' + $node['Get Chat Token'].json['data']['token'] + '&active=1&SB-User-Agent=JS%2Fc3.0.149%2F%2F&Request-Sent-Timestamp=' + Date.now() + '&include_extra_data=premium_feature_list%2Cfile_upload_size_limit%2Capplication_attributes%2Cemoji_hash', {\n perMessageDeflate: true\n })\n client.on('message', (data) => {\n if (data.startsWith('LOGI') && !sent) {\n sent = true\n for (const i in items) {\n try {\n const item = items[i].json\n let text = 'Hello @' + item['user']['username'] + '!\\n\\n' +\n 'Thank you for your '\n if (item['latest_price_formatted'] !== '0') {\n text += (item['price_changed'] ? 'new ' : '') + 'offer of ' + item['currency_symbol'] + item['latest_price_formatted'] + ' on'\n } else {\n text += 'interest in'\n }\n text += ' my item: ' + item['product']['title'] + '.'\n if (!item['is_product_sold'] && item['product']['status'] !== 'R') {\n if (item['low_balled'])\n text += '\\n\\nWARNING: Offer price is more than 15% below listing price, it is too low and may not get a future reply unless you increase it!'\n\n if (!item['price_changed']) {\n text += '\\n\\nFAQ:\\n' +\n '» Where do I normally deal?\\n' +\n 'Choa Chu Kang or Bukit Panjang area if I am not in office, Bencoolen area if I am.\\n' +\n '» What payment methods do I accept?\\n' +\n 'In order of preference: Google Pay, PayLah, PayNow, Cash, CarouPay, Bank Transfer\\n' +\n '» What happens if I did not reply?\\n' +\n 'Very likely you offered too low-ball of a price' + (item['low_balled'] ? ' (which you probably did)' : '') + '. If not, feel free to message me again.'\n }\n } else {\n text += '\\n\\nHowever, this listing has already been ' + (item['is_product_sold'] ? 'sold' : 'reserved') + ' and not available anymore.'\n }\n text += '\\n\\n- @jarylc'\n\n const msg = {\n channel_url: '' + item['channel_url'],\n message: text,\n data: JSON.stringify({\n offer_id: '' + item['id'],\n source: 'web'\n }),\n mention_type: 'users',\n mentioned_user_ids: [],\n custom_type: 'MESSAGE',\n req_id: Date.now()\n }\n\n client.send('MESG' + JSON.stringify(msg) + '\\n')\n } catch\n (err) {\n reject(err)\n }\n }\n } else if (data.startsWith('READ')) {\n client.terminate()\n resolve(true)\n }\n })\n client.on('error', (err) => {\n reject(err)\n })\n client.on('open', () => {\n setTimeout(() => {\n client.terminate()\n resolve(true)\n }, 10000)\n })\n}).then(_ => {\n return []\n}).catch(err => {\n throw err\n})\n\n\n\nTELEGRAM NOTIFY FLOW\n\nThe other branch just forwards this to my personal Telegram bot so that I can be\nnotified via another channel. Right now it will also tell me if the offer was a\nlow-ball or not.","date_published":"2021-06-17T10:02:00.000Z","_microfeed":{"web_url":"https://kb-jarylchng-com.pages.dev/i/n8n-carousellsg-automatic-reply-bot-k95O_GJ6Sdf/","json_url":"https://kb-jarylchng-com.pages.dev/i/k95O_GJ6Sdf/json/","rss_url":"https://kb-jarylchng-com.pages.dev/i/k95O_GJ6Sdf/rss/","guid":"k95O_GJ6Sdf","status":"published","itunes:episodeType":"full","date_published_short":"Thu Jun 17 2021","date_published_ms":1623924120000}}],"_microfeed":{"microfeed_version":"0.1.2","base_url":"https://kb-jarylchng-com.pages.dev","categories":[{"name":"Technology"}],"subscribe_methods":[{"name":"RSS","type":"rss","url":"https://kb-jarylchng-com.pages.dev/rss/","image":"https://kb-jarylchng-com.pages.dev/assets/brands/subscribe/rss.png","enabled":true,"editable":false,"id":"sQbXXExV58H"},{"name":"JSON","type":"json","url":"https://kb-jarylchng-com.pages.dev/json/","image":"https://kb-jarylchng-com.pages.dev/assets/brands/subscribe/json.png","enabled":true,"editable":false,"id":"nC8cjLCnOOi"}],"description_text":"Welcome to the index page of my knowledge base, if you haven't done so, do visit\nmy website at https://jarylchng.com.\n\nI will mainly use this site to document stuff, most of which will likely be in\nthe public domain.","copyright":"©2024","itunes:type":"episodic","items_sort_order":"newest_first"}}