Server-side ECID

27 May 2018 » MSA , Server Side

[UPDATE 12/05/2019] There are some limitations, which I have added towards the bottom

So, after reading my introduction to server-side digital marketing, you have deeply thought about it and now you are convinced that you should be going server-side. As with the JavaScript implementation, the first step you need is to get the Experience Cloud ID, or ECID for short (new name for the MCID). I assume you have a good understanding of the HTTP protocol, as this is going to be a more technical post.

One ECID per visitor

The first think you need to take into account is that, for each of the visitors of your website, you need to request a new ECID. This is exactly the same as with a JavaScript implementation: each visitor gets his own ECID. Do not even try to generate one ECID for all your connections: the reporting will stop working and there will be other unexpected consequences.

One consequence of the previous paragraph is that you have to implement a method to uniquely identify a visitor. Generally, this is managed with a 1st party cookie, which contains a random string. It is, however, beyond the scope of this post to explain how to generate and manage this unique ID.

Initial request

[UPDATE] The call is fully documented in here, in case you want to know more details than those I explain in this post.

The very first time you make a request, you need to call the following URL (always use HTTPS):

https://dpm.demdex.net/id?d_visid_ver=[VER]&d_fieldgroup=MC&d_rtbd=json&d_ver=2&d_verify=1&d_nsid=0&d_orgid=[ORGID]&ts=[TS]

The parameters your need are:

  • [VER] is the version of the JavaScript library you are trying to mimic. I have tested it for version 2.5.0, which is what I am going to use in this post. However, if new versions are released and you test the changes, feel free to update this value.
  • [ORGID] is the Organization ID. It it this string that ends in @AdobeOrg. In the examples below, remember to replace this placeholder with your own ORG ID. You can find it in the Administration section of your Experience Cloud UI:
    ORG ID
  • [TS] is the timestamp of when you are generating the call.

At the end of the post I have attached script that requests the ECID. I will use snippets of its output to show the process.

The initial request to get the server-side ECID will return a 302 redirect.

> GET /id?d_visid_ver=2.5.0&d_fieldgroup=MC&d_rtbd=json&d_ver=2&d_verify=1&d_nsid=0&d_orgid=[ORGID]&ts=1526812385 HTTP/1.1
> Host: dpm.demdex.net
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Cache-Control: no-cache,no-store,must-revalidate,max-age=0,proxy-revalidate,no-transform,private
< Date: Sun, 20 May 2018 10:33:05 GMT
< Expires: Thu, 01 Jan 2009 00:00:00 GMT
< Location: https://dpm.demdex.net/id/rd?d_visid_ver=2.5.0&d_fieldgroup=MC&d_rtbd=json&d_ver=2&d_verify=1&d_nsid=0&d_orgid=[ORGID]&ts=1526812385
< P3P: policyref="/w3c/p3p.xml", CP="NOI NID CURa ADMa DEVa PSAa PSDa OUR SAMa BUS PUR COM NAV INT"
< Pragma: no-cache
< Set-Cookie: demdex=08638584756313977084117322027292346811;Path=/;Domain=.demdex.net;Expires=Fri, 16-Nov-2018 10:33:05 GMT
< X-TID: TJYRdeinQpQ=
< Content-Length: 0
< Connection: keep-alive

As you can see, there is a “Set-Cookie” HTTP header (line 13) in the response from the server. You need to read this cookie, store all its parameters and send it again when following the redirect. Make now a request to the URL in the “Location” HTTP header (line 10), including this time the cookie you have just received.

> GET /id/rd?d_visid_ver=2.5.0&d_fieldgroup=MC&d_rtbd=json&d_ver=2&d_verify=1&d_nsid=0&d_orgid=[ORGID]&ts=1526812385 HTTP/1.1
> Host: dpm.demdex.net
> User-Agent: curl/7.47.0
> Accept: */*
> Cookie: demdex=08638584756313977084117322027292346811
> 
< HTTP/1.1 200 OK
< Cache-Control: no-cache,no-store,must-revalidate,max-age=0,proxy-revalidate,no-transform,private
< Content-Type: application/json; charset=UTF-8
< Date: Sun, 20 May 2018 10:33:05 GMT
< DCS: irl1-prod-dcs-05ee0df1f.edge-irl1.demdex.com 5.29.3.20180516081603 3ms
< Expires: Thu, 01 Jan 2009 00:00:00 GMT
< P3P: policyref="/w3c/p3p.xml", CP="NOI NID CURa ADMa DEVa PSAa PSDa OUR SAMa BUS PUR COM NAV INT"
< Pragma: no-cache
< Set-Cookie: demdex=08638584756313977084117322027292346811;Path=/;Domain=.demdex.net;Expires=Fri, 16-Nov-2018 10:33:05 GMT
< Vary: Accept-Encoding, User-Agent
< X-TID: oga+FiXcSz8=
< Content-Length: 365
< Connection: keep-alive
< 
{ [365 bytes data]

This request will return a JSON object (from d_rtbd=json), something like:

{
  "d_mid": "36800664351771965062969965047613395449",
  "id_sync_ttl": 604800,
  "d_blob": "RKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y",
  "dcs_region": 6,
  "d_ottl": 7200,
  "ibs": [
    {
      "id": "411",
      "ttl": 10080,
      "tag": "img",
      "fireURLSync": 1,
      "syncOnPage": 1,
      "url": [
        "//cm.everesttech.net/cm/dd?d_uuid=08638584756313977084117322027292346811"
      ]
    }
  ],
  "tid": "h3SU6B8yQEw="
}

The parameters that you need to extract and store are:

  • d_mid: this is the ECID, which you will need to send in Analytics and Target calls.
  • id_sync_ttl: this is the time in seconds you can keep this values in this file without refreshing them.
  • d_blob: I still do not quite understand why it is needed, but it is, so keep it too, for Analytics and Target.
  • dcs_region: demdex.net DCS region ID closest to where you are making the call.

So, by now you have requested and received an ECID. I will show you how to use it in the Analytics and Target calls in future posts.

Subsequent requests

This is not the end of the server-side ECID procedures. You need to honour the id_sync_ttl parameter in the response. By default, the value you get is 7 days. This means, that every 7 days you need to refresh the demdex cookie and the MID. Obviously, you only need to do that if the visitor has come back to your website. In order to do this refresh, you need to make again a similar call, now including the ECID you received from the first call, using the d_mid parameter:

https://dpm.demdex.net/id?d_visid_ver=[VER]&d_fieldgroup=MC&d_rtbd=json&d_ver=2&d_verify=1&d_nsid=0&d_orgid=[ORGID]&d_mid=[ECID]ts=[TS]

Remember to also include the demdex cookie you received in the initial call:

> GET /id?d_visid_ver=2.5.0&d_fieldgroup=MC&d_rtbd=json&d_ver=2&d_verify=1&d_nsid=0&d_orgid=[ORGID]&d_mid=36800664351771965062969965047613395449&ts=1526840994 HTTP/1.1
> Host: dpm.demdex.net
> User-Agent: curl/7.47.0
> Accept: */*
> Cookie: demdex=08638584756313977084117322027292346811
> 
< HTTP/1.1 200 OK
< Cache-Control: no-cache,no-store,must-revalidate,max-age=0,proxy-revalidate,no-transform,private
< Content-Type: application/json; charset=UTF-8
< Date: Sun, 20 May 2018 18:29:55 GMT
< DCS: irl1-prod-dcs-097e388bf.edge-irl1.demdex.com 5.29.3.20180516081603 2ms
< Expires: Thu, 01 Jan 2009 00:00:00 GMT
< P3P: policyref="/w3c/p3p.xml", CP="NOI NID CURa ADMa DEVa PSAa PSDa OUR SAMa BUS PUR COM NAV INT"
< Pragma: no-cache
< Set-Cookie: demdex=08638584756313977084117322027292346811;Path=/;Domain=.demdex.net;Expires=Fri, 16-Nov-2018 18:29:55 GMT
< Vary: Accept-Encoding, User-Agent
< X-TID: 7BWqVvnrSPA=
< Content-Length: 360
< Connection: keep-alive
< 
{ [360 bytes data]

The response will be, again, a JSON object. The d_mid parameter will be the same as before, but other parameters may have changed. Update your data accordingly.

Storage of the data

I have been a bit vague when it comes to explaining how to store the data. In the typical situation, when we are running the client-side VisitorAPI.js, the JavaScript code takes care of the storage of the data in cookies. However, when you are implementing a full server-side digital marketing, you need to put all the code and storage in your servers.

Many of the parameters I have mentioned will be unique per visitor, so you need a data structure to store this data. My recommendation is to create a table similar to the following:

Unique ID Demdex Demdex expiration ECID ECID refresh Blob DCS Region
123456 98765 1558378053 13579 1527446853 aqu38jda8q93jd9 6
654321 56789 1558458903 97531 1527446853 54k438fekf943oe 6

The columns are:

  • Unique ID. The unique ID of the visitor. This must be managed by the CMS, as no Adobe code is executed on the page. This will be your primary key to access the table.
  • Demdex. The value of the demdex cookie, as received in the “Set-Cookie” parameter. In a sense, this table acts as a cookie jar.
  • Demdex expiration. Expiration timestamp of the demdex cookie, converted from the “Expires” field of the “Set-Cookie” HTTP header.
  • ECID. The ECID value received from the call dpm.demdex.net, i.e., the d_mid value.
  • ECID refresh. Timestamp of when the ECID needs to be refreshed as documented above in the “subsequent requests” section. This is the addition of the timestamp of the request and the id_sync_ttl parameter. I think it is easier to store this value, as all you need to do for every request is to check the current timestamp against this value and it will tell you directly whether you need to refresh.
  • Blob. The blob received from the call dpm.demdex.net.
  • DCS Region. If you have servers around the world, this value will change. If all your servers are in the same data centre, this is probably going to be a constant.

Since this table could grow indefinitely, I also recommend a process to delete rows with expired demdex cookies.

Limitations [added 12/05/2019]

Although this solutions looks so fantastic, it has a severe limitation in a web environment. Browsers prevent scripts and servers from reading or writing 3rd party cookies. Therefore:

  • If the browser already has a demdex cookie, you will not be able to read it. In this case, the ECID will be disconnected from any existing demdex cookie, as a new one will be generated, which will only be stored server-side.
  • You will not be able to write the demdex cookie in the browser. Any segments the user qualifies for, you will not be able to share them with DSPs.

Since the ECID service is linked to AAM, the consequence is that AAM will become almost useless. Even if you do not have AAM today, in the future you may, in which case, you will have shot yourself in the foot.

Obviously, if you are using the ECID in a headless environment (voice assistants, IoT, chatbots…) then this is the only option you have, so use it.

Example of server-side ECID

As I said above, I have created a bash script to show how this works. I assume you have enough knowledge to run bash scripts and the command line:

  1. Make sure you have the command-line applications curl and jq installed.
  2. Download the script.
  3. Rename the script from .txt to .sh.
  4. Edit the file and set your ORG_ID at the top of the file. Do not use Windows Notepad for this change.
  5. Run the script: ./ecid.sh

The first time you run this script, it will behave like the section above “Initial request”. It will then generate two files:

  • cookies.txt: cookie jar with the demdex cookie.
  • ecid.json: full response of the the call to dpm.demdex.net in JSON format.

If you execute the script again, it will check whether it is time to refresh the MID, in which case, it will behave like the “Subsequent requests” section above. If you want to generate a new ECID, just delete these two files and execute again the script.

Let me know in the comments if you need any clarifications.

Related Posts