Telemetry Implementation

Following are the 3 ways to generate and send telemetry events. However note that this is applicable only when operating on DSEP 1.0

1. Using the Protocol Server

Protocol Server is enhanced to emit telemetry and send to ONEST Obsrv (the data platform where telemetry events are captured and stored). Use this GitHub Repo to get the updated protocol server code and use it. Backup the logs generated by protocol server till now before doing a deployment. Extract the API data from the backed up logs and ingest Telemetry Implementation Add the following configuration in default-bap-client.yml , default-bap-network.yml , default-bpp-client.yml , default-bpp-network.yml files and deploy the protocol server.

telemetry:
    network:
      url: "TELEMETRY_SERVER_URL"
    # raw events are optional
    raw:
      url: ""  
    batchSize: 100
    # In minutes
    syncInterval: 5
    # Either can be LOCAL or REDIS
    storageType: "LOCAL"
    backupFilePath: "backups"
    redis:
      db: 4
    messageProperties: []
 service: 
   name: "network_service"
   version: "1.0.0"

The protocol server will generate telemetry events and send them in batches of 100 or every 5 minutes to ONEST Obsrv.

Note: Configure the telemetry URL to "https://data.onest.network/data/v1/in/onest-telemetry-testing" for lower environments such as dev, staging, and preprod during testing. For the production telemetry URL, contact the ONEST team when the application is ready for go-live

2. Using the Open Network Telemetry SDK

Open Network Telemetry SDK is a javascript SDK which takes API request message and context object and generates the telemetry events.

Please refer to the readme for more details.

3. Using the API

Use the following API to send telemetry events to ONEST Obsrv.

API curl command

curl --location '{TELEMETRY_SERVER_URL}' \
--header 'Content-Type: application/json' \
--data '{
    "data": {
        "id": "onest-batch-1",
        "events": [{}]
    }
}'

Note: Send the telemetry events in a batch size of 50, 100 or send once in every 30 min.

Sample API Payloads

curl --location '{TELEMETRY_SERVER_URL}' \
--header 'Content-Type: application/json' \
--data '{
    "data": {
        "id": "onest-batch-1-1",
        "events": [
            {
                "resourceSpans": [
                    {
                        "resource": {
                            "attributes": [
                                {
                                    "key": "eid",
                                    "value": {
                                        "stringValue": "API"
                                    }
                                },
                                {
                                    "key": "producer",
                                    "value": {
                                        "stringValue": "fs-ps-bap-network.onest.network"
                                    }
                                },
                                {
                                    "key": "domain",
                                    "value": {
                                        "stringValue": "ONDC:ONEST12"
                                    }
                                }
                            ]
                        },
                        "scopeSpans": [
                            {
                                "scope": {
                                    "name": "network_service",
                                    "version": "1.0.0",
                                    "attributes": [
                                        {
                                            "key": "scope_uuid",
                                            "value": {
                                                "stringValue": "640a7f6438e722b31a8b44e23762767f"
                                            }
                                        },
                                        {
                                            "key": "checksum",
                                            "value": {
                                                "stringValue": ""
                                            }
                                        },
                                        {
                                            "key": "count",
                                            "value": {
                                                "intValue": 1
                                            }
                                        }
                                    ]
                                },
                                "spans": [
                                    {
                                        "name": "search",
                                        "traceId": "15e956e5-2a8d-4a3a-8556-e33108e70226",
                                        "spanId": "9509845c-5a32-4a8c-b5bd-78a1de071bd6",
                                        "span_uuid": "b63cc6bcc452a73fd479f2e888a5b044",
                                        "startTimeUnixNano": "1708691608401",
                                        "endTimeUnixNano": "1708691609440",
                                        "status": "Ok",
                                        "attributes": [
                                            {
                                                "key": "sender.id",
                                                "value": {
                                                    "stringValue": "fs-ps-bap-network.onest.network"
                                                }
                                            },
                                            {
                                                "key": "recipient.id",
                                                "value": {
                                                    "stringValue": "fs-ps-bpp-network.onest.network"
                                                }
                                            },
                                            {
                                                "key": "sender.uri",
                                                "value": {
                                                    "stringValue": "https://fs-ps-bap-network.onest.network/"
                                                }
                                            },
                                            {
                                                "key": "recipient.uri",
                                                "value": {
                                                    "stringValue": "https://fs-ps-bpp-network.onest.network"
                                                }
                                            },
                                            {
                                                "key": "sender.type",
                                                "value": {
                                                    "stringValue": "seeker"
                                                }
                                            },
                                            {
                                                "key": "recipient.type",
                                                "value": {
                                                    "stringValue": "provider"
                                                }
                                            },
                                            {
                                                "key": "observedTimeUnixNano",
                                                "value": {
                                                    "intValue": 1708691601754
                                                }
                                            },
                                            {
                                                "key": "http.request.method",
                                                "value": {
                                                    "stringValue": "POST"
                                                }
                                            },
                                            {
                                                "key": "http.request.route",
                                                "value": {
                                                    "stringValue": "/search"
                                                }
                                            },
                                            {
                                                "key": "http.request.host",
                                                "value": {
                                                    "stringValue": "8da0-49-207-215-34.ngrok-free.app"
                                                }
                                            },
                                            {
                                                "key": "http.request.scheme",
                                                "value": {
                                                    "stringValue": "http"
                                                }
                                            },
                                            {
                                                "key": "http.status.code",
                                                "value": {
                                                    "intValue": 200
                                                }
                                            }
                                        ],
                                        "events": []
                                    }
                                ]
                            }
                        ]
                    }
                ]
            }
        ]
    }
}'

Telemetry Event

{
 "resourceSpans": [{
   "resource": { // Required. Contextual information as per envelope structure
     Attribute("eid", String), // Required. Event ID - must be API for all api events
     Attribute("producer", String), // Required. Identifier of the system that produced this event
     Attribute("domain", String), // Required. Domain where the event has occurred.
   }, 
   "scopeSpans": [{
     "scope": { // Optional. Contextual information as per envelope structure
       "name": String, 
       "version": String, 
       "attributes": [
         Attribute("scope_uuid", String), // Optional. unique id for the batch for idempotency
         Attribute("checksum", String), // Optional. checksum to enable tampering checks
         Attribute("count", String), // Optional. Total count of spans in this batch 
       ]
     },
     "spans": [{ // Required. One or more API events in detail
       "name": String, // Required. API Name
       "traceId": String, // Required. Unique ID of the entire transaction
       "spanId": String, // Required. Unique ID of the API event call
       "span_uuid": String, //Required. Unique identifier for this span record
       "startTimeUnixNano": String, // Required. Start time of the API call in nano-seconds
       "endTimeUnixNano": String, // Required. End time of the API call in nano-seconds
       "status": String, // Required. one of Unset, Error, Ok
       "attributes": [ // Required. List of attributes providing additional details about the span
         Attribute("observedTimeUnixNano", String), // Required. Event generated time as ISO datetime
         Attribute("sender.id", String), // Required. Identifier of the system initiated the API call
         Attribute("recipient.id", String), // Required. Identifier of the system that is expected to be the recipient of the API call
         Attribute("http.method", String), // Required. Http method one of GET/POST/PATCH/DELETE etc
         Attribute("http.server_name", String), // Optional. Server name if any
         Attribute("http.route", String), // Required. URL of the request
         Attribute("http.scheme", String), // Optional. Is the call http or https
         Attribute("http.host", String), // Required. Host ip or domain name
         Attribute("http.status.code", Int), // Required. Status code of the API call
       ],
       "events": [{ // Optional. Capture additional API specific data like errors, req & res data, or multiple events that occurred during the API processing
         "name": String, // Required. Name of the additional data or event
         "time": String, // Required. Event time as ISO datetime
         "attributes": [Attribute] // Optional. List of attributes 
       }]
     }]
   }]
 }]
}

Example Sample Event

[
    {
        "resourceSpans": [
            {
                "resource": {
                    "attributes": [
                        {
                            "key": "eid",
                            "value": {
                                "stringValue": "API"
                            }
                        },
                        {
                            "key": "producer",
                            "value": {
                                "stringValue": "fs-ps-bap-network.onest.network"
                            }
                        },
                        {
                            "key": "domain",
                            "value": {
                                "stringValue": "ONDC:ONEST12"
                            }
                        }
                    ]
                },
                "scopeSpans": [
                    {
                        "scope": {
                            "name": "network_service",
                            "version": "1.0.0",
                            "attributes": [
                                {
                                    "key": "scope_uuid",
                                    "value": {
                                        "stringValue": "640a7f6438e722b31a8b44e23762767f"
                                    }
                                },
                                {
                                    "key": "checksum",
                                    "value": {
                                        "stringValue": ""
                                    }
                                },
                                {
                                    "key": "count",
                                    "value": {
                                        "intValue": 1
                                    }
                                }
                            ]
                        },
                        "spans": [
                            {
                                "name": "search",
                                "traceId": "15e956e5-2a8d-4a3a-8556-e33108e70226",
                                "spanId": "9509845c-5a32-4a8c-b5bd-78a1de071bd6",
                                "span_uuid": "b63cc6bcc452a73fd479f2e888a5b044",
                                "startTimeUnixNano": "1708691608401",
                                "endTimeUnixNano": "1708691609440",
                                "status": "Ok",
                                "attributes": [
                                    {
                                        "key": "sender.id",
                                        "value": {
                                            "stringValue": "fs-ps-bap-network.onest.network"
                                        }
                                    },
                                    {
                                        "key": "recipient.id",
                                        "value": {
                                            "stringValue": "fs-ps-bpp-network.onest.network"
                                        }
                                    },
                                    {
                                        "key": "sender.uri",
                                        "value": {
                                            "stringValue": "https://fs-ps-bap-network.onest.network/"
                                        }
                                    },
                                    {
                                        "key": "recipient.uri",
                                        "value": {
                                            "stringValue": "https://fs-ps-bpp-network.onest.network"
                                        }
                                    },
                                    {
                                        "key": "sender.type",
                                        "value": {
                                            "stringValue": "seeker"
                                        }
                                    },
                                    {
                                        "key": "recipient.type",
                                        "value": {
                                            "stringValue": "provider"
                                        }
                                    },
                                    {
                                        "key": "observedTimeUnixNano",
                                        "value": {
                                            "intValue": 1708691601754
                                        }
                                    },
                                    {
                                        "key": "http.request.method",
                                        "value": {
                                            "stringValue": "POST"
                                        }
                                    },
                                    {
                                        "key": "http.request.route",
                                        "value": {
                                            "stringValue": "/search"
                                        }
                                    },
                                    {
                                        "key": "http.request.host",
                                        "value": {
                                            "stringValue": "8da0-49-207-215-34.ngrok-free.app"
                                        }
                                    },
                                    {
                                        "key": "http.request.scheme",
                                        "value": {
                                            "stringValue": "http"
                                        }
                                    },
                                    {
                                        "key": "http.status.code",
                                        "value": {
                                            "intValue": 200
                                        }
                                    }
                                ],
                                "events": []
                            }
                        ]
                    }
                ]
            }
        ]
    },
    {
        "resourceSpans": [
            {
                "resource": {
                    "attributes": [
                        {
                            "key": "eid",
                            "value": {
                                "stringValue": "API"
                            }
                        },
                        {
                            "key": "producer",
                            "value": {
                                "stringValue": "fs-ps-bpp-network.onest.network"
                            }
                        },
                        {
                            "key": "domain",
                            "value": {
                                "stringValue": "ONDC:ONEST12"
                            }
                        }
                    ]
                },
                "scopeSpans": [
                    {
                        "scope": {
                            "name": "network_service",
                            "version": "1.0.0",
                            "attributes": [
                                {
                                    "key": "scope_uuid",
                                    "value": {
                                        "stringValue": "640a7f6438e722b31a8b44e23762767f"
                                    }
                                },
                                {
                                    "key": "checksum",
                                    "value": {
                                        "stringValue": ""
                                    }
                                },
                                {
                                    "key": "count",
                                    "value": {
                                        "intValue": 1
                                    }
                                }
                            ]
                        },
                        "spans": [
                            {
                                "name": "on_search",
                                "traceId": "15e956e5-2a8d-4a3a-8556-e33108e70226",
                                "spanId": "9509845c-5a32-4a8c-b5bd-78a1de071bd6",
                                "span_uuid": "b63cc6bcc452a73fd479f2e888a5b044",
                                "startTimeUnixNano": "1708691608401",
                                "endTimeUnixNano": "1708691609440",
                                "status": "Ok",
                                "attributes": [
                                    {
                                        "key": "sender.id",
                                        "value": {
                                            "stringValue": "fs-ps-bpp-network.onest.network"
                                        }
                                    },
                                    {
                                        "key": "recipient.id",
                                        "value": {
                                            "stringValue": "fs-ps-bap-network.onest.network"
                                        }
                                    },
                                    {
                                        "key": "sender.uri",
                                        "value": {
                                            "stringValue": "https://fs-ps-bpp-network.onest.network/"
                                        }
                                    },
                                    {
                                        "key": "recipient.uri",
                                        "value": {
                                            "stringValue": "https://fs-ps-bap-network.onest.network"
                                        }
                                    },
                                    {
                                        "key": "sender.type",
                                        "value": {
                                            "stringValue": "provider"
                                        }
                                    },
                                    {
                                        "key": "recipient.type",
                                        "value": {
                                            "stringValue": "seeker"
                                        }
                                    },
                                    {
                                        "key": "observedTimeUnixNano",
                                        "value": {
                                            "intValue": 1708691601754
                                        }
                                    },
                                    {
                                        "key": "http.request.method",
                                        "value": {
                                            "stringValue": "POST"
                                        }
                                    },
                                    {
                                        "key": "http.request.route",
                                        "value": {
                                            "stringValue": "/on_search"
                                        }
                                    },
                                    {
                                        "key": "http.request.host",
                                        "value": {
                                            "stringValue": "8da0-49-207-215-34.ngrok-free.app"
                                        }
                                    },
                                    {
                                        "key": "http.request.scheme",
                                        "value": {
                                            "stringValue": "http"
                                        }
                                    },
                                    {
                                        "key": "http.status.code",
                                        "value": {
                                            "intValue": 200
                                        }
                                    }
                                ],
                                "events": [
                                    {
                                        "name": "error",
                                        "time": "2023-04-29T18:52:58.114561Z",
                                        "attributes": [
                                            {
                                                "key": "type",
                                                "value": {
                                                    "stringValue": "CORE-ERROR"
                                                }
                                            },
                                            {
                                                "key": "code",
                                                "value": {
                                                    "stringValue": "ERR_BPP_1010"
                                                }
                                            },
                                            {
                                                "key": "msg",
                                                "value": {
                                                    "stringValue": "Unable to fetch the catalog. Please try later"
                                                }
                                            }
                                        ]
                                    }
                                ]
                            }
                        ]
                    }
                ]
            }
        ]
    }
]

Refer to this repo, for more details about the open network telemetry specifications

Test Telemetry Generation

1. Use the https://webhook.site/ to generate an endpoint URL.

2. Set the web hook URL as Telemetry URL.

3. Ensure telemetry events are getting generated for all the relevant APIs in this format.