{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# The Bigscreen Analytics Endpoint `/event`\n", "\n", "Bigscreen has an internal analytics system that collects data from a variety of sources, and then allows that data to be analysed with SQL queries. We primarily use a visualization tool called \"Metabase\" (available at https://metabase.bigscreencloud.com) to create queries and visualize the data.\n", "\n", "You can send analytics data into the Bigscreen analytics system by sending JSON data to the `/event` HTTP endpoint with a POST request. This system is explained in detail in this document, including working python examples.\n", "\n", "The analytics system internally is a write-only, append-only, distributed, relational, database. Behind the scenes we *mostly* use Google BigQuery for data storage and querying. Tables and columns are created and updated dynamically according to the demands of the programmer, so you can *mostly* work with the system without having to worry about database correctness and so on.\n", "\n", "The analytics system also has a lot of quality-of-life systems. All events automatically receive event timestamps, IP address data, sender device info, and so forth. This data is available in Metabase." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Preamble to all API calls\n", "import requests\n", "from dotenv import dotenv_values\n", "config = dotenv_values(\"default.env\")\n", "\n", "# The auth server is used for authentication and account management\n", "apiUrl = config['BIGSCREEN_API_URL']\n", "\n", "# An API key is required for each API call.\n", "apiBearerToken = config['BIGSCREEN_API_KEY']\n", "\n", "def getHeaders():\n", " return {\n", " \"Content-Type\": \"application/json\",\n", " \"Authorization\": f\"Bearer {apiBearerToken}\"\n", " }\n", "\n", "def login(email, password):\n", " payload = {\n", " \"email\": email,\n", " \"password\": password\n", " }\n", " r = requests.post(f\"{apiUrl}/login\", headers=getHeaders(), json=payload)\n", " return r.headers[\"x-refresh-token\"], r.headers[\"x-access-token\"]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Basic Usage Example\n", "\n", "A simple \"hello world\" example with exactly one data entry." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "payload = { \n", " \"event\": \"basic_event\",\n", " \"data\": {\n", " \"hello\": \"world\"\n", " }\n", "}\n", "\n", "r = requests.post(f\"{apiUrl}/event\", headers=getHeaders(), json=payload)\n", "assert r.status_code == 200, f\"status code should be 200 (actual: {r.status_code} {r.text})\"" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## What to send to the `/event` endpoint\n", "\n", "The data payload you send to the `/event` endpoint is a json object, and must always have at least two entries at the top level: \n", "* The `event` value - a string without any spaces. Allowed characters regex: `[A-Za-z0-9_-]+`\n", "* The `data` entry - a json dictionary.\n", "\n", "The `event` value is a string used to group together all the data together. It is is analogous to a table name in a database, or a class name in an object oriented system. Some rules for the event name:\n", "1. The event name should change often. It should not be randomly generated per call, or contain granular grouping identifiers such as the customer's account id.\n", "2. It should not clash with another `event` name that is already in use, if you're sending different data to the endpoint for that event.\n", "3. If you want to version the data, you can simply insert a version number in the event name (e.g. `my_analytics_event_v2`).\n", "\n", "The `data` entry must be a key-value dictionary. The following rules apply:\n", "1. The values can be any valid JSON datatype.\n", "2. The values *can* be dictionaries, and you can nest many levels deep, but be wary that this can make database querying difficult when we come to analyse the data.\n", "3. It is best if the `data` entry is a single level dictionary, e.g.:\n", "```\n", "{ \n", " \"event\": \"basic_event_trigger\",\n", " \"data\": {\n", " \"a\": \"b\",\n", " \"b\": 123,\n", " \"c\": true,\n", " \"d\": \"This is a sentence\",\n", " }\n", "}\n", "```\n", "4. The internal analytics system will insert \"quality of life\" data for you. All events will include a timestamp, and the IP address of the sender, if that can be determined. From unity, additional data is sent with every event, such as the unique device identifier, access token, and so forth.\n", "\n", "### Important points to consider when designing your analytics data\n", "\n", "When the analytics system receives a new event which it has never received before, it will create a new table to store that data. The columns in the table will map to the entries you send in the first level of the `data` entry.\n", "\n", "It is fine if you want to change the `data` you send with the event, but it is important to remember that no data is removed in the analytics system. If you change the structure of the data you send to an event endpoint, new column names will be created to support the new data, but the old columns will *not* be removed.\n", "\n", "However, **if the data type you send to an column changes (e.g. you send a number to a field that was previously a string), then your analytics events will fail to be recorded**. For this reason its best to get your analytics working correctly on the `dev` or `test` networks before pushing the data into production. If you want to change the datatype for a field, change the name of the field too.\n", "\n", "Here's an example:\n", "\n", "The following payload will create data in the `basic_data` table with an `a` column with strings, and the `b` column for numeric values.\n", "```\n", "{ \n", " \"event\": \"basic_data\",\n", " \"data\": {\n", " \"a\": \"b\",\n", " \"b\": 123\n", " }\n", "}\n", "```\n", "\n", "If you change the data to the following then a new `c` column will be created for strings. The `b` column will still exist.\n", "```\n", "{ \n", " \"event\": \"basic_data\",\n", " \"data\": {\n", " \"a\": \"b\",\n", " \"c\": \"hello world\"\n", " }\n", "}\n", "```\n", "\n", "However, if you send the following data then the attempt will fail because you are trying to store a string in a column that was previously designed to store numeric values:\n", "```\n", "{ \n", " \"event\": \"basic_data\",\n", " \"data\": {\n", " \"a\": \"b\",\n", " \"b\": \"hello world\"\n", " }\n", "}\n", "```\n", "\n", "### Analytics is write only\n", "\n", "The basic api only provides write access to the `/event` endpoint. The data can only be retrieved in Metabase, or from the admin API. The admin API is not available to the various Bigscreen clients (e.g. the Unity app has no access to the Admin API).\n", "\n", "The analytics system is not a fast lookup system, so it should not be used to store customer data the might be retrieved on a per-customer basis (e.g. do not store customer settings in there, for example, unless that data is important for analytics purposes)." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Examples\n", "\n", "## Example 1\n", "\n", "A data dictionary with different data types:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "payload = { \n", " \"event\": \"basic_event_trigger_2\",\n", " \"data\": {\n", " \"a\": \"b\",\n", " \"b\": 123,\n", " \"c\": True,\n", " \"d\": \"This is a sentence\",\n", " }\n", "}\n", "\n", "r = requests.post(f\"{apiUrl}/event\", headers=getHeaders(), json=payload)\n", "assert r.status_code == 200, f\"status code should be 200 (actual: {r.status_code} {r.text})\"" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Example 2 - tying analytics data to a user's account\n", "\n", "If a user is logged in, you can simply attach their access token to the payload and the analytics system will internally tie the data to the user's account." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "refreshToken, accessToken = login(config['BASIC_ACCOUNT_EMAIL'], config['BASIC_ACCOUNT_PASSWORD'])\n", "\n", "loggedInHeaders = {\n", " \"Content-Type\": \"application/json\",\n", " \"Authorization\": f\"Bearer {apiBearerToken}\",\n", " \"x-access-token\": accessToken\n", "}\n", "\n", "payload = { \n", " \"event\": \"event_with_access_token\",\n", " \"data\": {\n", " \"hello\": \"world\"\n", " }\n", "}\n", "\n", "r = requests.post(f\"{apiUrl}/event\", headers=loggedInHeaders, json=payload)\n", "assert r.status_code == 200, f\"status code should be 200 (actual: {r.status_code} {r.text})\"" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Example 3 - how to send engagement data\n", "\n", "**All events automatically receive event timestamps, IP address data, sender device info, and so forth.**\n", "\n", "This means it is very simple to record engagement data of any feature. The client can send data repeatedly on a fixed interval to the same endpoint. The usual way to do this is to send data every minute.\n", "\n", "In Metabase, we can simply count the number of rows in a table to determine the number of minutes spent using a feature, and we can further partition by account, ip address, day, week and so on.\n", "\n", "Here's an example that we could use to measure Beyond engagement - the client would send this data every minute:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "UNIQUE_SERIAL_NUMBER_HERE = \"whatever you want it to be\"\n", "\n", "payload = { \n", " \"event\": \"example_beyond_engagement_v1\",\n", " \"data\": {\n", " \"device_version\": \"v1\",\n", " \"serial_number\": UNIQUE_SERIAL_NUMBER_HERE\n", " }\n", "}\n", "\n", "r = requests.post(f\"{apiUrl}/event\", headers=getHeaders(), json=payload)\n", "assert r.status_code == 200, f\"status code should be 200 (actual: {r.status_code} {r.text})\"\n", "r.text" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Example 4 - Integration tests\n", "\n", "See `tests/api/054.BasicAnalytics.ts`, `test/api/062.AnalyticsUpgrade.ts` and `tests/api/069.AnalyticsEngagementV2.ts` for more complex examples." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Usage in Unity\n", "\n", "The Unity app already has an API for this - search for the `SendEvent` function in the `Accounts.cs` file. Internally, the event system attaches a lot of device information to each event, including the `uniqueDeviceIdentifier` as generated by Unity, so the programmer should just worry about the basic event data that is unique to that event. In many cases, just a unique event name will be enough.\n", "\n", "Some C# examples:\n", "\n", "### Example - basic click counter:\n", "```\n", "Accounts.SendEvent(\"BeyondAdClick\");\n", "```\n", "\n", "### Example - feature usage:\n", "```\n", "Accounts.SendEvent(\"StoreCheckout\",\"VR\");\n", "```\n", "\n", "### Example - custom data:\n", "```\n", "var eventData = new SimpleJSONBigscreen.JSONObject();\n", "eventData[\"bigMediaId\"] = mediaProduct.BigMediaId;\n", "eventData[\"mediaProductName\"] = mediaProduct.Title;\n", "eventData[\"store\"] = \"Oculus\";\n", "await Accounts.SendEvent(\"Purchase_Attempt\", eventData);\n", "```" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.11" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }