Reading SystemLink DataFinder File Information in a Jupyter Notebook

This article demonstrates how to interact with a SystemLink DataFinder Instance from a Jupyter Notebook by using the Data Navigator Data Service API.

The example code below demonstrates how to write a Jupyter Notebook that retrieves file metadata and bulk data from a DataFinder Instance. The code is split into the following sections:

• Connecting to a DataFinder Instance
• Retrieving File Metadata
• Retrieving Group Metadata
• Retrieving Channel Metadata
• Retrieving Channel Values
   

Prerequisites

Software installation on the SystemLink Server PC:

• NI SystemLink Server - JupyterHub Module
• NI SystemLink TDM DataFinder Module

SystemLink Server configuration requirements:

• An existing DataFinder Instance that has indexed some files.
• Custom properties that should be accessible from Jupyter must be optimized from the DataFinder.

Connecting to a DataFinder Instance

Follow the steps below to create a Jupyter Notebook that connects to a specific SystemLink DataFinder Instance. Once a connection is established, the open connection can be used to retrieve file, group and channel data.

1. Access the SystemLink web UI and log in.
2. From the navigation menu, select Utilities >> Jupyter.
3. Create a new Jupyter Notebook by clicking File >> New >> Notebook.

4. Rename the notebook to something useful.
   • The example in this article is called Query DataFinder Example.ipynb.
5. In the first notebook cell, import the pandas, scrapbook, requests and base64 modules (as shown below).
   • The pandas module is required to build DataFrames.
   • The scrapbook module is required to output the DataFrame from the notebook.
   • The requests module is used to make HTTP requests to the Data Navigator API.
   • The base64 module is used for creating an authentication string for the API.

import pandas as pd
import scrapbook as sb
import requests
from base64 import b64encode

6. In the next code cell, start by defining the variables shown below. Modify their values to suit your server.
   • serverUrl stores the URL to your SystemLink Server.
   • datafinderName stores the name of the DataFinder Instance that you wish to connect to.
   • userPwd stores the username and password for your SystemLink log in.

serverUrl = "http://my-server-name"
datafinderName = "MyDataFinder"
userPwd = b"username:password"

7. Next, enter the following code to define an authorization string, HTTP request header, and HTTP request data.
   • authStrValue takes the userPwd variable and encodes it appropriately for the HTTP request.
   • headers is a dictionary that contains the required headers for the HTTP request.
   • data is a string formatted to contain information about the DataFinder you wish to connect to.

authStrValue = 'Basic %s' % b64encode(userPwd).decode("ascii")

headers = {
	'accept': 'application/x-asamods+json',
	'authorization': authStrValue,
	'Content-Type': 'application/x-asamods+json'
}

data = "{\"variables\":{\"$URL\":{\"stringArray\":{\"values\":[\"corbaname::#" + datafinderName + ".ASAM-ODS\"]}}}}"

8. Then, enter the below to create a session, which returns a URL that must be in front of all further HTTP calls.
   • response creates a session.
   • sessionUrl stores the URL for future calls.

response = requests.post(serverUrl + 'ni/asam/ods', headers=headers, data=data)
sessionUrl = response.headers["Location"]

9. Now that a HTTP session has been established, HTTP POST requests can be made to retrieve specific file data in the DataFinder.
   
   • Refer to the relevant sections in this article, depending on if you wish to retrieve root file, group or channel information.
10. IMPORTANT: Once all HTTP requests have been made, close the HTTP session using the below code.

response.close()

Retrieving File Metadata

To retrieve a list of files and their root metadata, a JSON string formatted as "{\"test\":{\"name\":\"*\"}}" can be used with the /ni/data-read HTTP POST request. Here, the test key maps to the root of the file. By setting the name key to an asterisk (wildcard), this tells the API to return a list of all files that have been indexed by the DataFinder.

The steps below demonstrate how to obtain root metadata such as the file names, IDs, descriptions, locations and creation dates from this HTTP request.

1. Ensure that there is an active HTTP session by following the Connecting to a DataFinder Instance section above.
2. Modify the data and headers variables defined previously to retrieve a list of root metadata.

• • data is changed so that root level metadata is returned from the request. For detailed information on this string, navigate to <your-server-URL>/systemlink-tdm-offline-doc/DataNavigatorDataService/DataNavigatorDataService.html in a browser.
  • headers now defines that the response is a JSON object.

data = "{\"test\":{\"name\":\"*\"}}"
headers["Content-Type"] = "application/json"

3. Enter the below code to make a HTTP request that uses the new data and headers values to obtain a list of files and file metadata.
   • The HTTP response is stored in responseBody.

response = requests.post(sessionUrl +  "/ni/data-read", headers=headers, data=data)
responseBody = response.json()

4. The responseBody consists of matrices storing the file metadata. The below lines of code show how to access a range of metadata from the matrices.
   
   • To understand the full structure of the responseBody, refer to section 1.3 Meta ods::DataMatrices example at <your-server-URL>/systemlink-tdm-offline-doc/DataNavigatorDataService/DataNavigatorDataService.html

filenames = responseBody["matrices"][0]["columns"][1]["stringArray"]["values"]
fileIDs = responseBody["matrices"][0]["columns"][0]["longlongArray"]["values"]
filedescriptions = responseBody["matrices"][0]["columns"][4]["stringArray"]["values"]
filelocations = responseBody["matrices"][0]["columns"][7]["stringArray"]["values"]
creationdate = responseBody["matrices"][0]["columns"][9]["stringArray"]["values"]

print("FILE NAMES:", filenames)
print("FILE IDs:", fileIDs)
print("DESCRIPTIONS:", filedescriptions)
print("LOCATIONS:", filelocations)
print("CREATION DATE:", creationdate)

Retrieving Group Metadata

To retrieve group metadata from a specific file, a JSON string formatted as "{\"measurement\":{\"test\":{\"name\":\"<filename>"\"}}}" can be used with the /ni/data-read HTTP POST request. Here, the measurement key maps to a channel group. By setting the measurement to a specific test name (i.e. a specific file), the API will return a list of groups associated with that file.

The steps below demonstrate how to obtain group metadata such as the names, IDs, descriptions, and parent file IDs from this HTTP request.

1. Ensure that there is an active HTTP session by following the Connecting to a DataFinder Instance section above.
2. Modify the data and headers variables defined previously to retrieve a list of group metadata for a specific file.
   • data is changed so that group metadata is returned from the request. For detailed information on this string, navigate to <your-server-URL>/systemlink-tdm-offline-doc/DataNavigatorDataService/DataNavigatorDataService.html in a browser.
   • headers now defines that the response is a JSON object.

currentfile = "Your_File_Name.tdm"

data = "{\"measurement\":{\"test\":{\"name\":\"" + currentfile + "\"}}}"
headers["Content-Type"] = "application/json"

3. Enter the below code to make a HTTP request that uses the new data and headers values to obtain a list of files and file metadata.
   • The HTTP response is stored in responseBody.

response = requests.post(sessionUrl + "/ni/data-read", headers=headers, data=data)
responseBody = response.json()

4. The responseBody consists of matrices storing the group metadata. The below lines of code show how to access a range of metadata from the matrices.
   • For more examples of retrieving group metadata, refer to section 4.8 Getting Groups of files at <your-server-URL>/systemlink-tdm-offline-doc/DataNavigatorDataService/DataNavigatorDataService.html

groupnames = responseBody["matrices"][0]["columns"][1]["stringArray"]["values"]
ids = responseBody["matrices"][0]["columns"][0]["longlongArray"]["values"]
descriptions = responseBody["matrices"][0]["columns"][3]["stringArray"]["values"]
parentfileID = responseBody["matrices"][0]["columns"][4]["longlongArray"]["values"]

print("GROUP NAMES:", groupnames)
print("IDs:", ids)
print("DESCRIPTIONS:", descriptions)
print("PPARENT FILE IDS:", parentfileID)

Retrieving Channel Metadata

To retrieve channel metadata from a specific file, a JSON string formatted as "{ \"measurementquantity\": {\"measurement\": <Parent Group ID>}, \"$attributes\": { \"*\" }}" can be used with the /ni/data-read HTTP POST request. Here, the measurementquantity key maps to a channel. By setting the measurement to a specific group ID, the API will return a list of channels associated with that group.

The attributes key defines which properties to be returned. By setting this to an asterisk (wildcard), the HTTP request returns all available channel properties.

The steps below demonstrate how to obtain channel metadata such as the names, IDs, descriptions, units, minimums, maximums and parent group ID from this HTTP request.

1. Ensure that there is an active HTTP session by following the Connecting to a DataFinder Instance section above.
2. Modify the data and headers variables defined previously to retrieve a list of channel metadata for a specific group.
   • data is changed so that all channel properties within a specific group are returned from the request. If you are unsure where to find the group ID, refer to the Retrieving Group Metadata section above.
   • headers now defines that the response is a JSON object.

groupID = 0
data = "{ \"measurementquantity\": {\"measurement\": " +  groupID +"}, \"$attributes\": { \"*\" }}"
headers["Content-Type"] = "application/json"

3. [OPTIONAL] With the code in the previous step, the HTTP request will return all channel properties since the attributes key was set to an asterisk. To retrieve specific properties instead, the attributes key can identify specific properties to return, as shown below.

groupID = "4"
data = "{ \"measurementquantity\": {\"measurement\": " +  groupID +"}, \"$attributes\": { \"id\":1, \"name\":1 }}"

3. Enter the below code to make a HTTP request that uses the new data and headers values to obtain a list of files and file metadata.
   • The HTTP response is stored in responseBody.

response = requests.post(sessionUrl + "/ni/data-read", headers=headers, data=data)
responseBody = response.json()

4. The responseBody consists of matrices storing channel metadata. The below lines of code show how to access a range of metadata from the matrices.
   • For more examples of retrieving channel metadata, refer to section 4.9 Getting Channels of files at <your-server-URL>/systemlink-tdm-offline-doc/DataNavigatorDataService/DataNavigatorDataService.html

channelnames = responseBody["matrices"][0]["columns"][1]["stringArray"]["values"]
ids = responseBody["matrices"][0]["columns"][0]["longlongArray"]["values"]
descriptions = responseBody["matrices"][0]["columns"][3]["stringArray"]["values"]
units = responseBody["matrices"][0]["columns"][4]["stringArray"]["values"]
minimums = responseBody["matrices"][0]["columns"][6]["doubleArray"]["values"]
maximums = responseBody["matrices"][0]["columns"][7]["doubleArray"]["values"]
parentgroupID = responseBody["matrices"][0]["columns"][8]["longlongArray"]["values"]

print("CHANNEL NAMES:", channelnames)
print("IDs:", ids)
print("DESCRIPTIONS:", descriptions)
print("UNITS:", units)
print("MINIMUMS:", minimums)
print("MAXIMUMS:", maximums)
print("PARENT GROUP IDS:", parentgroupID)

Retrieving Channel Values

To retrieve channel values from a specific file, a JSON string formatted as "{\"localcolumn\":<channel ID>,\"$attributes\":{\"id\":1,\"values\":1},\"$options\":{\"$valskip\":0,\"$vallimit\":10000}}" can be used with the /ni/data-read HTTP POST request. Here, the localcolumn key maps to the ID of a specific channel. The attributes key defines that both the channel names and values will be returned.

The steps below demonstrate how to obtain channel values from this HTTP request.

1. Ensure that there is an active HTTP session by following the Connecting to a DataFinder Instance section above.
2. Modify the data and headers variables defined previously to retrieve a list of values for a specific channel.
   • data is changed so that up to 10,000 values for the channel with the specified ID will be returned. If you are unsure where to find the channel ID, refer to the Retrieving Channel Metadata section above.
   • headers now defines that the response is a JSON object.

channelID = "36"
data = "{\"localcolumn\":" + channelID + ",\"$attributes\":{\"id\":1,\"values\":1},\"$options\":{\"$valskip\":0,\"$vallimit\":10000}}"
headers["Content-Type"] = "application/json"

3. Enter the below code to make a HTTP request that uses the new data and headers values to obtain a list of files and file metadata.
   • The HTTP response is stored in responseBody.

response = requests.post(sessionUrl + "/ni/data-read", headers=headers, data=data)
responseBody = response.json()

4. The responseBody consists of matrices storing channel values. The below lines of code show how to access this array of values.
   • For more examples of retrieving channel values, refer to section 5.2 Retrieving values of a channel at <your-server-URL>/systemlink-tdm-offline-doc/DataNavigatorDataService/DataNavigatorDataService.html

channelvalues = responseBody["matrices"][0]["columns"][1]["unknownArrays"]["values"][0]["doubleArray"]["values"]
print("CHANNEL VALUES:", channelvalues)

An example Jupyter Notebook has been attached to demonstrate each of the sections described in this article.

To use this example, follow the instructions below:

1. Download the attached Query DataFinder Example.zip file onto your SystemLink Server PC.
2. Unzip the contents.
3. Copy the Query DataFinder Example.ipynb file into C:\ProgramData\National Instruments\Skyline\JupyterHub\notebooks\_shared\reports
4. In JupyterHub, open the Jupyter Notebook. Modify the code to connect to your SystemLink Server and identify files, groups and channels indexed within your DataFinder Instance.