Set the Name of a Jupyter Analysis Procedure Report in SystemLink

This article demonstrates how to programmatically set the name of an output report in a SystemLink Analysis Automation Procedure (ANP) created in JupyterHub.

When the Export Report option is enabled within a Jupyter ANP, the output report(s) is/are given the same name as the ANP and uploaded to the SystemLink File Service. The report(s) can be seen when viewing the details of the Analysis Task:

If your ANP generates many reports (for example, one report per analyzed data file), it may be desirable to programmatically change the report names to match the analyzed data file. This article aims to show how this can be achieved.

To programmatically set the name of an output report in a Jupyter ANP, the following process is required:

1. Create a secondary Jupyter Notebook who's purpose is to:
   1. Contain the visuals required for the output report.
   2. Export itself as a PDF or HTML file with a custom name.
2. Use the main Jupyter ANP to programmatically execute the Notebook from step 1.
3. The Jupyter ANP then uses the SystemLink HTTP API to programmatically upload the HTML/PDF report to the File Service.
4. Output a URL to the report within the File Service from the ANP.

For demonstration purposes, this tutorial uses TDMS files as the data file for analysis.

1. Creating a Secondary Jupyter Notebook to Contain Visuals

This section details how to create a dedicated Notebook to act as the final exported HTML/PDF report. For demonstration purposes, this Notebook plots file-specific data on a graph and exports a HTML file.

1. In a browser, access your SystemLink Server URL.
2. From the navigation menu, select Utilities >> Jupyter.
3. From the Jupyter toolbar, select File >> New >> Notebook.

4. In the Select Kernel pop-up, ensure Python 3 is selected and click Select.
5. Right-click the tab for the current Notebook and select Rename Notebook...

6. In the pop-up window, set the name to Generate Report.ipynb and click Rename.
7. In the first Cell, enter the below code to import the required libraries.
   • This notebook is responsible for generating report visuals and exporting a PDF/HTML. Therefore, any libraries for graphics should be imported (such as matplotlib.pyplot) and nbconvert for the export.

# Import required libraries
import nbconvert
import matplotlib.pyplot as plt

8. The next Cell will be used to define the inputs (or Parameters) of the Notebook. Variables defined here are given a default value, which are automatically updated when the Notebook is called from an external source (i.e. from another Notebook). Declare the following Parameters:
   • export_path used to store the path for the exported report.
   • file_name used to store the name of the exported report.
   • bulk_data used to store an array of data points to plot on a graph. Replace this with any file-specific data that needs to be visualized.

# Parameters here - these values will be updated when this Notebook is called from another Notebook
export_path = "C:\\Users\\Public\\Documents\\Jupyter Analysis Rename Output Report\\"
file_name = "Test Data"
bulk_data = [0,1,2,3,4,5,6,7,8,9,10]

9. Apply the Parameters tag to the current Cell. The Parameters tag allows Jupyter to recognize the variables inside the current Cell as inputs:
   1. With the current Cell selected, click the cog icon on the right-hand side to open the Property Inspector.
   2. Click the tag area underneath Cell Tags and type parameters.
   3. Click Enter to submit.
   4. If successful, the parameters tag appears gray with a tick icon.
   5. Click the cog icon again to close the Property Inspector.

10. In the next Cell use the matplotlib.pyplot library to generate a graph of your data.
    • The code snippet below plots bulk_data on the Y Axis, and labels it as "Amplitude (V)".

# Create Plot
curr_fontsize = 18
axis_lable_fontsize = curr_fontsize - 5

fig, (ax1) = plt.subplots(nrows=1, ncols=1, figsize=(15, 10))
fig.suptitle (file_name, fontsize=curr_fontsize, color='blue')

ax1.grid(True)
ax1.set_xlabel('Samples', fontsize=axis_lable_fontsize)
ax1.set_ylabel('Amplitude [V]', fontsize=axis_lable_fontsize)
ax1.plot(bulk_data)
    
plt.tight_layout()
plt.show()

11. After, use nbconvert to export the graph from the current Notebook (while excluding any code) as a HTML file. The file name and path can be customized, as shown below.
    • Here, current_notebook_path must be set explicitly, rather than using the os library to obtain the path programmatically. This is because the current Notebook's path will change to a temporary location when the Notebook is called from another Notebook.

# Configure the exporter to exclude input (code) cells
current_notebook_path = 'C:\\ProgramData\\National Instruments\\Skyline\\JupyterHub\\notebooks\\admin\\Plot Data.ipynb' # Ensure this path is set correctly for your machine
exporter = nbconvert.HTMLExporter()
exporter.exclude_input = True

# Export the current notebook's visuals to export_path + file_name + .html
output, _ = exporter.from_filename(current_notebook_path)
with open(export_path + file_name + ".html", "w", encoding="utf-8") as f:
    f.write(output)

12. To verify that the Notebook works, run all the Cells from top to bottom. A simple straight line should be plotted on the graph. A HTML report with the graph should be saved to export_path.

13. Save and close the Notebook.
    • A completed version of this Notebook (called Generate Report.ipynb) has been attached to the bottom of this article.

2. Creating the Main Jupyter ANP

This section details the steps necessary to read measurements from a file, execute the Notebook created in section 1 to plot the measurements, and upload the HTML report to the SystemLink File Service.

1. From the Jupyter sidebar, select the fx tab and click the + icon to create a new Analysis Procedure.

2. In the Edit Procedure window:
   • Set Data Source to whichever DataFinder Instance this ANP should execute on.
   • Optionally set Search Query, if you wish to filter which data files the ANP should execute on.

3. Click the pencil icon in the Edit Procedure window, and configure the following:
   • Name should be Customize Report Name.
   • Description should be a description of the ANP.
   • Workspace can be left as Default, unless your DataFinder Instance and data files live in a custom Workspace.
   • Execution Mode should be set to Parallel. This option will generate 1 report per analyzed data file, which is suited for the purposes of this exercise.
   • Search Query must be set to a valid search when using Parallel Execution Mode.
   • Ensure that Export Report is disabled, since we are exporting our report using dedicated code.

4. Scroll to the bottom of the configuration window and click Save.
5. From the Analysis Procedures window, double click Customize Report Name to open the Notebook.
6. In the first Cell, enter the code below to import the required libraries:
   1. requests is required to make HTTP requests to the SystemLink File Service.
   2. b64encode is required to encode usernames and passwords.
   3. scrapbook is required to send outputs from the Notebook into SystemLink.
   4. papermill is required to programmatically execute other Notebooks.
   5. json is required to convert HTTP json string responses to dictionaries.
   6. systemlink.clients.nitdmreader is required to read metadata from data files.

# Import required libraries
import requests
from base64 import b64encode
import scrapbook as sb
import papermill as pm
import json

import systemlink.clients.nitdmreader as tdmreader
metadata_api = tdmreader.MetadataApi()
data_api = tdmreader.DataApi()

7. [OPTIONAL] Read metadata from the currently analyzed file. Refer to Reading SystemLink DataFinder File Information in a Jupyter Notebook for a tutorial.
   • Note: If skipping this step, ensure that you have defined a list called bulk_data that stores values to plot on a graph, and a string called data_file_name that stores the name of your output report.
   • The code snippet below is used to retrieve the current data file's file name, and measurements from the 1st data channel. These will be used to customize the contents and name of the final HTML report.

# Define function that retrieves properties from file
def get_property(element, property_name):
    """Gets a property of the given element.
    
    The element can be a file, channel group, or channel.
    
    Args:
        element: Element to get the property from.
        property_name: Name of the property to get.
        
    Returns:
        The according property of the element or ``None`` if the property doesn't exist.
    """
    return next((e.value for e in element.properties.properties if e.name == property_name), None)


# Retrieve datalinks and file properties
data_links = ni_analysis_automation["data_links"]
file_ids = [d["fileId"] for d in data_links]
file_ids = file_ids
file_infos = await metadata_api.get_multiple_file_info(tdmreader.FileList(file_ids))
file_info = file_infos[0]
file_id = data_links[0]['fileId']
group = file_info.groups[0]

# Store file name - to be passed to the secondary Jupyter Notebook
data_file_name = get_property(file_info, "name").split('.')[0]


# Retrieve measurements from the 1st channel in the file
desired_channel = group.channels[0]
channel_specifier = tdmreader.OneChannelSpecifier(
    file_id=file_id, 
    group_name=group.name, 
    channel_name=desired_channel.name)
        
chn_specs = [channel_specifier]
xy_chns = tdmreader.ChannelSpecificationsXyChannels(y_channels=chn_specs)
channel_specs = tdmreader.ChannelSpecifications(xy_channels=[xy_chns])
query = tdmreader.QueryDataSpecifier(channel_specs)
data = await data_api.query_data(query)
        
y_channels = data.data[0].y
values = list(map(lambda c: c.numeric_data, y_channels))

# Store measurements - to be passed to the secondary Jupyter Notebook
bulk_data = values[0]

8. Programmatically execute the Generate Report.ipynb Notebook, making sure to pass the data_file_name and bulk_data as inputs.
   • Here, a variable report_path is set, which stores the path to export the final HTML report to. Jupyter requires full read and write access to this path.
   • The execute_notebook function programmatically executes another Jupyter Notebook:
     • The 1st parameter is the path of the Notebook to execute.
     • The 2nd parameter is the path of the Notebook to store the execution's output (usually the current Notebook).
     • The 3rd parameter defines the inputs to pass to the Notebook. Here, we pass the report_path, data_file_name, and bulk_data.

# Execute notebook to generate report
report_path = "C:\\Users\\Public\\Documents\\Jupyter Analysis Rename Output Report\\" # Set this to your export path (Jupyter requires full read/write access)
notebook_path = report_path + "Customize Report Name.ipynb" # Current notebook's path (Jupyter requires full read/write access)

pm.execute_notebook(
   'C:\\ProgramData\\National Instruments\\Skyline\\JupyterHub\\notebooks\\admin\\Generate Report.ipynb',
   notebook_path,
   parameters=dict(export_path=report_path, file_name=data_file_name, bulk_data=bulk_data)
)

9. Upload the generated HTML report to the SystemLink File Service using HTTP:
   • The code snippet below uploads the HTML report to the SystemLink File Service using the HTTP API. Using HTTP requires that the authorization header is set, and the file data (read in binary format) is attached as the HTTP multipart/form data.
   • The HTTP request will respond with a unique file ID if the upload is successful.
   • A URL to the uploaded file can be outputted from the Notebook using sb.glue().

# Upload file to File Service

# Set to your server
server_url = "<your server URL and port>"
api_url = "/nifile/v1/service-groups/Default/upload-files?workspace="

# Set to your user account credentials
user_pwd = b"<your username>:<your password>"

#Supply workspace ID - https://knowledge.ni.com/KnowledgeArticleDetails?id=kA03q000000xB2uCAE for tutorial
workspace = '<your workspace ID>'

auth_str_value = 'Basic %s' % b64encode(user_pwd).decode("ascii")

headers = {
    'authorization': auth_str_value
}

# Read HTML file contents in binary and pass it as the multipart/form data for the HTTP request
with open(report_path + data_file_name + ".html", "rb") as file:
    files = {'file': file}
    response = requests.post(server_url + api_url + workspace, files=files, headers=headers)

# read response to get file ID in the File Service
response_text = json.loads(response.text)
file_id = response_text["uri"].split('/')[6]
file_viewer_path = server_url + "#fileviewer/file/"

# Attach a URL to the file as an output of this analysis
sb.glue("Report", file_viewer_path + file_id)

# Close the HTTP connection
response.close()

10. Save the Notebook. A completed version of this Notebook (Customize Report Name.ipynb) is attached to the bottom of this article.
11. From the Jupyter menu, select File >> Download to save a copy of the ANP.
12. Place the downloaded copy in the report_path location (feel free to change this location in the code, this is just an example of a folder where Jupyter has full read/write permissions).
13. In the SystemLink UI, navigate to Measurement Data Analysis >> Analysis Automation.
14. On the Procedures tab, your Customize Report Name procedure should be automatically uploaded.

15. On the Tasks tab, create a new Task to execute your procedure. For testing purposes, it may be easiest to create a Manual task.
16. Execute the task.
17. If successful, the customized report name for each processed file can be viewed by clicking Show Details.

18. Copying any of the report URLs in a new tab will take you to the location of the file within the SystemLink File Service.