Azure Data Explorer

Processing a ProgressiveDataSet from Microsoft.Azure.Kusto.Data

ICslQueryProvider’s ExecuteQueryV2Async method is a powerful way to obtain streaming results from Azure Data Explorer (ADX) as well as obtaining performance metrics about the queries being executed. Below is a walkthrough of processing the frames returned by a ProgressiveDataSet.

Getting the frames

The frames are access via the IEnumerator<ProgressiveDataSetFrame> that is returned from calling ProgressiveDataSet.GetFrames().

var frames = progressiveDataSet.GetFrames();
var tablesById = new Dictionary<int, DataTable>();
var tableKindsById = new Dictionary<int, WellKnownDataSet>();

while (frames.MoveNext())
{
    var currentFrame = frames.Current;
}

Determining the Frame Type

Every ProgressiveDataSetFrame has a FrameType property. This provides a hint for the actual type of the frame.

switch (currentFrame.FrameType)
{
    case FrameType.DataSetHeader:
    ...
}

Data Set Header

The data set header frame contains informational properties about the data set version and if it is progressive.

case FrameType.DataSetHeader:
    var frameEx = (ProgressiveDataSetHeaderFrame)currentFrame;
    Trace.WriteLine($"DataSet/HeaderFrame: Version={frameEx.Version}, IsProgressive={frameEx.IsProgressive}");
    break;

Table Header

The table header frame contains the ID, name, kind and schema for a table being returned from ADX. The schema and kind should be cached by ID so that they can be associated with the table rows later.

case FrameType.TableHeader:
    var frameEx = (ProgressiveDataSetDataTableSchemaFrame)currentFrame;
    var dataTable = frameEx.ToEmptyDataTable();
    tablesById.Add(frameEx.TableId, dataTable);
    tableKindsById.Add(frameEx.TableId,frameEx.TableKind);
    Trace.WriteLine($"DataTable/SchemaFrame: TableId={frameEx.TableId}, TableName={frameEx.TableName}, TableKind={frameEx.TableKind}");
    break;

Table Fragment

The table fragment frame contains the table ID, field count, frame subtype, and a subset of data for the associated data table. The records for the table can be accessed via the ProgressiveDataSetDataTableFragmentFrame.GetNextRecord method.

case FrameType.TableFragment:
    var frameEx = (ProgressiveDataSetDataTableFragmentFrame)currentFrame;
    var dataTable = tablesById[frameEx.TableId];
    var record = new object[frameEx.FieldCount];
    while (frameEx.GetNextRecord(record))
    {
         dataTable.Rows.Add(record);
    }
    Trace.WriteLine($"DataTable/FragmentFrame: TableId={frameEx.TableId}, FieldCount={frameEx.FieldCount}, FrameSubType={frameEx.FrameSubType}");
    break;

Table Completion

The table completion frame indicates that the end of a table has been reached. It contains the Table ID and the final row count.

case FrameType.TableCompletion:
    var frameEx = (ProgressiveDataSetTableCompletionFrame)currentFrame;
    Trace.WriteLine($"DataTable/TableCompletionFrame: TableId={frameEx.TableId}, RowCount={frameEx.RowCount}");
    break;

Table Progress

The table progress frame is periodically sent to indicate the progress of sending a progressive table. It contains the Table ID and progress value.

case FrameType.TableProgress:
    var frameEx = (ProgressiveDataSetTableProgressFrame)currentFrame;
    Trace.WriteLine($"DataTable/TableProgressFrame: TableId={frameEx.TableId}, TableProgress={frameEx.TableProgress}");
    break;

Data Table

The data table frame contains an entire table. It also contains the Table ID, name, and kind. The frame provides an IDataReader for obtaining the table schema and data. The IDataReader must be consumed.

case FrameType.DataTable:
    var frameEx = (ProgressiveDataSetDataTableFrame)currentFrame;
    var schema = frameEx.TableData.GetSchemaTable()
    var dataTable = new DataTable(frameEx.TableName);
    foreach (var row in schema.Rows.OfType<DataRow>())
    {
        var columnType = row.Field<string?>("ColumnType");
        var columnName = row.Field<string?>("ColumnName");
        var correspondingClrType = CslType.FromCslType(columnType)
             .GetCorrespondingClrType();
        dataTable.Columns.Add(columnName, correspondingClrType);
    }
    dataTable.Load(frameEx.TableData);
    tablesById.Add(frameEx.TableId, dataTable);
    tableKindsById.Add(frameEx.TableId,frameEx.TableKind);
    Trace.WriteLine($"DataTable/DataTableFrame: TableId={frameEx.TableId}, TableName={frameEx.TableName}, TableKind={frameEx.TableKind}");
    break;

Data Set Completion

The data set completion frame indicates that the data set is complete. It indicates if the query was cancelled, if it errored, and what the error was.

case FrameType.DataSetCompletion:
    var frameEx = (ProgressiveDataSetCompletionFrame)currentFrame;
    Trace.WriteLine($"DataSet/CompletionFrame: HasErrors={frameEx.HasErrors}, Cancelled={frameEx.Cancelled}, Exception={frameEx.Exception}");
    if (frameEx.Cancelled)
    {
        throw new TaskCanceledException();
    }
    if (frameEx.HasErrors)
    {
        ExceptionDispatchInfo.Capture(frameEx.Exception).Throw();
    }
    break;

Well Known Data Sets

Each table returned by the ProgressiveDataSet has a WellKnownDataSet value. This value indicates how the data table should be used.

• Primary Results — The main query results returned by the request. Some requests may return more than a single primary result.
• Query Completion Information — Returns information about the query completion. The EventType/EventTypeName columns indicate the type of query completion event. 0/QueryResourceConsumtion events will have performance data in the Payload column as JSON.
• Query Trace Log — ???
• Query Perf Log — ???
• Table of Contents — Provides information about all other tables in the set.
• Query Properties — The @ExtendedProperties table which has visualization and data cursor information.
• Query Plan — ???