Skip to content

reference-azdata-bdc-spark-batch.md

Latest commit

 

History

History
236 lines (211 loc) · 8.03 KB

reference-azdata-bdc-spark-batch.md

File metadata and controls

236 lines (211 loc) · 8.03 KB
title azdata bdc spark batch reference
titleSuffix SQL Server Big Data Clusters
description Reference article for azdata bdc spark batch commands.
author MikeRayMSFT
ms.author mikeray
ms.reviewer seanw
ms.date 10/05/2021
ms.prod sql
ms.technology big-data-cluster
ms.topic reference

azdata bdc spark batch

Applies to [!INCLUDE azure-data-cli-azdata]

The following article provides reference for the sql commands in the azdata tool. For more information about other azdata commands, see azdata reference

Commands

Command Description
azdata bdc spark batch create Create a new Spark batch.
azdata bdc spark batch list List all the batches in Spark.
azdata bdc spark batch info Get information about an active Spark batch.
azdata bdc spark batch log Get execution logs for a Spark batch.
azdata bdc spark batch state Get execution state for a Spark batch.
azdata bdc spark batch delete Delete a Spark batch.

azdata bdc spark batch create

This creates a new batch Spark job that executes the provided code.

azdata bdc spark batch create --file -f 
                              [--class-name -c]  
                              
[--arguments -a]  
                              
[--jar-files -j]  
                              
[--py-files -p]  
                              
[--files]  
                              
[--driver-memory]  
                              
[--driver-cores]  
                              
[--executor-memory]  
                              
[--executor-cores]  
                              
[--executor-count]  
                              
[--archives]  
                              
[--queue -q]  
                              
[--name -n]  
                              
[--config]

Examples

Create a new Spark batch.

azdata spark batch create --code "2+2"

Required Parameters

--file -f

Path to file to execute.

Optional Parameters

--class-name -c

Name of the class to execute when passing in one or more jar files.

--arguments -a

List of arguments. To pass in a list JSON encode the values. Example: '["entry1", "entry2"]'.

--jar-files -j

List of jar file paths. To pass in a list JSON encode the values. Example: '["entry1", "entry2"]'.

--py-files -p

List of python file paths. To pass in a list JSON encode the values. Example: '["entry1", "entry2"]'.

--files

List of file paths. To pass in a list JSON encode the values. Example: '["entry1", "entry2"]'.

--driver-memory

Amount of memory to allocate to the driver. Specify units as part of value. Example 512M or 2G.

--driver-cores

Amount of CPU cores to allocate to the driver.

--executor-memory

Amount of memory to allocate to the executor. Specify units as part of value. Example 512M or 2G.

--executor-cores

Amount of CPU cores to allocate to the executor.

--executor-count

Number of instances of the executor to run.

--archives

List of archives paths. To pass in a list JSON encode the values. Example: '["entry1", "entry2"]'.

--queue -q

Name of the Spark queue to execute the session in.

--name -n

Name of the Spark session.

--config

List of name value pairs containing Spark configuration values. Encoded as JSON dictionary. Example: '{"name":"value", "name2":"value2"}'.

Global Arguments

--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--output -o

Output format. Allowed values: json, jsonc, table, tsv. Default: json.

--query -q

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

azdata bdc spark batch list

List all the batches in Spark.

azdata bdc spark batch list

Examples

List all the active batches.

azdata spark batch list

Global Arguments

--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--output -o

Output format. Allowed values: json, jsonc, table, tsv. Default: json.

--query -q

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

azdata bdc spark batch info

This gets the information for an Spark batch with the given ID. The batch id is returned from 'spark batch create'.

azdata bdc spark batch info --batch-id -i

Examples

Get batch info for batch with ID of 0.

azdata spark batch info --batch-id 0

Required Parameters

--batch-id -i

Spark batch ID number.

Global Arguments

--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--output -o

Output format. Allowed values: json, jsonc, table, tsv. Default: json.

--query -q

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

azdata bdc spark batch log

This gets the batch log entries for a Spark batch with the given ID. The batch id is returned from 'spark batch create'.

azdata bdc spark batch log --batch-id -i

Examples

Get batch log for batch with ID of 0.

azdata spark batch log --batch-id 0

Required Parameters

--batch-id -i

Spark batch ID number.

Global Arguments

--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--output -o

Output format. Allowed values: json, jsonc, table, tsv. Default: json.

--query -q

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

azdata bdc spark batch state

This gets the batch state for a Spark batch with the given ID. The batch id is returned from 'spark batch create'.

azdata bdc spark batch state --batch-id -i

Examples

Get batch state for batch with ID of 0.

azdata spark batch state --batch-id 0

Required Parameters

--batch-id -i

Spark batch ID number.

Global Arguments

--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--output -o

Output format. Allowed values: json, jsonc, table, tsv. Default: json.

--query -q

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

azdata bdc spark batch delete

This deletes a Spark batch. The batch id is returned from 'spark batch create'.

azdata bdc spark batch delete --batch-id -i

Examples

Delete a batch.

azdata spark batch delete --batch-id 0

Required Parameters

--batch-id -i

Spark batch ID number.

Global Arguments

--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--output -o

Output format. Allowed values: json, jsonc, table, tsv. Default: json.

--query -q

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Next steps

For more information about other azdata commands, see azdata reference.

For more information about how to install the azdata tool, see Install azdata.