Spark Job Definition

Warning

The functions are not fully tested yet. Use with caution. Please report any issues to the GitHub repository.

spark_job_definition_create

spark_job_definition_create(workspace_id: str, display_name: str, spark_job_definition_path: str, description: str = None, await_lro: bool = None, timeout: int = 60 * 5, preview: bool = True) -> requests.Response

Create a spark job definition.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to create the spark job definition in.	required
display_name	str	The display name of the spark job definition.	required
spark_job_definition_path	str	The path to the spark job definition to load content from.	required
description	str | None	The description of the spark job definition.	None
await_lro	bool | None	Whether to await the long running operation.	None
timeout	int	Timeout for the long running operation (seconds). Defaults to 5 minutes.	60 * 5
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.

spark_job_definition_get

spark_job_definition_get(workspace_id: str, spark_job_definition_id: str, preview: bool = True) -> requests.Response

Get a spark job definition.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to get the spark job definition from.	required
spark_job_definition_id	str	The id of the spark job definition to get.	required
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.

spark_job_definition_list

spark_job_definition_list(workspace_id: str, continuation_token: str = None, preview: bool = True) -> requests.Response

List spark job definitions for a workspace.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to list spark job definitions for.	required
continuation_token	str | None	A token for retrieving the next page of results.	None
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.

spark_job_definition_update

spark_job_definition_update(workspace_id: str, spark_job_definition_id: str, display_name: str = None, description: str = None, preview: bool = True) -> requests.Response

Update a spark job definition.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to update.	required
spark_job_definition_id	str	The id of the spark job definition to update.	required
display_name	str | None	The display name of the spark job definition.	None
description	str | None	The description of the spark job definition.	None
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.

spark_job_definition_delete

spark_job_definition_delete(workspace_id: str, spark_job_definition_id: str, preview: bool = True) -> requests.Response

Delete a spark job definition.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to delete.	required
spark_job_definition_id	str	The id of the spark job definition to delete.	required
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.

spark_job_definition_get_definition

spark_job_definition_get_definition(workspace_id: str, spark_job_definition_id: str, await_lro: bool = None, timeout: int = 60 * 5, preview: bool = True) -> requests.Response

Get the definition of a spark job definition.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to get the spark job definition definition from.	required
spark_job_definition_id	str	The id of the spark job definition to get the definition from.	required
await_lro	bool | None	Whether to await the long running operation.	None
timeout	int	Timeout for the long running operation (seconds). Defaults to 5 minutes.	60 * 5
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.

spark_job_definition_update_definition

spark_job_definition_update_definition(workspace_id: str, spark_job_definition_id: str, spark_job_definition_path: str, update_metadata: bool = None, await_lro: bool = None, timeout: int = 60 * 5, preview: bool = True) -> requests.Response

Update the definition of a spark job definition.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to update.	required
spark_job_definition_id	str	The id of the spark job definition to update.	required
spark_job_definition_path	str	The path to the spark job definition to load content from.	required
update_metadata	bool | None	When set to true, the item's metadata is updated using the metadata in the .platform file.	None
await_lro	bool | None	Whether to await the long running operation.	None
timeout	int	Timeout for the long running operation (seconds). Defaults to 5 minutes.	60 * 5
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.

spark_job_definition_run_background_job

spark_job_definition_run_background_job(workspace_id: str, spark_job_definition_id: str, job_type: str, await_lro: bool = None, timeout: int = 60 * 5, preview: bool = True) -> requests.Response

Run on-demand spark job instance.

Parameters:

Name	Type	Description	Default
workspace_id	str	The id of the workspace to create a job for.	required
spark_job_definition_id	str	The id of the spark job definition to create a job for.	required
job_type	str	The type of the job to create. Must be "sparkJob".	required
await_lro	bool | None	Whether to await the long running operation.	None
timeout	int	Timeout for the long running operation (seconds). Defaults to 5 minutes.	60 * 5
preview	bool	Whether to preview the request. You will be asked to confirm the request before it is executed. Defaults to True.	True

Returns:

Type	Description
Response	The response from the request.