support.huaweicloud.com · 12 Permissions Policies and Supported Actions This chapter describes...

Data Lake Insight

API Reference

Issue 01

Date 2020-04-13

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2020-04-13) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 Before You Start....................................................................................................................... 11.1 Overview.................................................................................................................................................................................... 11.2 API Calling..................................................................................................................................................................................11.3 Endpoints....................................................................................................................................................................................11.4 Basic Concepts.......................................................................................................................................................................... 2

2 API Overview............................................................................................................................ 4

3 Calling APIs............................................................................................................................... 63.1 Making an API Request......................................................................................................................................................... 63.2 Authentication....................................................................................................................................................................... 103.3 Returned Values.................................................................................................................................................................... 11

4 Permission-related APIs....................................................................................................... 134.1 Granting Users with the Queue Usage Permission................................................................................................... 134.2 Querying Queue Users........................................................................................................................................................ 154.3 Granting Users with the Data Usage Permission...................................................................................................... 174.4 Querying Database Users.................................................................................................................................................. 194.5 Querying Table Users.......................................................................................................................................................... 214.6 Querying a User's Table Permissions............................................................................................................................. 23

5 Queue-related APIs...............................................................................................................265.1 Creating a Queue.................................................................................................................................................................. 265.2 Deleting a Queue.................................................................................................................................................................. 285.3 Querying All Queues............................................................................................................................................................285.4 Modifying CIDR......................................................................................................................................................................315.5 Restarting, Scaling Out, and Scaling In Queues........................................................................................................ 335.6 Viewing Details of a Queue.............................................................................................................................................. 35

6 APIs Related to SQL Jobs..................................................................................................... 396.1 Database-related APIs......................................................................................................................................................... 396.1.1 Creating a Database......................................................................................................................................................... 396.1.2 Deleting a Database......................................................................................................................................................... 416.1.3 Querying All Databases...................................................................................................................................................436.1.4 Modifying a Database Owner....................................................................................................................................... 446.2 Table-related APIs................................................................................................................................................................. 46

Data Lake InsightAPI Reference Contents

Issue 01 (2020-04-13) Copyright © Huawei Technologies Co., Ltd. ii

6.2.1 Creating a Table................................................................................................................................................................. 466.2.2 Deleting a Table................................................................................................................................................................. 496.2.3 Querying All Tables (Recommended)........................................................................................................................ 506.2.4 Querying All Tables (Discarded).................................................................................................................................. 536.2.5 Describing the Table Information................................................................................................................................ 556.2.6 Previewing Table Content............................................................................................................................................... 596.2.7 Obtaining the Partition List........................................................................................................................................... 606.3 Job-related APIs..................................................................................................................................................................... 636.3.1 Importing Data................................................................................................................................................................... 636.3.2 Exporting Data................................................................................................................................................................... 676.3.3 Submitting a SQL Job (Recommended).................................................................................................................... 706.3.4 Submitting a SQL Job (Deprecated)........................................................................................................................... 736.3.5 Canceling a Job (Recommended)................................................................................................................................ 776.3.6 Canceling a Job (Discarded).......................................................................................................................................... 786.3.7 Querying All Jobs............................................................................................................................................................... 796.3.8 Querying the Job Execution Result (Recommended)........................................................................................... 826.3.9 Querying the Job Execution Result (Discarded)..................................................................................................... 846.3.10 Querying Job Status....................................................................................................................................................... 876.3.11 Querying Job Details...................................................................................................................................................... 896.3.12 Checking SQL Syntax..................................................................................................................................................... 916.3.13 Exporting Query Results............................................................................................................................................... 936.4 Queue-related APIs (Discarded)...................................................................................................................................... 956.4.1 Creating a Queue (Discarded)...................................................................................................................................... 966.4.2 Deleting a Queue (Discarded)...................................................................................................................................... 976.4.3 Querying the Queue List (Discarded)........................................................................................................................ 99

7 APIs Related to Stream Jobs.............................................................................................1017.1 Granting OBS Permissions to DLI................................................................................................................................. 1017.2 Creating a SQL Job.............................................................................................................................................................1037.3 Updating a SQL Job........................................................................................................................................................... 1077.4 Creating a Custom Flink Job........................................................................................................................................... 1127.5 Updating a Custom Flink Job......................................................................................................................................... 1157.6 Running Jobs in Batches.................................................................................................................................................. 1197.7 Querying the Job List........................................................................................................................................................ 1217.8 Querying Job Details......................................................................................................................................................... 1297.9 Querying the Job Execution Plan.................................................................................................................................. 1347.10 Querying Job Monitoring Information......................................................................................................................1417.11 Querying the APIG Address of a Job......................................................................................................................... 1447.12 Stopping Jobs in Batches............................................................................................................................................... 1477.13 Deleting a Job................................................................................................................................................................... 1497.14 Deleting Jobs in Batches................................................................................................................................................ 150

8 APIs Related to Spark jobs................................................................................................1538.1 Session-related APIs.......................................................................................................................................................... 153


Issue 01 (2020-04-13) Copyright © Huawei Technologies Co., Ltd. iii

8.1.1 Creating a Session (Recommended)........................................................................................................................ 1538.1.2 Creating a Session (Discarded).................................................................................................................................. 1588.1.3 Canceling a Session........................................................................................................................................................ 1638.1.4 Viewing the Session List (Recommended)............................................................................................................. 1648.1.5 Viewing the Session List (Discarded)....................................................................................................................... 1668.1.6 Viewing the Session Details.........................................................................................................................................1688.1.7 Checking Status of a Session...................................................................................................................................... 1698.1.8 Viewing Session Logs..................................................................................................................................................... 1708.2 Statement-related APIs.................................................................................................................................................... 1728.2.1 Creating a Statement.................................................................................................................................................... 1728.2.2 Canceling Execution of a Statement........................................................................................................................ 1748.2.3 Querying List of Statements....................................................................................................................................... 1758.2.4 View Information About Statements....................................................................................................................... 1778.3 Batch Processing-related APIs........................................................................................................................................ 1788.3.1 Creating a Batch Processing Job................................................................................................................................ 1788.3.2 Canceling a Batch Processing Job..............................................................................................................................1838.3.3 Obtaining the List of Batch Processing Jobs..........................................................................................................1848.3.4 Querying Batch Job Details......................................................................................................................................... 1868.3.5 Querying a Batch Job Status....................................................................................................................................... 1888.3.6 Querying Batch Job Logs.............................................................................................................................................. 1898.4 Resource Package-related APIs......................................................................................................................................1918.4.1 Uploading a Resource Package.................................................................................................................................. 1918.4.2 Deleting Resource Packages........................................................................................................................................1938.4.3 Querying All Resource Packages................................................................................................................................1948.4.4 Querying a Specified Resource Package................................................................................................................. 1968.5 Package Group-related APIs........................................................................................................................................... 1978.5.1 Uploading a Package Group....................................................................................................................................... 1978.5.2 Querying Package Group List..................................................................................................................................... 1998.5.3 Uploading a JAR Package Group............................................................................................................................... 2028.5.4 Uploading a PyFile Package Group...........................................................................................................................2048.5.5 Uploading a File Package Group............................................................................................................................... 2058.5.6 querying Resource Packages in a Group.................................................................................................................2078.5.7 Deleting a Resource Package from a Group..........................................................................................................2088.6 Cluster-related APIs........................................................................................................................................................... 2088.6.1 Creating a Cluster (Discarded)................................................................................................................................... 2098.6.2 Deleting a Cluster........................................................................................................................................................... 2108.6.3 Querying Information of a Specified Cluster (Discarded)................................................................................ 2118.6.4 Querying All Cluster Information (Discarded)...................................................................................................... 213

9 APIs Related to Stream Job Templates.......................................................................... 2159.1 Creating a Template.......................................................................................................................................................... 2159.2 Updating a Template........................................................................................................................................................ 2179.3 Deleting a Template.......................................................................................................................................................... 219


Issue 01 (2020-04-13) Copyright © Huawei Technologies Co., Ltd. iv

9.4 Querying the Template List............................................................................................................................................ 221

10 APIs Related to Basic Datasource Connections......................................................... 22510.1 Creating a Basic Datasource Connection................................................................................................................. 22510.2 Deleting a Basic Datasource Connection................................................................................................................. 22910.3 Querying a Basic Datasource Connection List....................................................................................................... 23110.4 Querying a Basic Datasource Connection............................................................................................................... 232

11 APIs Related to the Enhanced Datasource Connections......................................... 23511.1 Creating an Enhanced Datasource Connection..................................................................................................... 23511.2 Deleting an Enhanced Datasource Connection..................................................................................................... 23711.3 Querying an Enhanced Datasource Connection List........................................................................................... 23911.4 Querying an Enhanced Datasource Connection....................................................................................................24011.5 Binding a Queue...............................................................................................................................................................24311.6 Unbinding a Queue......................................................................................................................................................... 24411.7 Modifying the Host Information.................................................................................................................................246

12 Permissions Policies and Supported Actions.............................................................. 249

13 Appendix............................................................................................................................. 25613.1 Status Codes...................................................................................................................................................................... 25613.2 Error Code........................................................................................................................................................................... 25913.3 Obtaining a Project ID.................................................................................................................................................... 26113.4 Obtaining an Account ID...............................................................................................................................................262

A Change History....................................................................................................................264


Issue 01 (2020-04-13) Copyright © Huawei Technologies Co., Ltd. v

1 Before You Start

1.1 OverviewWelcome to Data Lake Insight API Reference. Data Lake Insight (DLI) is a fullyhosted big data processing and analysis service powered on the Apache Sparkecosystem. Enterprises can use standard SQL or Spark programs to performfederated analysis of heterogeneous data sources and get valuable insights fromdata.

This document describes APIs of DLI, including SQL and Spark queries anddatasource connection. For details about all supported operations, see APIOverview.

Before calling APIs of DLI, you are recommended to get to know the basicconcepts from Service Overview.

1.2 API CallingDLI provides RESTful (Representational State Transfer) APIs, allowing you to useHTTPS to call them. For details, see Making an API Request.

Unless otherwise specified, DLI APIs are synchronous. Specifically, is_success,indicating whether a request is successfully executed, is returned.

If the DLI API is asynchronous, run the API related to Querying Job Statusaccording to the value of job_id in the response message to learn whether the APIis successfully executed.

In addition, DLI provides SDKs in multiple programming languages for you. Fordetails about how to use SDKs, see the HUAWEI CLOUD SDKs.

1.3 EndpointsAn endpoint is the request address for calling an API. Endpoints vary dependingon services and regions. For the endpoints of all services, see Regions andEndpoints

Data Lake InsightAPI Reference 1 Before You Start

Issue 01 (2020-04-13) Copyright © Huawei Technologies Co., Ltd. 1

https://support.huaweicloud.com/en-us/productdesc-dli/dli_07_0001.html

https://developer.huaweicloud.com/en-us/sdk?DLI

https://developer.huaweicloud.com/en-us/endpoint?Batch%20Service

https://developer.huaweicloud.com/en-us/endpoint?Batch%20Service

The following table describes the endpoints of DLI. Select the endpoints based onyour requirements.

Table 1-1 DLI endpoints

RegionName

Region Endpoint

CN North-Beijing4

cn-north-4 dli.cn-north-4.myhuaweicloud.com

CN North-Beijing1

cn-north-1 dli.cn-north-1.myhuaweicloud.com

CN East-Shanghai2

cn-east-2 dli.cn-east-2.myhuaweicloud.com

CN South-Guangzhou

cn-south-1 dli.cn-south-1.myhuaweicloud.com

AP-Bangkok

ap-southeast-2

dli.ap-southeast-2.myhuaweicloud.com

AP-HongKong

ap-southeast-1

dli.ap-southeast-1.myhuaweicloud.com

1.4 Basic Concepts

AccountAn account is created upon successful registration with HUAWEI CLOUD. Theaccount has full access permissions for all of its cloud services and resources. Itcan be used to reset user passwords and grant user permissions. Accounts makepayments and should not be used for daily work. They can create users and usethe users to manage resources and cloud services.

UserAn IAM user is created by an account in IAM to use cloud services. Each IAM userhas its own identity credentials (password and access keys).

An IAM user can view the account ID and user ID on the My Credentials page ofthe management console. API authentication requires information such as theaccount name, username, and password.

RegionA region is a geographic area in which cloud resources are deployed. Availabilityzones (AZs) in the same region can communicate with each other over anintranet, while AZs in different regions are isolated from each other. Deployingcloud resources in different regions can better suit certain user requirements orcomply with local laws or regulations.



https://console.huaweicloud.com/iam/?locale=en-us#/mine/apiCredential

AZAn AZ is a physical region where resources use independent power supplies andnetworks. AZs are physically isolated but interconnected through an internalnetwork. AZs within a region are interconnected using high-speed optical fibers torealize cross-AZ high availability for the system.

ProjectA project corresponds to a HUAWEI CLOUD region. Default projects are defined toa group and have physically isolated resources (including computing, storage, andnetwork resources) across regions. Users can be granted permissions in a defaultproject to access all resources in the region associated with the project. If you needmore refined access control, create subprojects under a default project andpurchase resources in subprojects. Then you can assign users the permissionsrequired to access only the resources in the specific subprojects.

Under My Credentials, you can view the project ID.

Figure 1-1 Project isolating model



https://console.huaweicloud.com/iam/?locale=en-us#/mine/apiCredential

2 API Overview

This section describes the APIs provided by DLI.

Table 2-1 DLI APIs

Type Subtype Description

Queue-related APIs(Recommended)

- You can create, delete, query all queues, modifyqueue CIDRs, restart, scale out, and scale inqueues, and query queue details.

APIs Relatedto SQL Jobs

Database-relatedAPIs

You can create a database, delete a database, viewall databases, and modify database users.

Table-relatedAPIs

You can create, delete, and query tables, describetable information, preview table content, modifytable users, and obtain the partition informationlist.

Job-relatedAPIs

You can import and export data, submitting SQLjobs, canceling jobs, querying all jobs, querying jobresults, querying job status, querying job details,checking SQL syntax, and exporting query results.

Permission-relatedAPIs

Including assigning rights to queues, viewing theusers of queues, assigning rights to data, viewingdatabase users, viewing table users, and viewingtable users.

APIs relatedto Spark jobs

Session-relatedAPIs

You can create a session, cancel a session, view thesession list, view session information, view sessionstatus, and view session logs.

Statement-relatedAPIs

Including creating statements, canceling statementexecution, viewing statement lists, and viewingstatement information.

Data Lake InsightAPI Reference 2 API Overview


Type Subtype Description

BatchProcessing-relatedAPIs

Including creating batch jobs, deleting batch jobs,querying batch job lists, querying batch job details,querying batch job status, and querying batch joblogs.

ResourcePackage-relatedAPIs

Including uploading resource packages, deletingresource packages,querying the resource packagelist, and querying resource packages.

PackageGroup-relatedAPIs

You can upload a group resource, query the groupresource list, upload a group resource in JARformat, upload a pyfile group resource, upload afile type group resource, query a resource packagein a group, and delete a resource package in agroup.

APIs Relatedto BasicDatasourceConnections

- You can create, delete, and query basic datasourceconnections, as well as query the basic datasourceconnection lists.

APIs Relatedto theEnhancedDatasourceConnections

- You can create, delete, and query enhanceddatasource connections, enhanced datasourceconnection lists, and bind and unbind queues.

APIs Relatedto StreamJobs

- You can grant OBS permissions to DLI, create andupdate SQL jobs, create and update user-definedFlink jobs, run jobs in batches, query job lists anddetails, query job execution plans, query jobmonitoring information, query job API (GSMnetwork gateway service access address), stop jobsin batches, delete jobs, and batch delete jobs.

APIs relatedto Streamjobs

- You can create, update, and delete a template, andquery the template list.

Data Lake InsightAPI Reference 2 API Overview


3 Calling APIs

3.1 Making an API RequestThis section describes the structure of a REST API, and uses the IAM API forobtaining a user token as an example to describe how to call an API. Theobtained token is used to authenticate the calling of other APIs.

Request URIA request URI is in the following format:

{URI-scheme} :// {Endpoint} / {resource-path} ? {query-string}

Although a request URI is included in a request header, most programminglanguages or frameworks require the request URI to be separately transmitted,rather than being conveyed in a request message.

● URI-scheme: Protocol used to transmit requests. All APIs use HTTPS.● Endpoint: Domain name or IP address of the server bearing the REST service

endpoint. Different services have different endpoints in different regions.Obtain the value from Regions and Endpoints. Contact the administrator toobtain regions and endpoints. For example, the endpoint of IAM in region CNNorth-Beijing1 is iam.cn-north-1.myhuaweicloud.com.

● resource-path: API access path for performing a specified operation. Obtainthe value from the URI of the API. For example, the resource-path of the APIfor obtaining a user token is /v3/auth/tokens.

● query-string: Query parameter, which is optional. Not all APIs have a queryparameter. Ensure that a question mark (?) is included before a queryparameter that is in the format of "Parameter name=Parameter value". Forexample, ? limit=10 indicates that a maximum of 10 pieces of data is to beviewed.

For example, to obtain the IAM token in region CN North-Beijing1, obtain theendpoint (iam.cn-north-1.myhuaweicloud.com) of this region and the resource-path (/v3/auth/tokens) in the URI of the API, which is used for obtaining a usertoken. Then, assemble these fields as follows:

https://iam.cn-north-1.myhuaweicloud.com/v3/auth/tokens

Data Lake InsightAPI Reference 3 Calling APIs


https://support.huaweicloud.com/en-us/api-iam/iam_30_0001.html

https://developer.huaweicloud.com/en-us/endpoint?all



Figure 3-1 Example URI

To simplify the URI display, each API is provided with only a resource-path and a requestmethod. This is because the URI-scheme value of all APIs is HTTPS, and the endpoints in aregion are the same. Therefore, the two parts are omitted.

Request MethodsHTTP-based request methods, which are also called operations or actions, specifythe type of operations that you are requesting.

● GET: Requests the server to return specified resources.● PUT: Requests the server to update specified resources.● POST: Requests the server to add a resource or perform special operations.● DELETE: Requests a server to delete specified resources, for example, an

object.● HEAD: Requests a server resource header.● PATCH: Requests the server to update the partial content of a specified

resource. If the resource is unavailable, the PATCH method is used to create aresource.

For example, in the URI for obtaining a user token, the request method is POST,and the request is as follows:

POST https://iam.cn-north-1.myhuaweicloud.com/v3/auth/tokens

Request HeaderYou can also add additional fields to a request, such as the fields required by aspecified URI or an HTTP method. For example, add Content-Type that defines arequest body type to request for the authentication information.

Common request headers are as follows:




Table 3-1 Common request headers

Parameter Mandatory Description Example

Content-Type Yes Specifies the requestbody type or format. Thisfield is mandatory and itsdefault value isapplication/json. Forother values, thedescription will beprovided for specific APIs.

application/json

X-Auth-Token Thisparameterismandatoryonly forauthentication usingtokens.

Specifies the user token.It is a response to the APIfor obtaining a usertoken (only this API doesnot requireauthentication).

-

X-Project-ID No Specifies a sub-project ID.It is mandatory in multi-project scenarios toobtain tokens fordifferent projects.

e9993fc787d94b6c886cbaa340f9c0f4

Authorization This headerfield ismandatoryif AK/SKauthentication is in use.

Specifies the signatureauthenticationinformation. The valuecan be obtained from therequest signing result.

-

X-Sdk-Date This headerfield ismandatoryif AK/SKauthentication is in use.

Specifies the time whenthe request is sent. Thetime is inYYYYMMDD'T'HHMMSS'Z' format.The value is the currentGreenwich Mean Time(GMT) of the system.

20150907T101459Z





Parameter Mandatory Description Example

Host This headerfield ismandatoryif AK/SKauthentication is in use.

Specifies the informationabout the requestedserver. The value can beobtained from the URL ofthe service API. The valueis hostname[:port]. Ifthe port number is notspecified, the default portis used. The default portnumber for https is port443.

code.test.comorcode.test.com:443

Content-Length

This field ismandatoryfor POSTand PUTrequests,but mustbe leftblank forGETrequests.

Indicates the length ofthe request body. Theunit is byte.

3495

X-Language No Specifies the requestlanguage, which valuecan be:● zh-cn: Chinese● en-us: English

en-us

In addition to supporting authentication using tokens, the public cloud APIs supportauthentication using AK/SK, which uses SDKs to sign a request. During the signature, theAuthorization (signature authentication) and X-Sdk-Date (time when a request is sent)headers are automatically added in the request.

For more details, see Authentication Using AK/SK.

The API for obtaining a user token does not require authentication. Therefore,this API only requires adding the Content-Type field. The request with the addedContent-Type header is as follows:

POST https://iam.cn-north-1.myhuaweicloud.com/v3/auth/tokens Content-Type: application/json

Request Body

A request body is generally sent in a structured format. It corresponds to Content-Type in the request header and transfers data except for the request header. If therequest body contains Chinese characters, these characters must be coded inUTF-8.




The request body varies according to the APIs. Certain APIs do not require therequest body, such as the GET and DELETE APIs.

For the API of obtaining a user token, obtain the request parameters andparameter description in the API request. The following provides an examplerequest with a body included. Replace username, domainname, ******** (loginpassword), and xxxxxxxxxxxxxxxxxx (project ID) with the actual values. Obtain aproject ID from Regions and Endpoints, for example, cn-north-1.Obtain a projectID from the administrator.

scope specifies where a token takes effect. In the following example, the token takes effectonly on the resources specified by the project ID. You can set scope to an account or aproject under an account. For details, see Obtaining a User Token.

POSThttps: //iam.cn-north-1.myhuaweicloud.com/v3/auth/tokensContent-Type: application/json{ "auth": { "identity": { "methods": [ "password" ], "password": { "user": { "name": "username", "password": "********", "domain": { "name": "domainname" } } } }, "scope": { "project": { "name": "xxxxxxxxxxxxxxxxxx" } } }}

If all data required by a request is available, you can send the request to call anAPI through curl, Postman, or coding. For the API of obtaining a user token, x-subject-token in the response header is the desired user token. Then, you can usethe token to authenticate the calling of other APIs.

3.2 AuthenticationAPI calling is authenticated using either of the following methods:

● Token authentication: Common requests are authenticated using Tokens.● AK/SK authentication: Requests are encrypted using the access key ID and

secret access key (AK/SK pair) to provide higher security. AK/SKauthentication is recommended because it provides higher security than tokenauthentication.

Token AuthenticationA token specifies certain permissions in a computer system. Authentication using atoken adds the token in a request as its header during API calling to obtainpermissions to operate APIs through IAM.




https://developer.huaweicloud.com/en-us/endpoint?all


https://curl.haxx.se/

https://www.getpostman.com/


The validity period of a token is 24 hours. When using a token for authentication, cache itto prevent frequently calling the IAM API.

Making an API Request is an example of IAM calling the API for obtaining theuser token. After obtaining the token, add the X-Auth-Token header in a requestto specify the token when calling other APIs. For example, if the token isABCDEFJ...., add X-Auth-Token: ABCDEFJ.... in a request as follows.

GET https://iam.cn-north-1.myhuaweicloud.com/v3/auth/projects Content-Type: application/json X-Auth-Token: ABCDEFJ....

AK/SK AuthenticationIn authentication using AK/SK, AK/SK is used to sign a request and add thesignature in a request as its header for authentication.

Authentication using AK/SK supports API requests with a body not larger than 12 MB. ForAPI requests with a larger body, authentication using tokens is recommended.

● AK: access key ID, which is a unique identifier associated with a secret accesskey and is used in conjunction with a secret access key to sign requestscryptographically.

● SK: secret access key used in conjunction with an AK to sign requestscryptographically. It identifies a request sender and prevents the request frombeing modified.

In authentication using AK/SK, you can use AK/SK to sign requests based on thesignature algorithm or use a dedicated signature SDK to sign the requests. Fordetails about how to sign requests and use the signature SDK, see API SigningGuide.

The signature SDK only supports signature, which is different from the SDKs provided byservices.

3.3 Returned Values

Status CodeAfter sending a request, you will receive a response, including the status code,response header, and response body.

A status code is a group of digits ranging from 1xx to 5xx. It indicates the status ofa response. For more information, see Status Codes.

If status code 201 is returned for the calling of the API for obtaining a usertoken, the request is successful.

Response HeaderA response header corresponds to a request header, for example, Content-Type.





https://support.huaweicloud.com/en-us/devg-apisign/api-sign-provide.html

https://support.huaweicloud.com/en-us/devg-apisign/api-sign-provide.html



Figure 3-2 shows the response header for the API of obtaining a user token, inwhich x-subject-token is the desired user token. Then, you can use the token toauthenticate the calling of other APIs.

Figure 3-2 Header of the response to the request for obtaining a user token

Response Body

A response body is generally returned in a structured format, corresponding to theContent-Type in the response header, and is used to transfer content other thanthe response header.

The following shows part of the response body for the API to obtain a user token.

{ "token": { "expires_at": "2019-02-13T06:52:13.855000Z", "methods": [ "password" ], "catalog": [ { "endpoints": [ { "region_id": "cn-north-1", ......

If an error occurs during API calling, the system returns an error code and amessage to you. The following shows the format of an error response body:

{ "error_msg": "The format of message is error", "error_code": "AS.0001"}

In the preceding information, error_code is an error code, and error_msgdescribes the error.





4 Permission-related APIs

4.1 Granting Users with the Queue Usage Permission

FunctionThis API is used to share a specific queue with other users. You can grant userswith the permission to use the specified queue or revoke the permission.

The user group containing the authorized user must have the Tenant Guest permission inthe region to which the user group belongs. For details about the Tenant Guest permissionand how to apply for the permission, see Permissions Policies and Creating a User Groupand Assigning Permissions in the Identity and Access Management User Guide.

URI● URI format

PUT /v1.0/{project_id}/queues/user-authorization● Parameter description

Table 4-1 URI parameter

Parameter

Mandatory

Description

project_id

Yes Project ID, which is used for resource isolation. For detailsabout how to obtain its value, see Obtaining a ProjectID.

Data Lake InsightAPI Reference 4 Permission-related APIs



https://support.huaweicloud.com/en-us/usermanual-iam/iam_03_0001.html


Request

Table 4-2 Request parameters

Parameter

Mandatory

Type Description

queue_name

Yes String Name of a queue.

user_name

Yes String Name of the user who is granted with usagepermission on a queue or whose queue usagepermission is revoked or updated.

action Yes String Grants or revokes the permission. The parametervalue can be grant, revoke, or update. Users canperform the update operation only when they havebeen granted with the grant and revokepermissions.● grant: Indicates to grant users with permissions.● revoke: Indicates to revoke permissions.● update: Indicates to clear all the original

permissions and assign the permissions in theprovided permission array.

privileges

Yes Arrayofstrings

List of permissions to be granted, revoked, orupdated. The following permissions are supported:● SUBMIT_JOB: indicates to submit a job.● CANCEL_JOB: indicates to cancel a job.● DROP_QUEUE: indicates to a delete a queue.

NOTEIf the update list is empty, all permissions of the queuegranted to the user are revoked.

Response

Table 4-3 Response parameters

Parameter

Mandatory

Type Description

is_success

No Boolean Indicates whether the request is successfullyexecuted. Value true indicates that the requestis successfully executed.

message No String System prompt. If execution succeeds, theparameter setting may be left blank.



Example● Example request

{ "queue_name": "queue1", "user_name": "tenant2", "action": "grant", "privileges" : ["DROP_QUEUE", "submit_job"]}

● Example response (successful request){ "is_success": true, "message": ""}

If an error occurs when this API is invoked, the system does not return the resultsimilar to the preceding example, but returns the error code and error information. Fordetails, see Error Code.

4.2 Querying Queue Users

FunctionThis API is used to query names of all users who can use a specified queue.

URI● URI format

GET /v1.0/{project_id}/queues/{queue_name}/users● Parameter description

Table 4-4 URI parameters

Parameter

Mandatory

Description

project_id

Yes Project ID, which is used for resource isolation. For detailsabout how to obtain its value, see Obtaining a Project ID.

queue_name

Yes Name of a queue.

RequestNone



Response


Parameter

Mandatory

Type Description

is_success

No Boolean Indicates whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.

message

No String System prompt. If execution succeeds, theparameter setting may be left blank.

queue_name

No String Name of a queue.

privileges

No Array ofstrings

Users who are granted with the permission touse this queue and the permission array towhich users belong.


None

● Example response (successful request){ "is_success": true, "message": "", "privileges": [ { "is_admin": true, "privileges": [ "ALL" ], "user_name": "tenant1" }, { "is_admin": false, "privileges": [ "SUBMIT_JOB" ], "user_name": "user2" } ], "queue_name": "queue1"}




4.3 Granting Users with the Data Usage Permission

Function

This API is used to grant database or table data usage permission to specifiedusers.

The user group containing the authorized user must have the Tenant Guest permission inthe region to which the user group belongs. For details about the Tenant Guest permissionand how to apply for the permission, see Permissions Policies and Creating a User Groupand Assigning Permissions in the Identity and Access Management User Guide.

URI● URI format

PUT /v1.0/{project_id}/user-authorization

● Parameter description


Parameter

Mandatory

Description

project_id


Request


Parameter

Mandatory

Type Description

user_name

Yes String Name of the user who is granted with usagepermission on a queue or whose queue usagepermission is revoked or updated.

action Yes String Grants or revokes the permission. Theparameter value can be grant, revoke, orupdate.NOTE

Users can perform the update operation only whenthey have been granted with the grant and revokepermissions.






Parameter

Mandatory

Type Description

privileges

Yes Array ofobjects

Permission granting information. For details,see Table 4-8.

Table 4-8 privileges parameters

Parameter

Mandatory

Type Description

object Yes String Data objects to be assigned. If they are named● databases.Database name, data in the entire

database will be shared.● databases.Database name.tables.Table name,

data in the specified table will be shared.● databases.Database name.tables.Table

name.columns.Column name, data in thespecified column will be shared.

privileges

Yes Array ofstrings

List of permissions to be granted, revoked, orupdated.NOTE

If Action is Update and the update list is empty, allpermissions of the user in the database or table arerevoked.

Response


Parameter

Mandatory

Type Description

is_success

No Boolean

Indicates whether the request is successfullyexecuted. Value true indicates that the request issuccessfully executed.

message

No String System prompt. If execution succeeds, the parametersetting may be left blank.


{"user_name": "user2","action": "grant",



"privileges": [{"object": "databases.db1.tables.tb2.columns.column1","privileges": ["SELECT"]},{ "object": "databases.db1.tables.tbl", "privileges": ["DROP_TABLE"]},{ "object": "databases.db1", "privileges": ["SELECT"]}]}

● Example response (successful request){ "is_success": true, "message": "" }


4.4 Querying Database Users

FunctionThis API is used query names of all users who have permission to use or access thedatabase.

URI● URI format

GET /v1.0/{project_id}/databases/{database_name}/users● Parameter description


Parameter

Mandatory

Description

project_id

Yes Project ID, which is used for resource isolation. Fordetails about how to obtain its value, see Obtaining aProject ID.

database_name

Yes Name of the database to be queried.

RequestNone



Response


Parameter

Mandatory

Type Description

is_success No Boolean Indicates whether the request issuccessfully executed. Value true indicatesthat the request is successfully executed.


database_name

No String Name of the database to be queried.

privileges No Array ofobjects

Permission information. For details, seeTable 4-12.


Parameter

Mandatory

Type Description

is_admin No Boolean

Indicates whether the database user is anadministrator.

user_name

No String Name of the user who has permission on thecurrent database.

privileges No String Permission of the user on the database.


None

● Example response (successful request){ "is_success": true, "message": "", "database_name": "dsstest", "privileges": [ { "is_admin": true, "privileges": [ "ALL" ], "user_name": "test" }, { "is_admin": false, "privileges": [ "ALTER_TABLE_ADD_PARTITION" ], "user_name": "scuser1"



}, { "is_admin": false, "privileges": [ "CREATE_TABLE" ], "user_name": "scuser2" } ]}


4.5 Querying Table Users

FunctionThis API is used to query users who have permission to access the specified tableor column in the table.

URI● URI format

GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/users● Parameter description


Parameter Mandatory

Description

project_id Yes Project ID, which is used for resource isolation. Fordetails about how to obtain its value, see Obtaininga Project ID.

database_name

Yes Name of the database where the table to bequeried resides.

table_name Yes Name of a table that is to be queried.

RequestNone



Response


Parameter

Mandatory

Type Description

is_success


message


privileges

No Array ofobjects

Permission information. For details, see Table4-15.


Parameter

Mandatory

Type Description

is_admin

No Boolean Indicates whether the table user is an administrator.

object No String Objects on which a user has permission.● If the object is in the format of

databases.Database name.tables.Table name,the user has permission on the database.

● If the object is in the format ofdatabases.Database name.tables.Tablenamecolumns.Column name, the user haspermission on the table.

privileges

No String Permission of the user on the object.

user_name

No String Name of the user who has the permission.


None

● Example response (successful request){ "is_success": true, "message": "", "privileges": [ {



"is_admin": false, "object": "databases.dsstest.tables.csv_par_table", "privileges": [ "SELECT" ], "user_name": "tent2" }, { "is_admin": true, "object": "databases.dsstest.tables.csv_par_table", "privileges": [ "ALL" ], "user_name": "tent4" } ]}


4.6 Querying a User's Table Permissions

Function

This API is used to query the permission of a specified user on a table.

URI● URI format

GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/users/{user_name}



Parameter

Mandatory

Description

project_id


database_name

Yes Name of the database where the table to be queriedresides.

table_name

Yes Name of a table that is to be queried.

user_name

Yes Name of the user whose permission is to be queried.



RequestNone

Response


Parameter

Mandatory

Type Description

is_success


message


user_name

No String Name of the user whose permission is to bequeried.

privileges

No Array ofobjects

Permission information. For details, see Table4-18.


Parameter

Mandatory

Type Description

object No String Objects on which a user has permission.● If the object is in the format of

databases.Database name.tables.Table name,the user has permission on the database.

● If the object is in the format ofdatabases.Database name.tables.Tablenamecolumns.Column name, the user haspermission on the table.

privileges

No Array ofstrings

Permission of the user on a specified object.


None

● Example response (successful request){ "is_success": true, "message": "", "privileges": [ {



"object": "databases.dsstest.tables.obs_2312", "privileges": [ "DESCRIBE_TABLE" ] }, { "object": "databases.dsstest.tables.obs_2312.columns.id", "privileges": [ "SELECT" ] } ], "user_name": "scuser1"}




5 Queue-related APIs

5.1 Creating a Queue

FunctionThis API is used to create a queue. The queue will be bound to specified computeresources.

URI● URI format

POST /v1.0/{project_id}/queues● Parameter description


Parameter

Mandatory

Description

project_id


Request


Parameter Mandatory

Type Description

queue_name Yes String Name of a newly created resource queue.The name can contain only digits, letters,and underscores (_), but cannot containonly digits or start with an underscore (_).

Data Lake InsightAPI Reference 5 Queue-related APIs


Parameter Mandatory

Type Description

queue_type No String Indicates the queue type. The options are asfollows:● sql● general● spark (discarded)NOTE

● If the type is not specified, the default valuesql is used.

● The spark type is marked as discarded and isnot recommended.

● The general queues include but are notlimited to Spark queues.

description No String Description of a queue.

cu_count Yes Integer

Minimum number of CUs that are bound toa queue. Currently, the value can only be 16,64, or 256.

charging_mode No Integer

Billing mode of a queue. This value can onlybe set to 1, indicating that the billing isbased on the CUH used.

enterprise_project_id

No String Enterprise project ID. The value 0 indicatesthe default enterprise project. For detailsabout how to set an enterprise project, seeAccessing the Enterprise Center in theEnterprise Management User Guide.NOTE

Users who have enabled Enterprise Managementcan set this parameter to bind a specified project.

platform No String CPU architecture of queue computeresources. The value can be x86_64 oraarch64. If this parameter is left blank, thedefault value x86_64 is used.NOTE

The aarch64 architecture uses Huawei Kunpengseries servers. For details about the availablesites, see the notice.



https://support.huaweicloud.com/en-us/usermanual-em/en-us_topic_0108763975.html

Response

Table 5-3 Response parameter

Parameter

Mandatory

Type Description

is_success No Boolean Indicates whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.


queue_name

No String Name of the created queue.


{ "queue_name": "queue1", "description": "test", "cu_count": 16, "charging_mode": 1, "enterprise_project_id": "0", "queue_type": "spark"}

● Example response (successful request){ "is_success": true, "message": "", "queue_name": "queue1"}


5.2 Deleting a Queue

5.3 Querying All Queues

FunctionThis API is used to list all queues under the project.

URI● URI format

GET/v1.0/{project_id}/queues





Parameter

Mandatory

Description

project_id


Request


Parameter

Mandatory

Type Description

queue_type

No String

Indicates the queue type. The options are as follows:● sql● spark● allIf this parameter is not specified, the default valuesql is used.

Response


Parameter Mandatory

Type Description

is_success No Boolean

Indicates whether the request is successfullyexecuted. Value true indicates that the requestis successfully executed.


queues No Object

Queue information




Parameter Mandatory

Type Description

queue_name No String Name of a queue.

description No String Queue description.

owner No String User who creates a queue.

create_time No Long Time when the queue is created. Thetimestamp is expressed in milliseconds.

queue_type No String Indicates the queue type. The options are asfollows:● sql● general● spark (deprecated)

cu_count No Integer

Number of compute units (CUs) that are boundto a queue. Currently, the value can only be 16,64, or 256.

charging_mode

No Integer

Billing mode of a queue.● Value 1 indicates that queues are billed

based on the used CUH.● Value 2 indicates that queues are billed

based on yearly/monthly packages.

resource_id No String Resource ID of a queue.


No String Enterprise project ID. 0 indicates the defaultenterprise project. For details about how to setan enterprise project, see Accessing theEnterprise Center in the EnterpriseManagement User Guide.NOTE

Users who have enabled Enterprise Management canset this parameter to bind a specified project.

cidr_in_vpc No String The VPC CIDR of the queue. RecommendedCIDR: 10.0.0.0/8-24, 172.16.0.0/12-24, and192.168.0.0/16-24.

cidr_in_mgntsubnet

No String CIDR of the management subnet

cidr_in_subnet

No String Subnet CIDR

resource_mode

No Integer

Indicates the queue type.● 0: Shared queue● 1: Dedicated queue





Parameter Mandatory

Type Description

platform No String CPU architecture of queue compute resources.The value can be x86_64 or aarch64.

is_restarting No Boolean

Whether to restart the queue. The default valueis false.


None

● Example response (successful request){ "queue_name": "datasource_connection", "owner": "ei_dlics_c00228924", "description": "", "create_time": 1562221422671, "queue_type": "spark", "cu_count": 16, "charging_mode": 2, "resource_id": "26afb850-d3c9-42c1-81c0-583d1163e80f", "enterprise_project_id": "0", "cidr_in_vpc": "10.0.0.0/8", "cidr_in_subnet": "10.0.0.0/24", "cidr_in_mgntsubnet": "10.23.128.0/24", "resource_mode": 1, "platform": "x86_64", "is_restarting": "false" }


5.4 Modifying CIDR

FunctionThis API is used to modify the CIDR of the queues using the yearly/monthlypackages.

If the queue whose CIDR is to be modified has jobs that are being submitted or running, orthe queue has been bound to enhanced datasource connections, the CIDR cannot bemodified.

URI● URI format

PUT /v1.0/{project_id}/queues/{queue_name}





Parameter

Mandatory

Description

project_id


Request


Parameter

Mandatory

Type Description

cidr_in_vpc

No String

Queue VPC CIDR.CIDR range supported by queues with different CUspecifications:● 16 CUs:

10.0.0.0/8 ~ 10.255.255.0/24172.16.0.0/12 ~ 172.31.255.0/24192.168.0.0/16 ~ 192.168.255.0/24

● 64 CUs:10.0.0.0/8 ~ 10.255.252.0/22172.16.0.0/12 ~ 172.31.252.0/22192.168.0.0/16 ~ 192.168.252.0/22

● 256 CUs:10.0.0.0/8 ~ 10.255.252.0/20172.16.0.0/12 ~ 172.31.252.0/20192.168.0.0/16 ~ 192.168.252.0/20

● 512cu:10.0.0.0/8 ~ 10.255.252.0/19172.16.0.0/12 ~ 172.31.252.0/19192.168.0.0/16 ~ 192.168.252.0/19



Response


Parameter Mandatory

Type Description





{ "cidr_in_vpc": "172.16.0.0/12"}



5.5 Restarting, Scaling Out, and Scaling In Queues

FunctionThis API is used to restart, scale out, and scale in queues.

URI● URI format

PUT /v1.0/{project_id}/queues/{queue_name}/action● Parameter description




Parameter Mandatory

Description

project_id Yes Project ID, which is used for resource isolation. Fordetails about how to obtain its value, seeObtaining a Project ID.

queue_name


Request


Parameter

Mandatory

Type Description

action Yes String Operations to be performed:● restart: Restart the queue● scale_out: Scale out the queue● scale_in: Scale in the queueNOTE

Currently, only restart, scale_out, and scale_in operationsare supported.

force No Boolean Specifies whether to forcibly restart the queue. Thisparameter is optional when action is set to restart.The default value is false.

cu_count

No Integer Number of CUs to be scaled in or out. Thisparameter is optional when action is set toscale_out or scale_in.

Response


Parameter Mandatory

Type Description






Parameter Mandatory

Type Description

job_id No String Specifies the job ID returned when force is setto true.

queue_name No String Name of the queue to be scaled in or out.

result No Boolean

Indicates the scaling result.


– Restarting the queue:{ "action": "restart", "force": "false"}

– Scaling out the queue:{ "action": "scale_out", "cu_count": 4}

● Example response (successful request)– Set force to false.

{ "is_success": true, "message": "Restart success"}

– Set force to true.{ "is_success": true, "message": "Submit restart job success, it need some time to cancel jobs, please wait for a while and check job status", "job_id": "d90396c7-3a25-4944-ad1e-99c764d902e7"}

– Scaling{ "queue_name": "myQueue", "result": true}


5.6 Viewing Details of a Queue

FunctionThis API is used to list details of a specific queue in a project.



URI● URI format

GET /v1.0/{project_id}/queues/{queue_name}



Parameter Mandatory

Description


queue_name


Request

Table 5-15 Parameter description

Parameter

Mandatory

Type Description

queue_type

No String Queue type. The options are as follows:● sql● general● allIf this parameter is not specified, the default valuesql is used.

Response


Parameter

Mandatory

Type Description






Parameter

Mandatory

Type Description

queue_name


description

No String Queue description.


create_time

No Long Time when the queue is created. The timestamp isexpressed in milliseconds.

cu_count No Integer

Number of compute units (CUs) that are bound toa queue. Currently, the value can only be 16, 64, or256.

charging_mode

No integer

Billing mode of a queue.● Value 1 indicates that queues are billed based

on the used CUH.● Value 2 indicates that queues are billed based

on yearly/monthly packages.

resource_id

No String Resource ID of a queue.

resource_mode

No Integer

Queue type.● 0: Shared queue● 1: Dedicated queue


No String Enterprise project ID.0 indicates the default enterprise project.NOTE

Users who have enabled Enterprise Management can setthis parameter to bind a specified project.

Example● Example response (successful request)

{ "is_success": true, "message": "", "queue_name": "carbon", "owner": "ei_dlics_c00228924", "description": "test0", "create_time": 1559529407904, "queue_type": "sql", "cu_count": 16, "charging_mode": 1, "resource_id": "3fa02d0e-8a6a-4ad8-bc22-b1053a5124e6", "resource_mode": 0, "enterprise_project_id": "0"}






6 APIs Related to SQL Jobs

6.1 Database-related APIs

6.1.1 Creating a Database

FunctionThis API is used to add a database.

URI● URI format

POST /v1.0/{project_id}/databases● Parameter description


Parameter

Mandatory

Description

project_id


Data Lake InsightAPI Reference 6 APIs Related to SQL Jobs


Request


Parameter

Mandatory

Type Description

database_name

Yes String Name of the created database.NOTE

The default database is a built-in database. You cannotcreate a database named default.

description

No String Information about the created database.

Response


Parameter

Mandatory

Type

Description

is_success

No Boolean

Indicates whether the request is successfully executed. Valuetrue indicates that the request is successfully executed.

message

No String

System prompt. If execution succeeds, the parameter settingmay be left blank.


{ "database_name": "db1", "description": "this is for test"}

● Example response (successful request){ "is_success": true,"message": ""}




6.1.2 Deleting a Database

Function

This API is used to delete an empty database. If there are tables in the database tobe deleted, delete all tables first. For details about the API used to delete tables,see Deleting a Table.

URI● URI format

DELETE /v1.0/{project_id}/databases/{database_name}?cascade={is_cascade}&async={is_async}



Parameter Mandatory

Description


database_name

Yes Name of the database to be deleted.

cascade No Specifies whether to forcibly delete the database.The value can be true or false. Default value: false.

async No Indicates whether to delete the database inasynchronous mode. The value can be true or false.Default value: false.

Request

None

Response


Parameter

Mandatory

Type Description

is_success

No Boolean

Indicates whether the request is successfully executed.Value true indicates that the request is successfullyexecuted.



Parameter

Mandatory

Type Description

message No String System prompt. If execution succeeds, the parametersetting may be left blank.

job_id No String ID of a job returned after a job is generated andsubmitted by using SQL statements. The job ID can beused to query the job status and results.

job_type No String Type of a job. The options are as follows:● DDL● DCL● IMPORT● EXPORT● QUERY● INSERT

job_mode

No String Job execution mode. The options are as follows:● async: asynchronous● sync: synchronous


None

● The following is an example of a successful response in synchronous mode:{ "is_success": true,"message": "","job_mode": "sync"}

– The following is an example of a successful response in asynchronousmode:{ "is_success": true,"message": "","job_id": "208b08d4-0dc2-4dd7-8879-ddd4c020d7aa","job_type": "DDL","job_mode": "async"}

If an error occurs when this API is invoked, the system does not return the resultsimilar to the preceding example, but returns the error code and error information. Fordetails, see Error Code.If the database is deleted asynchronously, you can view the current job status bycalling the API for querying job status. For details, see 5.3.10-Querying the Job Status.If cascade is set to true, all tables in the database will be deleted. Exercise cautionwhen performing this operation.



6.1.3 Querying All Databases

FunctionThis API is used to query the information about all the databases.

URI● URI format

GET /v1.0/{project_id}/databases● Parameter description


Parameter

Mandatory

Description

project_id

Yes Project ID, which is used for resource isolation. Fordetails about how to obtain its value, see Obtaininga Project ID.

RequestNone

Response


Parameter

Mandatory

Type Description

is_success



database_count

No Integer Total number of databases.

databases

No Array ofobjects

Database information. For details, see Table6-8.



Table 6-8 databases parameters

Parameter

Mandatory

Type Description

database_name

No String

Name of a database.

owner No String

Creator of a database.

table_number

No Integer

Number of tables in a database.

description

No String

Information about a database.


None

● Example response (successful request){ "is_success": true, "message": "", "database_count": 1, "databases": [ { "database_name": "db2", "description": "this is for test", "owner": "tenant1", "table_number": 15

} ]}


6.1.4 Modifying a Database Owner

Function

This API is used to modify the owner of a database.

URI● URI format

PUT /v1.0/{project_id}/databases/{database_name}/owner





Parameter

Mandatory

Description

project_id


database_name

Yes Name of a database.

Request


Parameter Mandatory

Type

Description

new_owner Yes String

Name of the new owner.

Response


Parameter

Mandatory

Type Description

is_success

No Boolean


message


Examples● Example request

{ "new_owner": "scuser1"}

● Example response (successful request){ "is_success": true,



"message": ""}


6.2 Table-related APIs

6.2.1 Creating a Table

Function

This API is used to create a table.

URI● URI format

POST /v1.0/{project_id}/databases/{database_name}/tables● Parameter description


Parameter

Mandatory

Description

project_id


database_name

Yes Name of the database where the new table resides.

Request


Parameter

Mandatory

Type

Description

table_name

Yes String

Name of the created table.

data_location

Yes String

Data storage location. OBS tables and DLI tables areavailable.



Parameter

Mandatory

Type

Description

description

No String

Information about the new table.

columns Yes Array

Columns of the new table. For details about columnparameters, see Table 6-14.

For the OBS table

data_type Yes String

Type of the data to be added to the OBS table. Theoptions are as follows: Parquet, ORC, CSV, JSON,Carbon, and Avro.

data_path Yes String

Storage path of data in the new OBS table, whichmust be a path on OBS and must begin with s3a.

with_column_header

No Boolean

Whether the table header is included in the OBS tabledata. Only data in CSV files has this attribute.

delimiter No String

User-defined data delimiter. Only data in CSV files hasthis attribute.

quote_char

No String

User-defined reference character. Double quotationmarks ("\") are used by default. Only data in CSV fileshas this attribute.

escape_char

No String

User-defined escape character. Backslashes (\\) areused by default. Only data in CSV files has thisattribute.

date_format

No String

User-defined date type. yyyy-MM-dd is used bydefault. For details about the characters involved inthe date format, see Table 6-35. Only data in CSV andJSON files has this attribute.

timestamp_format

No String

User-defined timestamp type. yyyy-MM-dd HH:mm:ssis used by default. For definitions about characters inthe timestamp format, see Table 6-35. Only data inCSV and JSON files has this attribute.

Table 6-14 columns parameters

Parameter

Mandatory

Type

Description

column_name

Yes String

Name of a column.



Parameter

Mandatory

Type

Description

type Yes String

Data type of a column.

description

No String

Description of a column.

is_partition_column

No Boolean

Whether the column is a partition column. The valuetrue indicates a partition column, and the value falseindicates a non-partition column. The default value isfalse.NOTE

When creating a partition table, ensure that at least onecolumn in the table is a non-partition column. For details, see"Request example".

Response


Parameter

Type Description

is_success

Boolean

Whether the request is successfully executed. Value trueindicates that the request is successfully executed.

message

String System prompt. If execution succeeds, the parameter settingmay be left blank.


{"table_name": "tb1","data_location": "OBS","description": "","data_type": "csv","data_path": "s3a://obs/path1","columns": [{ "column_name": "column1", "type": "string", "description": "", "is_partition_column": true},{ "column_name": "column2", "type": "string", "description": "", "is_partition_column": false}



],"with_column_header": true,"delimiter": ",","quote_char": "\"","escape_char": "\\","date_format": "yyyy/MM/dd","timestamp_format": "yyyy/MM/dd hh:mm:ss"}



6.2.2 Deleting a Table

FunctionThis API is used to delete a specified table.

URI● URI format

DELETE /v1.0/{project_id}/databases/{database_name}/tables/{table_name}● Parameter description


Parameter

Mandatory

Description

project_id


database_name

Yes Name of the database where the table to be deletedresides.

table_name

Yes Name of the table to be deleted.

RequestNone



Response


Parameter

Mandatory

Type Description

is_success

No Boolean


message



None

● Example response (successful request){ "is_success": true,"message": ""}


6.2.3 Querying All Tables (Recommended)

FunctionThis API is used to query information about tables that meet the filtering criteriaor all the tables in the specified database.

URI● URI format

GET /v1.0/{project_id}/databases/{database_name}/tables?keyword=tb&with-detail=true



Parameter

Mandatory

Description

project_id Yes Project ID, which is used for resource isolation. Fordetails about how to obtain its value, see Obtaining aProject ID.



Parameter

Mandatory

Description

database_name

Yes Name of the database where the table resides.

keyword No Keywords used to filter table names.

with-detail

No Whether to obtain detailed information about tables(such as owner and size).

Request

None

Response


Parameter Mandatory

Type Description

is_success Yes Boolean Whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.

message Yes String System prompt. If execution succeeds, theparameter setting may be left blank.

table_count Yes Integer Total number of tables.

tables Yes Array ofobjects

Table information. For details, see Table6-20.

Table 6-20 tables parameters

Parameter Mandatory

Type Description

create_time Yes Long Time when a table is created. The timestamp isexpressed in milliseconds.

data_type No String Type of the data to be added to the OBS table.The options are as follows: Parquet, ORC, CSV,JSON, Carbon, and Avro. This parameter isavailable only in the OBS table.

data_location

Yes String Data storage location, which can beMANAGED, EXTERNAL, or VIEW. EXTERNALincludes the OBS table and CLoudTable table.



Parameter Mandatory

Type Description

last_access_time

Yes Long Last update time. The timestamp is expressedin milliseconds.

location No String Storage path on the OBS table.

owner Yes String Table owner.

table_name Yes String Name of a table.

table_size Yes Long Size of a DLI table. Set this parameter to 0 fornon-DLI tables.

table_type Yes String Type of a table. Table types include thefollowing:● EXTERNAL: Indicates an OBS table.● MANAGED: Indicates a DLI table.● VIEW: Indicates a view.

partition_columns

No String Partition field. This parameter is valid only forOBS partition tables.

page-size No Integer Paging size. The minimum value is 1 and themaximum value is 100.

current-page

No Integer Current page number. The minimum value is 1.

If with-detail is set to false in the URI, only values of tables-related parametersdata_location, table_name, and table_type are returned.


None

● Example response (successful request){ "is_success": true, "message": "", "table_count": 1, "tables": [ { "create_time":1517364268000, "data_location":"OBS", "data_type":"csv", "last_access_time":1517364268000, "location":"s3a://DLI/sqldata/data.txt", "owner":"test", "partition_columns": ["a0"], "table_name":"obs_t", "table_size":0, "table_type":"EXTERNAL" }



]}


6.2.4 Querying All Tables (Discarded)

FunctionThis API is used to query information about tables that meet the filtering criteriaor all the tables in the specified database.

Information described in this section has been disused. You are advised to view theinformation described in Querying All Tables (Recommended).

URI● URI format

GET /v1.0/{project_id}/databases/{database_name}?keyword=tb&with-detail=true



Parameter

Mandatory

Description

project_id


database_name

Yes Name of the database where the table resides.

keyword No Keywords used to filter table names.

with-detail

No Whether to obtain detailed information about tables(such as owner and size).

RequestNone



Response


Parameter

Mandatory

Type Description

is_success

Yes Boolean


message Yes String System prompt. If execution succeeds, the parametersetting may be left blank.

table_count

Yes Integer

Total number of tables.

tables Yes Arrayofobjects

Table information. For details, see Table 6-23.

Table 6-23 tables parameters

Parameter Mandatory

Type Description

create_time Yes Long Time when a table is created. The timestampis expressed in milliseconds.

data_type No String Type of data in the OBS table. Currently, theparameter value can be parquet, ORC, CSV, orJSON. This parameter is valid only for OBStables.

data_location

Yes String Data storage location. OBS tables, DLI tables,and views are available.

last_access_time

Yes Long Last update time. The timestamp is expressedin milliseconds.

location No String Storage path on the OBS table.

owner Yes String Table owner.

table_name Yes String Name of a table.

table_size Yes Long Size of a DLI table. Set this parameter to 0 fornon-DLI tables.



Parameter Mandatory

Type Description

table_type Yes String Type of a table. Table types include thefollowing:● EXTERNAL: Indicates an OBS table.● MANAGED: Indicates a DLI table.● VIEW: Indicates a view.

partition_columns

No String Partition field. This parameter is valid only forOBS partition tables.

If with-detail is set to false in the URI, only values of parameters data_location,table_name, and table_type are returned.


None

● Example response (successful request){ "is_success": true, "message": "", "table_count": 1, "tables": [ { "create_time":1517364268000, "data_location":"OBS", "data_type":"csv", "last_access_time":1517364268000, "location":"s3a://DLI/sqldata/data.txt", "owner":"test", "partition_columns": ["a0"], "table_name":"obs_t", "table_size":0, "table_type":"EXTERNAL" } ]}


6.2.5 Describing the Table Information

FunctionThis API is used to describe metadata information in the specified table.

URI● URI format



GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}● Parameter description


Parameter Mandatory

Description


database_name

Yes Name of the database where the target tableresides.

table_name Yes Name of the target table.

RequestNone

Response


Parameter Mandatory

Type Description

is_success Yes Boolean Indicates whether the request issuccessfully executed. Value trueindicates that the request issuccessfully executed.

message Yes String System prompt. If execution succeeds,the parameter setting may be leftblank.

column_count Yes Integer Total number of columns in the table.

columns Yes Array ofstrings

Column information, including thecolumn name, type, and description.

table_type Yes String Table type, which can be MANAGED,EXTERNAL, and VIEW.

data_type No String Data type, including CSV, Parquet,ORC, JSON, Carbon, and Avro.

data_location No String Data storage path, which starts withs3a.



Parameter Mandatory

Type Description

storage_properties

No Array ofobjects

Storage attribute, which is in theformat of key/value and includesparameters delimiter, escape, quote,header, dateformat, andtimestampformat.

table_comment No String Table comment.

create_table_sql

No String Statement used to create a table.


None

● Example response (successful request)– MANAGED type table

{ "is_success": true, "message": "", "column_count": 3, "columns": [ { "column_name": "id", "description": "", "type": "int", "is_partition_column": false }, { "column_name": "name", "description": "", "type": "string", "is_partition_column": false }, { "column_name": "level", "description": "", "type": "string", "is_partition_column": true } ], "table_type":"MANAGED"}

– EXTERNAL type table{ "is_success": true, "message": "", "column_count": 2, "columns": [ { "type": "string", "description": "", "column_name": "col2", "is_partition_column": false }, { "type": "string", "description": "",



"column_name": "col1", "is_partition_column": true } ], "table_type": "EXTERNAL", "data_type": "parquet", "data_location": "s3a://obs-wangtao/savepoint/savepoint-d95437-039668840fff/_metadata", "storage_properties": [ { "key": "timestampformat", "value": "yyyy-MM-dd HH:mm:ss" }, { "key": "quote", "value": "\"" }, { "key": "dateformat", "value": "yyyy-MM-dd" }, { "key": "escape", "value": "\\" }, { "key": "header", "value": "false" }, { "key": "delimiter", "value": "," } ], "table_comment": "", "create_table_sql": "CREATE TABLE `default`.`wan_test` (`col2` STRING, `col1` STRING)\nUSING parquet\nOPTIONS (\n `timestampformat` 'yyyy-MM-dd HH:mm:ss',\n `quote` '\"',\n `dateformat` 'yyyy-MM-dd',\n `escape` '\\\\',\n `header` 'false',\n `delimiter` ','\n)\nPARTITIONED BY (col1)\nCOMMENT ''\nLOCATION 's3a://obs-wangtao/savepoint/savepoint-d95437-039668840fff/_metadata'\nTBLPROPERTIES (\n 'hive.serialization.extend.nesting.levels' = 'true'\n)\n"}

– VIEW type table{ "is_success": true, "message": "", "column_count": 3, "columns": [ { "column_name": "id", "description": "", "type": "int", "is_partition_column": false }, { "column_name": "name", "description": "", "type": "string", "is_partition_column": false }, { "column_name": "level", "description": "", "type": "string", "is_partition_column": true } ], "table_type":"VIEW"}



If an error occurs when this API is invoked, the system does not return the resultsimilar to the preceding example, but returns the error code and errorinformation. For details, see Error Code.

is_partition_column in columns indicates whether the column is a partitioncolumn. The value true indicates that the column is a partition column, and thevalue false indicates that the column is not a partition column. The default valueis false.

6.2.6 Previewing Table Content

Function

This API is used to preview the first ten rows of a table.

URI● URI format

GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/preview



Parameter Mandatory

Description

project_id Yes Project ID, which is used for resource isolation.For details about how to obtain its value, seeObtaining a Project ID.

database_name

Yes Name of the database where the table to bepreviewed resides.

table_name Yes Name of the table to be previewed.

Request

None

Response


Parameter

Mandatory

Type Description

is_success




Parameter

Mandatory

Type Description

message


schema

No Array ofobjects

Column name and type of a table.

rows No Array ofstrings

Previewed table content.


NONE

● Example response (successful request){ "is_success": true, "message": "", "schema": [ { "id": "int" }, { "name": "string" }, { "address": "string" } ], "rows": [ { "1", "John", "Shanghai" }, { "2", "Lily", "Hangzhou" } ] }


6.2.7 Obtaining the Partition List

FunctionThis API is used to obtain the partition list.



URI● URI format

GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/partitions



Parameter Mandatory

Description


database_name

Yes Name of a database.

table_name Yes Name of a table.

Request


Parameter

Mandatory

Type Description

limit No Integer Number of returned records displayed on eachpage. The default value is 100.

offset No Integer Offset.

filter No String Filtering condition. Currently, only the = conditionis supported. For example, name=name1indicates that the data whose name is name1 inthe partition is filtered.The key and value are case insensitive.

Response


Parameter

Mandatory

Type Description

is_success No Boolean Whether the request is successfully executed.Value true indicates that the request issuccessfully executed.



Parameter

Mandatory

Type Description


partitions No Json Partition information. For details, see Table6-31.

Table 6-31 partitions parameter description

Parameter Mandatory

Type Description

total_count Yes Long Total number of partitions.

partition_infos Yes Json Array List of partitions For details, see Table6-32.

Table 6-32 partition_info parameter description

Parameter Mandatory

Type Description

partition_name Yes String Partition name.

create_time Yes Long Time when a partition is created.

last_access_time Yes Long Last update time.

locations No String Path. This parameter is displayedonly for non-DLI tables.

last_ddl_time No Long Execution time of the last DDLstatement, in seconds.

num_rows No Long Total rows in the partition.

num_files No Long Number of files in a partition.

total_size No Long Total size of data in the partition,in bytes.


NONE

● Example response (successful request){ "is_success": true, "message": "list partitions succeed", "partitions": {



"total_count": 1, "partition_infos": [ { "partition_name": "name=test", "create_time": 1579520179000, "last_access_time": 1579520179000, "locations": [ "s3a://obs-jptest/partition" ] }, { "partition_name": "name=test1", "create_time": 1579521406000, "last_access_time": 1579521406000, "locations": [ "s3a://obs-jptest/partition" ] }, { "partition_name": "name=test2", "create_time": 1579521884000, "last_access_time": 1579521884000, "locations": [ "s3a://obs-jptest/partition" ] }, { "partition_name": "name=test3", "create_time": 1579522085000, "last_access_time": 1579522085000, "locations": [ "s3a://obs-jptest/partition" ] }, { "partition_name": "name=name1/age=age1", "create_time": 1581409182000, "last_access_time": 1581409182000, "locations": [ "s3a://obs-jptest/0117" ], "last_ddl_time": 1581409182, "total_size": 2130, "num_rows": -1, "num_files": 2 } ] }}


6.3 Job-related APIs

6.3.1 Importing Data

FunctionThis API is used to import data from a file to a DLI or OBS table. Currently, onlyOBS data can be imported to a DLI or OBS table.



● This API is asynchronous.● When importing data, you can select an existing OBS bucket path or create an OBS

bucket path, but only one OBS bucket path can be specified.● If you need to create an OBS bucket, ensure that the bucket name complies with the

following naming rules:● The name must be globally unique in OBS.● The name must contain 3 to 63 characters. Only lowercase letters, digits, hyphens

(-), and periods (.) are allowed.● The name cannot start or end with a period (.) or hyphen (-), and cannot contain

two consecutive periods (.) or contain a period (.) and a hyphen (-) adjacent toeach other.

● The name cannot be an IP address.● If the name contains any period (.), the security certificate verification may be

triggered when you access the bucket or objects in the bucket.● If the type of a column in the source file to be imported does not match that of the

target table, the query result of the row will be null.● Two or more concurrent tasks of importing data to the same table are not allowed.

URI● URI format

POST /v1.0/{project_id}/jobs/import-table● Parameter description


Parameter Mandatory

Description


Request


Parameter Mandatory

Type Description

data_path Yes String Path to which data is imported. Currently, onlydata on an OBS path starting with s3a can beimported.



Parameter Mandatory

Type Description

data_type Yes String Type of the data to be imported. Currently,data types of CSV, Parquet, ORC, JSON, andAvro are supported.NOTE

Data in Avro format generated by Hive tablescannot be imported.

database_name

Yes String Name of the database where the table towhich data is imported resides.

table_name Yes String Name of the table to which data is imported.

with_column_header

No Boolean

Whether the first line of the imported datacontains column names, that is, headers. Thedefault value is false, indicating that columnnames are not contained. This parameter canbe specified when CSV data is imported.

delimiter No String User-defined data delimiter. The default valueis a comma (,). This parameter can be specifiedwhen CSV data is imported.

quote_char No String User-defined quotation character. The defaultvalue is double quotation marks ("). Thisparameter can be specified when CSV data isimported.

escape_char No String User-defined escape character. The defaultvalue is a backslash (\). This parameter can bespecified when CSV data is imported.

date_format No String Specified date format. The default value isyyyy-MM-dd. For details about the charactersinvolved in the date format, see Table 6-35.This parameter can be specified when data inthe CSV or JSON format is imported.

bad_records_path

No String Bad records storage directory during jobexecution. After configuring this item, the badrecords is not imported into the target table.

timestamp_format

No String Specified time format. The default value isyyyy-MM-dd HH:mm:ss. For definitions aboutcharacters in the times format, see Table 6-35.This parameter can be specified when data inthe CSV or JSON format is imported.

queue_name No String Name of the queue that is specified to executea task. If no queue is specified, the defaultqueue is used.



Table 6-35 Definition of characters involved in the date and time patterns

Character

Date or TimeElement

Example

G Epoch ID AD

y Year 1996; 96

M Month July; Jul; 07

w Which week ina year

27 (Week 27 in the year)

W Which week ina month

2 (Second week in the month)

D Which day in ayear

189 (Day 189 in the year)

d Which day in amonth

10 (Day 10 in the month)

u Which day in aweek

1 (Monday), ..., 7 (Sunday)

a am/pm flag pm (Afternoon)

H Hour time(0-23)

2

h Hour time(1-12)

12

m Minute time 30

s Second time 55

S Whichmilliseconds

978

z Time zone Pacific Standard Time; PST; GMT-08:00

Response


Parameter

Mandatory

Type Description

is_success

No Boolean

Indicates whether the request is successfully sent.Value true indicates that the request is successfullysent.



Parameter

Mandatory

Type Description


job_id No String ID of a job returned after a job is generated andsubmitted by using SQL statements. The job ID canbe used to query the job status and results.

job_mode



{ "data_path": "s3a://home/data1/DLI/t1.csv", "data_type": "csv", "database_name": "db2", "table_name": "t2", "with_column_header": false, "delimiter": ",", "quote_char": ",", "escape_char": ",", "date_format": "yyyy-MM-dd", "timestamp_format": "yyyy-MM-dd'T'HH:mm:ss.SSSZZ", "queue_name": "queue2"}

● Example response (successful request){ "is_success": true, "message": "import data to table t2 started", "job_id": "6b29eb77-4c16-4e74-838a-2cf7959e9202", "job_mode":"async"}


6.3.2 Exporting Data

FunctionThis API is used to export data to a file. You can export data from DLI tables tofiles.



● This API is asynchronous.

● Currently, data can be exported only to OBS, and the OBS path must be specified to thefolder level. The OBS path cannot contain commas (,). The OBS bucket name cannotend with the regular expression format ".[0-9]+(.*)". Specifically, if the bucket namecontains dots (.), the last dot (.) cannot be followed by a digit, for example, "**.12abc"and "**.12".

URI● URI format

POST /v1.0/{project_id}/jobs/export-table



Parameter

Mandatory

Description

project_id


Request


Parameter Mandatory

Type Description

data_path Yes String Path for storing the exported data (currently,data can be stored on an OBS path startingwith s3a only). If export_mode is set toerrorifexists, the OBS path cannot contain thespecified folder, for example, the test folder inthe example request.

data_type Yes String Type of data to be exported. Currently, onlyCSV is supported.

database_name

Yes String Name of the database where the table fromwhich data is exported resides.

table_name Yes String Name of the table from which data isexported.

compress Yes String Compression mode for exported data.Currently, the compression modes gzip, bzip2,and deflate are supported. If you do not wantto compress data, enter none.



Parameter Mandatory

Type Description

queue_name No String Name of the queue that is specified toexecute a task. If no queue is specified, thedefault queue is used.

export_mode No String Export mode. The parameter value can beErrorIfExists or Overwrite. If export_mode isnot specified, this parameter is set toErrorIfExists by default.● ErrorIfExists: Ensure that the specified

export directory does not exist. If thespecified export directory exists, an error isreported and the export operation cannotbe performed.

● Overwrite: If you add new files to aspecific directory, existing files will bedeleted.

with_column_header

No Boolean

Whether to export column names whenexporting CSV data.● If this parameter is set to true, the column

names are exported.● If this parameter is set to false, the column

names are not exported.● If this parameter is left blank, the default

value false is used.

Response


Parameter

Mandatory

Type Description

is_success

No Boolean Indicates whether the request is successfully sent.Value true indicates that the request is successfullysent.

message


job_id No String ID of a job returned after a job is generated andsubmitted by using SQL statements. The job ID canbe used to query the job status and results.

job_mode





{ "data_path": "s3a://home/data1/DLI/test", "data_type": "csv", "database_name": "db2", "table_name": "t2", "compress": "gzip", "with_column_header": "true", "queue_name": "queue2"}

● Example response (successful request){"is_success": true, "message": "export all data from table db2.t2 to path s3a://home/data1/DLI/test started", "job_id": "828d4044-3d39-449b-b32c-957f7cfadfc9", "job_mode":"async"}


6.3.3 Submitting a SQL Job (Recommended)

FunctionThis API is used to submit jobs to a queue using SQL statements.

The job types support DDL, DCL, IMPORT, QUERY, and INSERT. Functions ofIMPORT and EXPORT are the same as those described in Importing Data. Thedifference lies in the implementation method.

Additionally, you can use other APIs to query and manage jobs. For details, see thefollowing sections:

● Querying Job Status● Querying Job Details● Querying the Job Execution Result (Recommended)● Exporting Query Results● Querying All Jobs● Canceling a Job (Recommended)

This API is synchronous if job_type in the response message is DCL.

URI● URI format

POST /v1.0/{project_id}/jobs/submit-job● Parameter description




Parameter

Mandatory

Description

project_id


Request


Parameter

Mandatory

Type Description

sql Yes String SQL statement that you want to execute.

currentdb No String Database where the SQL statement isexecuted. This parameter does not need to beconfigured during database creation.

queue_name

No String Name of the queue to which a job to besubmitted belongs. The name can containonly digits, letters, and underscores (_), butcannot contain only digits or start with anunderscore (_).

conf No Array ofobjects

You can set the configuration parameters forthe SQL job in the form of Key/Value. Fordetails about the supported configurationitems, see Table 6-42.

Table 6-42 Configuration parameters description

Parameter Description

dli.sql.autoBroadcastJoinThreshold

Data volume threshold for automatically using theBroadcastJoin.

dli.sql.shuffle.partitions The number of Partitions in the data shuffle process.

dli.sql.dynamicPartitio-nOverwrite.enabled

In dynamic partitioning mode, only the partitionsinvolved in the query data are overwritten. Thepartitions that are not involved are not deleted.

dli.sql.files.maxPartitionBytes

Maximum number of bytes in a partition. Sets themaximum number of bytes that can be processed bya partition. When the size of a file or partition isgreater than this value, the file or partition is splitinto multiple partitions.




dli.sql.badRecordsPath Storage directory of the bad records of the INSERTjob execution process. After configuring this item,the bad records is not inserted into the target table.

dli.sql.sqlasync.enabled Indicates whether DDL and DCL statements areexecuted asynchronously. The value true indicatesthat asynchronous execution is enabled.

Response


Parameter

Mandatory

Type Description

is_success

Yes Boolean Indicates whether the request is successfullysent. Value true indicates that the request issuccessfully sent.

message

Yes String System prompt. If execution succeeds, theparameter setting may be left blank.

job_id Yes String ID of a job returned after a job is generated andsubmitted by using SQL statements. The job IDcan be used to query the job status and results.

job_type

Yes String Type of a job. Job types include the following:● DDL● DCL● IMPORT● EXPORT● QUERY● INSERT

schema No Array ofobjects

If the statement type is DDL, the column nameand type of DDL are displayed.

rows No Array ofobjects

When the statement type is DDL, results of theDDL are displayed.

job_mode





{ "currentdb": "db1", "sql": "desc table1", "queue_name": "default", "conf": [ "dli.sql.shuffle.partitions = 200" ]}

● Example response (successful request){ "is_success": true, "message": "{""}", "job_id": "8ecb0777-9c70-4529-9935-29ea0946039c", "job_type": "DDL", "job_mode":"sync", "schema": [ { "col_name": "string" }, { "data_type": "string" }, { "comment": "string" } ], "rows": [ [ "c1", "int", null ], [ "c2", "string", null ] ]}


6.3.4 Submitting a SQL Job (Deprecated)

Function

This API is used to submit jobs to a queue using SQL statements.

The job types support DDL, DCL, IMPORT, EXPORT, QUERY, and INSERT. Functionsof IMPORT and EXPORT are the same as those described in Importing Data andExporting Data. The difference lies in the implementation method.

Additionally, you can use other APIs to query and manage jobs. For details, see thefollowing sections:

● Querying Job Status



● Querying Job Details● Querying the Job Execution Result (Discarded)● Exporting Query Results● Querying All Jobs● Canceling a Job (Discarded)

● This API is synchronous if job_type in the response message is DCL.● Information described in this section has been disused. You are advised to view the

information described in Submitting a SQL Job (Recommended).

URI● URI format

POST /v1.0/{project_id}/queues/{queue_name}/jobs/submit-job● Parameter description


Parameter

Mandatory

Description

project_id


queue_name

Yes Current queue name. Resources of the queue are usedfor computing if SQL requires resources.

Request


Parameter

Mandatory

Type Description


currentdb

No String Database where the SQL statement is executed. Thisparameter does not need to be configured duringdatabase creation.



Parameter

Mandatory

Type Description

conf No Arrayofobjects

User-defined parameter that applies to the job.Currently, the following configuration items aresupported:● dli.sql.join.preferSortMergeJoin (specifies

whether SortMergeJoin is preferred)● dli.sql.autoBroadcastJoinThreshold (specifies the

data volume threshold to use BroadcastJoin. If thedata volume exceeds the threshold, BroadcastJoinwill be automatically enabled.)

● dli.sql.caseSensitive (specifies whether SQLstatements are case-sensitive)

● dli.sql.shuffle.partitions (specifies the number ofpartitions during shuffling)

● dli.sql.cbo.enabled (specifies whether to enablethe CBO optimization policy)

● dli.sql.cbo.joinReorder.enabled (specifies whetherjoin reordering is allowed when CBO optimizationis enabled)

Response


Parameter

Mandatory

Type Description

is_success

Yes Boolean

If job_type is set to DCL, this parameter indicateswhether the request is successfully executed. Value trueindicates that the request is successfully executed.

message

Yes String System prompt. If execution succeeds, the parametersetting may be left blank.

job_id Yes String ID of a job returned after a job is generated andsubmitted by using SQL statements. The job ID can beused to query the job status and results.



Parameter

Mandatory

Type Description

job_type

Yes String Type of a job. Job types include the following:● DDL● DCL● IMPORT● EXPORT● QUERY● INSERT

schema

No Arrayofobjects

If the statement type is DDL, the column name andtype of DDL are displayed.

rows No Arrayofobjects

When the statement type is DDL, results of the DDL aredisplayed.


{ "currentdb": "db1", "sql": "desc table1", "conf": [ "dli.sql.shuffle.partitions = 200" ]}

● Example response (successful request){ "is_success": true, "message": "", "job_id": "8ecb0777-9c70-4529-9935-29ea0946039c", "job_type": "DDL", "schema": [ { "col_name": "string" }, { "data_type": "string" }, { "comment": "string" } ], "rows": [ [ "c1", "int", null ], [ "c2", "string",



null ] ]}


6.3.5 Canceling a Job (Recommended)

Function

This API is used to cancel a submitted job. If execution of a job completes or fails,this job cannot be canceled.

URI● URI format

DELETE /v1.0/{project_id}/jobs/{job_id}● Parameter description


Parameter

Mandatory

Description

project_id


job_id Yes Job ID.

Request

None

Response


Parameter

Mandatory

Type Description

is_success

No Boolean


message

No String

System prompt. If execution succeeds, the parametersetting may be left blank.




None



6.3.6 Canceling a Job (Discarded)

Function

This API is used to cancel a submitted job. If execution of a job completes or fails,this job cannot be canceled.

Information described in this section has been disused. You are advised to view theinformation described in Canceling a Job (Recommended).

URI● URI format

DELETE /v1.0/{project_id}/queues/{queue_name}/jobs/{job_id}



Parameter

Mandatory

Description

project_id


queue_name

Yes Name of the queue where the job to be canceled resides.

job_id Yes Job ID.

Request

None



Response


Parameter

Mandatory

Type Description

is_success

No Boolean


message



None



6.3.7 Querying All Jobs

FunctionThis API is used to query information about all jobs in the current project.

URI● URI format

GET/v1.0/{project_id}/jobs?page-size={size}&current-page={page_number}&start={start_time}&end={end_time}&job-type={QUERY}&queue_name={test}&order={duration_desc}



Parameter

Mandatory

Description




Parameter

Mandatory

Description

page-size No Maximum number of jobs displayed on each page. Thevalue range is as follows: [1, 100]. The default value is50.

current-page

No Current page number. The default value is 1.

start No Queries the jobs executed later than the time specifiedby this parameter. The time is a UNIX timestamp inmilliseconds.

end No Queries the jobs executed earlier than the timespecified by this parameter. The time is a UNIXtimestamp in milliseconds.

job-type No Type of a job to be queried. Job types include DDL,DCL, IMPORT, EXPORT, QUERY, and INSERT. To queryall types of jobs, enter ALL.

queue_name

No Specifies queue_name as the filter to query jobsrunning on the specified queue.

sql_pattern

No Specifies the SQL segment as the filter. It is caseinsensitive.

order No Specifies the job sorting mode. The default value isstart_time_desc (job submission time in descendingorder). Four sorting modes are supported:duration_desc (job running duration in descendingorder), duration_asc (job running duration inascending order), start_time_desc (job submissiontime in descending order), and start_time_asc (jobsubmission time in ascending order).

Request

None

Response


Parameter

Mandatory

Type Description

is_success

Yes Boolean Indicates whether the request is successfullyexecuted. Value true indicates that the requestis successfully executed.



Parameter

Mandatory

Type Description

message

Yes String System prompt. If execution succeeds, theparameter setting may be left blank.

job_count

Yes Integer Indicates the total number of jobs.

jobs Yes Array ofobjects

Indicates the information about a job. Fordetails, see Table 6-53.

Table 6-53 jobs parameters

Parameter Mandatory

Type Description

job_id Yes String Job ID.

job_type Yes String Type of a job.

queue_name Yes String Queue to which a job is submitted.

owner Yes String User who submits a job.

start_time Yes Long Time when a job is started. The timestamp isexpressed in milliseconds.

duration Yes Long Job running duration (unit: millisecond).

status Yes String Status of a job, including LAUNCHING,RUNNING, FINISHED, FAILED, andCANCELED.

input_row_count

No Long Number of records scanned during the Insertjob execution.

bad_row_count

No Long Number of error records scanned during theInsert job execution.

input_size Yes Long Size of scanned files during job execution.

result_count Yes Integer

Total number of records returned by thecurrent job or total number of records insertedby the Insert job.

database_name

No String Name of the database where the target tableresides. database_name is valid only for jobsof the Import and Export types.

table_name No String Name of the target table. table_name is validonly for jobs of the Import and Export types.



Parameter Mandatory

Type Description

with_column_header

No boolean

Import jobs, which record whether theimported data contains column names.

detail Yes String JSON character string of related columnsqueried by using SQL statements.

statement Yes String SQL statements of a job.


None

● Example response (successful request){ "is_success": true, "message": "", "job_count": 1, "jobs": [ {

"detail": "{\"type\":\"struct\",\"fields\":[{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"age\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}", "duration": 17731,

"end_time": 1502349821460, "input_size": 0, "job_id": "37286cc7-0508-4ffd-b636-951c8a5c75de", "job_type": "QUERY", "message": "", "owner": "tenant1", "queue_name": "queue1", "result_count": 3, "start_time": 1502349803729, "statement": "select * from t_json_002", "status": "FINISHED", "with_column_header": false } ]}


6.3.8 Querying the Job Execution Result (Recommended)

FunctionThis API is used to view the job execution result after a job is executed using SQLquery statements. Currently, you can only query execution results of jobs of theQUERY type.

You can view the first 1000 result records only. To view all result records, exportthem first. For details, see Exporting Query Results.



URI● URI format

GET/v1.0/{project_id}/jobs/{job_id}?page-size={size}&current-page={page_number}&queue-name={queue_name}



Parameter Mandatory

Description


job_id Yes Job ID.

page-size No Maximum number of lines displayed on each page.The value range is as follows: [1, 100]. The defaultvalue is 50.

current-page

No Current page number. The default value is 1.

queue-name

No Name of the execution queue for obtaining jobresults. If this parameter is not specified, the defaultsystem queue is used.

RequestNone

Response


Parameter

Mandatory

Type Description



job_id No String Job ID.

job_type No String Job type, including DDL, DCL, IMPORT,EXPORT, QUERY, and INSERT.Currently, you can only query executionresults of jobs of the QUERY type.



Parameter

Mandatory

Type Description

row_count

No Integer Total number of job results.

input_size

No long Amount of data scanned during jobexecution.


Name and type of the job result column.


Job result set.


None

● Example response (successful request){ "is_success": true, "message": "", "job_id": "ead0b276-8ed4-4eb5-b520-58f1511e7033", "job_type": "QUERY", "row_count": 1, "input_size": 74, "schema": [ { "c1": "int" }, { "c2": "string" } ], "rows": [ [ 23, "sda" ] ]}


6.3.9 Querying the Job Execution Result (Discarded)

FunctionThis API is used to view the job execution result after a job is executed using SQLquery statements. Currently, you can only query execution results of jobs of theQUERY type.



● You can view the first 1000 result records only. To view all result records, export themfirst. For details, see Exporting Query Results.

● Information described in this section has been disused. You are advised to view theinformation described in Querying the Job Execution Result (Recommended).

URI● URI format

GET/v1.0/{project_id}/queues/{queue_name}/jobs/{job_id}?page-size=size&current-page=page_number



Parameter Mandatory Description

project_id Yes Project ID, which is used for resourceisolation. For details about how toobtain its value, see Obtaining aProject ID.

queue_name Yes Current queue name. SQL usesresources of the current queue forcomputing.

job_id Yes Job ID.

page-size No Maximum number of lines displayedon each page. The value range is asfollows: [1, 100]. The default value is50.

current-page No Current page number. The defaultvalue is 1.

Request

None

Response


Parameter

Mandatory

Type Description

is_success




Parameter

Mandatory

Type Description

message


job_id No String Job ID.

job_type

No String Job type, including DDL, DCL, IMPORT,EXPORT, QUERY, and INSERT.Currently, you can only query execution resultsof jobs of the QUERY type.

row_count

No Integer Total number of job results.

input_size

No long Amount of data scanned during job execution.


Name and type of the job result column.


Job result set.


None

● Example response (successful request){ "is_success": true, "message": "", "job_id": "ead0b276-8ed4-4eb5-b520-58f1511e7033", "job_type": "QUERY", "row_count": 1, "input_size": 74, "schema": [ { "c1": "int" }, { "c2": "string" } ], "rows": [ [ 23, "sda" ] ]}




6.3.10 Querying Job Status

Function

This API is used to query the status of a submitted job.

URI● URI format

GET/v1.0/{project_id}/jobs/{job_id}/status● Parameter description


Parameter

Mandatory

Description

project_id


job_id Yes Job ID.

Request

None

Response


Parameter Mandatory

Type Description


message Yes String System prompt. If execution succeeds,the parameter setting may be leftblank.


job_type Yes String Type of a job, including DDL, DCL,IMPORT, EXPORT, QUERY, andINSERT.

queue_name Yes String Name of the queue where the job issubmitted.




Parameter Mandatory

Type Description

start_time Yes Long Time when a job is started. Thetimestamp is expressed in milliseconds.

duration Yes Long Job running duration (unit:millisecond).

status Yes String Status of a job, includingLAUNCHING, RUNNING, FINISHED,FAILED, and CANCELED.

input_row_count

No Long Number of records scanned during theInsert job execution.

bad_row_count

No Long Number of error records scannedduring the Insert job execution.

input_size Yes Long Size of scanned files during jobexecution (unit: byte).

result_count Yes Integer Total number of records returned bythe current job or total number ofrecords inserted by the Insert job.

database_name

No String Name of the database where thetarget table resides. database_name isvalid only for jobs of the IMPORTEXPORT, and QUERY types.

table_name No String Name of the target table. table_nameis valid only for jobs of the IMPORTEXPORT, and QUERY types.

detail Yes String JSON character string for informationabout related columns.

statement Yes String SQL statements of a job.


None

● Example response (successful request){ "is_success": true, "message": "{"error_code":"DLI.0999"}", "job_id": "208b08d4-0dc2-4dd7-8879-ddd4c020d7aa", "job_type": "QUERY", "queue_name": "default", "owner": "test", "start_time": 1509335108918, "duration": 2523, "status": "FINISHED", "input_size": 22, "result_count": 4, "database_name":"dbtest",



"table_name":"tbtest", "detail": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", "statement": "select * from t1"}


6.3.11 Querying Job Details

Function

This API is used to query details about jobs, including databasename, tablename,file size, and export mode.

URI● URI format

GET/v1.0/{project_id}/jobs/{job_id}/detail● Parameter description


Parameter

Mandatory

Description

project_id


job_id Yes Job ID.

Request

None

Response


Parameter Mandatory

Type Description





Parameter Mandatory

Type Description



start_time Yes Long Time when a job is started. Thetimestamp is expressed in milliseconds.

duration Yes Long Duration for executing the job (unit:millisecond).

export_mode No String Specified export mode during data exportand query result saving.

data_path Yes String Path to imported or exported files.

data_type Yes String Type of data to be imported or exported.Currently, only CSV is supported.

database_name Yes String Name of the database where the table,where data is imported or exported,resides.

table_name Yes String Name of the table where data is importedor exported.

with_column_header

No Boolean Whether the imported data contains thecolumn name during the execution of animport job.

delimiter No String User-defined data delimiter set when theimport job is executed.

quote_char No String User-defined quotation character setwhen the import job is executed.

escape_char No String User-defined escape character set whenthe import job is executed.

date_format No String Table date format specified when theimport job is executed.

timestamp_format

No String Table time format specified when theimport job is executed.

compress No String Compression mode specified when theexport job is executed.


None

● Example response (successful request)



– Querying jobs of the Import type{ "is_success": true, "message": "", "data_path": "s3a://DLI/computeCharging/test.csv", "data_type": "csv", "database_name": "iam_exist", "date_format": "yyyy-MM-dd", "delimiter": ",", "duration": 1623, "escape_char": "\\", "job_id": "a85d7298-ecef-47f9-bb31-499d2099d112", "owner": "iam_exist", "quote_char": "\"", "start_time": 1517385246111, "table_name": "DLI_table20", "timestamp_format": "yyyy-MM-dd HH:mm:ss", "with_column_header": false}

– Query jobs of the Export type{ "is_success": true, "message": "", "compress": "none", "data_path": "s3a://carbondb/dli/path6", "data_type": "csv", "database_name": "submitjob", "duration": 4142, "export_mode": "Overwrite", "job_id": "b89fccb2-de6a-4c6c-b9b2-21f08a2eb85e", "owner": "test", "start_time": 1524107798024, "table_name": "autotest"}

If an error occurs when this API is invoked, the system does not return the resultsimilar to the preceding example, but returns the error code and errorinformation. For details, see Error Code.

6.3.12 Checking SQL Syntax

FunctionThis API is used to check the SQL syntax.

URI● URI format

POST /v1.0/{project_id}/jobs/check-sql● Parameter description




Parameter

Mandatory

Description

project_id


Request


Parameter

Mandatory

Type Description


currentdb

No String Database where the SQL statement is executed.NOTE

● If the SQL statement contains db_name, for example,select * from db1.t1, you do not need to set thisparameter.

● If the SQL statement does not contain db_name, thesemantics check will fail when you do not set thisparameter or set this parameter to an incorrect value.

Response


Parameter

Mandatory

Type Description

is_success

No Boolean


message


job_type

No String Type of a job. Job types include the following: DDL,DCL, IMPORT, EXPORT, QUERY, and INSERT.


{ "currentdb": "db1",



"sql": "select * from t1" }

● Example response (successful request){ "is_success": true, "message": "the sql is ok", "job_type":QUERY}


6.3.13 Exporting Query Results

Function

This API is used to export results returned from the query using SQL statements toOBS. Only the query result of QUERY jobs can be exported.

● This API is asynchronous.

● Currently, data can be exported only to OBS, and the OBS path must be specified to thefolder level. The OBS path cannot contain commas (,). The OBS bucket name cannotend with the regular expression format ".[0-9]+(.*)". Specifically, if the bucket namecontains dots (.), the last dot (.) cannot be followed by a digit, for example, "**.12abc"and "**.12".

URI● URI format

POST /v1.0/{project_id}/jobs/{job_id}/export-result



Parameter Mandatory

Description


job_id Yes Job ID.

Request

Request Parameters

Table 6-66 describes the request parameters.




Parameter Mandatory

Type Description

data_path Yes String Path for storing the exported data (currently,data can be stored on an OBS path startingwith s3a or obs). The OBS path cannotcontain folders, for example, the path folder inthe sample request.

compress No String Compression format of exported data.Currently, gzip, bzip2, and deflate aresupported. The default value is none,indicating that data is not compressed.

data_type Yes String Storage format of exported data. Currently,only CSV is supported.

queue_name No String Name of the queue that is specified to executea task. If no queue is specified, the defaultqueue is used.

export_mode No String Export mode. The parameter value can beErrorIfExists or Overwrite. If export_mode isnot specified, this parameter is set toErrorIfExists by default.● ErrorIfExists: Ensure that the specified

export directory does not exist. If thespecified export directory exists, an error isreported and the export operation cannotbe performed.

● Overwrite: If you add new files to a specificdirectory, existing files will be deleted.

with_column_header

No Boolean

Whether to export column names whenexporting CSV data.● If this parameter is set to true, the column

names are exported.● If this parameter is set to false, the column

names are not exported.● If this parameter is left blank, the default

value false is used.



Response


Parameter

Mandatory

Type Description

is_success

Yes Boolean Indicates whether the request is successfullysent. Value true indicates that the request issuccessfully sent.


job_id No String ID of a job returned after a job is generatedand submitted by using SQL statements. Thejob ID can be used to query the job status andresults.

job_mode

No String Job execution mode. The options are asfollows:● async: asynchronous● sync: synchronous


{ "data_path": "s3a://obs-bucket1/path", "data_type": "csv", "compress": "gzip", "with_column_header": "true", "queue_name": "queue2"}

● Example response (successful request){ "is_success": true, "message": "", "job_id": "37a40ef9-86f5-42e6-b4c6-8febec89cc20", "job_mode":"async"}


6.4 Queue-related APIs (Discarded)



6.4.1 Creating a Queue (Discarded)

Function

This API is used to create a queue. The queue will be bound to specified computeresources.

URI● URI format

POST /v1.0/{project_id}/queues



Parameter

Mandatory

Description

project_id


Request


Parameter Mandatory

Type Description

queue_name Yes String

Name of a newly created resource queue. Thename can contain only digits, letters, andunderscores (_), but cannot contain only digitsor start with an underscore (_).

description No String

Description of a queue.


Minimum number of CUs that are bound to aqueue. Currently, the value can only be 16, 64,or 256.

charging_mode

No Integer

Billing mode of a queue. The value must be 1,indicating that queues are billed based on theCUH.

cidr_in_vpc No String

The VPC CIDR of the queue. RecommendedCIDR: 10.0.0.0/8-24, 172.16.0.0/12-24, and192.168.0.0/16-24.



Response


Parameter

Mandatory

Type Description



queue_name

No String Name of the created queue.


{ "queue_name": "queue1", "description": "test", "cu_count": 16, "charging_mode": 1}

● Example response (successful request){ "is_success": true, "message": "", "queue_name": "queue1"}


6.4.2 Deleting a Queue (Discarded)

Function

This API is used to delete a specified queue.

If a task is being executed in a specified queue, the queue cannot be deleted.

URI● URI format

DELETE /v1.0/{project_id}/queues/{queue_name}● Parameter description




Parameter

Mandatory

Description

project_id


queue_name

Yes Name of a queue to be deleted.

RequestNone

Response


Parameter

Mandatory

Type Description

is_success No Boolean Indicates whether the request is successfullyexecuted. Value true indicates that the requestis successfully executed.



None





6.4.3 Querying the Queue List (Discarded)

Function

This API is used to list all queues under the project.

URI● URI format

GET/v1.0/{project_id}/queues



Parameter

Mandatory

Description

project_id


Request

None

Response


Parameter Mandatory

Type Description




queue_name


description No String Queue description.


create_time

No Long Time when the queue is created. The timestamp isexpressed in milliseconds.



Parameter Mandatory

Type Description

cu_count No Integer

Number of compute units (CUs) that are bound toa queue. Currently, the value can only be 16, 64,or 256.

charging_mode

No Integer

Billing mode of a queue. Value 1 indicates thatqueues are billed based on the used CUH.


None

● Example response (successful request){ "is_success": true, "message": "", "queues": [ { "charging_mode":1, "create_time": 1508143955000, "cu_count": 16, "description": "test", "owner": "tenant1", "queue_name": "queue1" } ]}




7 APIs Related to Stream Jobs

7.1 Granting OBS Permissions to DLI

FunctionThis API is used to grant DLI the permission to access OBS buckets for saving jobcheckpoints and run logs.

URI● URI format

POST /v1.0/{project_id}/dli/obs-authorize● Parameter description



project_id Yes Project ID, which is used for resourceisolation. For details about how to obtainits value, see Obtaining a Project ID.

Request● Example request

{ "obs_buckets": [ "bucket1" ]}


Data Lake InsightAPI Reference 7 APIs Related to Stream Jobs



Parameter Mandatory

Type Description

obs_buckets Yes String List of OBS buckets.

Response● Example response

{ "is_success": "true", "message": "The following OBS bucket is authorized successfully, bucket1."}



Parameter Mandatory

Type Description

is_success No Boolean Whether the request issuccessfully executed. Value trueindicates that the request issuccessfully executed.

message No String Message content.

Status CodeTable 7-4 describes the status code.

Table 7-4 Status codes

Status Code Description

200 Authorization succeeds.

400 Request error.

Error CodeIf an error occurs when this API is invoked, the system does not return the resultsimilar to the preceding example, but returns the error code and error information.For details, see Error Code.



Table 7-5 Error code

Error Code Description

DLI.19005 The request contains the name of authorized OBSbucket. Filter out the authorized OBS bucket first.

7.2 Creating a SQL Job

FunctionThis API is used to use the POST method to submit a stream SQL job in JSONformat.

URI● URI format

POST /v1.0/{project_id}/streaming/sql-jobs● Parameter description





{ "name": "myjob", "desc": "This is a job used for counting characters.", "template_id": 100000, "queue_name": "testQueue", "sql_body": "select * from source_table", "run_mode": "shared_cluster", "cu_number": 2, "parallel_number": 1, "checkpoint_enabled": false, "checkpoint_mode": 1, "checkpoint_interval": 0, "obs_bucket": "my_obs_bucket", "log_enabled": false, "restart_when_exception": false, "idle_state_retention": 3600, "job_type": "flink_sql_job", "edge_group_ids": [ "62de1e1c-066e-48a8-a79d-f461a31b2ee1", "2eb00f85-99f2-4144-bcb7-d39ff47f9002" ], "dirty_data_strategy": "0", "udf_jar_url": "group/test.jar"}





Parameter Mandatory

Type Description

name Yes String Name of the job. Length range: 0 to 57characters.

desc No String Job description. Length range: 0 to 2,048characters.

template_id No Integer

Template ID.If both template_id and sql_body arespecified, sql_body prevails. If template_idis specified but sql_body is not, fill sql_bodywith the template_id value.

queue_name No String Name of a queue. Length range: 1 to 128characters.

sql_body No String Stream SQL statement, which includes atleast the following three parts: source, query,and sink. Length range: 1024x1024characters.

run_mode No String Job running mode. The options are asfollows:● shared_cluster: indicates that the job is

running on a shared cluster.● exclusive_cluster: indicates that the job

is running on an exclusive cluster.● edge_node: indicates that the job is

running on an edge node.The default value is shared_cluster.

cu_number No Integer

Number of CUs selected for a job. Thedefault value is 2.

parallel_number

No Integer

Number of parallel jobs set by a user. Thedefault value is 1.

checkpoint_enabled

No Boolean

Whether to enable the automatic jobsnapshot function.● true: indicates to enable the automatic

job snapshot function.● false: indicates to disable the automatic

job snapshot function.● Default value: false



Parameter Mandatory

Type Description

checkpoint_mode

No Integer

Snapshot mode. The values include:● 1: ExactlyOnce, indicates that data is

processed only once.● 2: AtLeastOnce, indicates that data is

processed at least once.The default value is 1.

checkpoint_interval

No Integer

Snapshot interval, in seconds. The defaultvalue is 10.

obs_bucket No String OBS path where users are authorized to savethe snapshot. This parameter is valid onlywhen checkpoint_enabled is set to true.OBS path where users are authorized to savethe snapshot. This parameter is valid onlywhen log_enabled is set to true.

log_enabled No Boolean

Whether to enable the function of uploadingjob logs to users' OBS buckets. The defaultvalue is false.

smn_topic No String SMN topic. If a job fails, the system will senda message to users subscribed to the SMNtopic.

restart_when_exception

No Boolean

Whether to enable the function ofautomatically restarting a job upon jobexceptions. The default value is false.

idle_state_retention

No Int Expiration time, in seconds. The defaultvalue is 3600.

job_type No String Job type. This parameter can be set toflink_sql_job or flink_sql_edge_job.● If run_mode is set to edge_node, this

parameter must be flink_sql_edge_job.● If run_mode is set to shared_cluster or

exclusive_cluster, this parameter mustbe flink_sql_job.

● The default value is flink_sql_job.

edge_group_ids

No String List of edge computing group IDs. Usecommas (,) to separate multiple IDs.



Parameter Mandatory

Type Description

dirty_data_strategy

No String Dirty data policy of a job.● 2:obs-wan-wulan3/jobs: Save.● 1: Trigger a job exception.● 0: Ignore.The default value is 0.

udf_jar_url No String Name of the resource package that has beenuploaded to the DLI resource managementsystem. The udf jar of the SQL job isspecified by this parameter.


{ "is_success": "true", "message": "A DLI job is created successfully.", "job": { "job_id": 148, "status_name": "job_init", "status_desc": "" }}



Parameter Mandatory

Type Description

is_success No String Whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.


job No object Information about the job status.

Table 7-9 job parameters

Parameter Mandatory

Type Description

job_id Yes Long Job ID.

status_name No String Name of job status.

status_desc No String Status description.



Status CodeTable 7-10 describes status codes.



200 The job is created successfully.

400 The input parameter is invalid.


Table 7-11 Error codes

Error Code Error Message

DLI.11032 The SMN topic does not exist. Select an available SMNtopic.

7.3 Updating a SQL Job

FunctionThe Stream SQL syntax is developed based on Apache Flink SQL. For details, seethe Data Lake Insight SQL Syntax Reference.

URI● URI format

PUT /v1.0/{project_id}/streaming/sql-jobs/{job_id}● Parameter description


Parameter

Mandatory Description


job_id Yes Job ID.




{ "name": "myjob", "desc": "My first job", "queue_name": "testQueue", "sql_body": "select * from source_table", "run_mode": "shared_cluster", "cu_number": 4, "parallel_number": 4, "checkpoint_enabled": false, "checkpoint_mode": 1, "checkpoint_interval": 10, "obs_bucket": "", "log_enabled": false, "smn_topic": "", "restart_when_exception": false, "idle_state_retention": 3600, "edge_group_ids": [ "62de1e1c-066e-48a8-a79d-f461a31b2ee1", "2eb00f85-99f2-4144-bcb7-d39ff47f9002" ], "dirty_data_strategy": "0", "udf_jar_url": "group/test.jar"}



Parameter Mandatory

Type Description

name No String Name of a job. Length range: 0to 57 characters.

desc No String Job description. Length range: 0to 2,048 characters.

queue_name No String Name of a queue. Length range:1 to 128 characters.

sql_body No String Stream SQL statement, whichincludes at least the followingthree parts: source, query, andsink. Length range: 0 to1024x1024 characters.



Parameter Mandatory

Type Description

run_mode No String Job running mode. The optionsare as follows:● shared_cluster: indicates that

the job is running on a sharedcluster.

● exclusive_cluster: indicatesthat the job is running on anexclusive cluster.

● edge_node: indicates that thejob is running on an edgenode.

The default value isshared_cluster.

cu_number No Integer Number of CUs selected for ajob. The default value is 2.

parallel_number No Integer Number of parallel jobs set by auser. The default value is 1.

checkpoint_enabled

No Boolean Whether to enable theautomatic job snapshot function.● true: indicates to enable the

automatic job snapshotfunction.

● false: indicates to disable theautomatic job snapshotfunction.

● Default value: false

checkpoint_mode

No Integer Snapshot mode. The valuesinclude:● 1: ExactlyOnce, indicates that

data is processed only once.● 2: AtLeastOnce, indicates

that data is processed at leastonce.

The default value is 1.

checkpoint_interval

No Integer Snapshot interval, in seconds.The default value is 10.



Parameter Mandatory

Type Description

obs_bucket No String OBS path where users areauthorized to save the snapshot.This parameter is valid onlywhen checkpoint_enabled is setto true.OBS path where users areauthorized to save the snapshot.This parameter is valid onlywhen log_enabled is set to true.

log_enabled No Boolean Whether to enable the functionof uploading job logs to users'OBS buckets. The default valueis false.

smn_topic No String SMN topic. If a job fails, thesystem will send a message tousers subscribed to the SMNtopic.


No Boolean Whether to enable the functionof automatically restarting a jobupon job exceptions. The defaultvalue is false.


No Int Expiration time, in seconds. Thedefault value is 3600.

edge_group_ids No String List of edge computing groupIDs. Use commas (,) to separatemultiple IDs.

dirty_data_strategy

No String Dirty data policy of a job.● 2:obs-wan-wulan3/jobs:

Save.● 1: Trigger a job exception.● 0: Ignore.The default value is 0.

udf_jar_url No String Name of the resource packagethat has been uploaded to theDLI resource managementsystem. The udf jar of the SQLjob is specified by thisparameter.




{ "is_success": "true", "message": "The job is updated successfully.", "job": { "update_time": 1578905682534 }}



Parameter

Mandatory

Type Description

is_success No Boolean Whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.


job No object Information about job update.


Parameter Mandatory

Type Description

update_time No Long Job update time, expressed bymilliseconds




200 The job is updated successfully.







DLI.12004 The job does not exist. Check the reason or create ajob.

7.4 Creating a Custom Flink Job

Function

This API is used to create custom jobs, which currently support the JAR format andrun in exclusive queues.

URI● URI format

POST /v1.0/{project_id}/streaming/flink-jobs● Parameter description


Parameter Mandatory

Description



{ "name": "test", "desc": "job for test", "queue_name": "testQueue", "manager_cu_number": 1, "cu_number": 2, "parallel_number": 1, "log_enabled": true, "obs_bucket": "bucketName", "smn_topic": "topic", "main_class": "org.apache.flink.examples.streaming.JavaQueueStream", "restart_when_exception": false, "entrypoint": "javaQueueStream.jar", "dependency_jars": [ "myGroup/test.jar", "myGroup/test1.jar" ], "dependency_files": [ "myGroup/test.csv", "myGroup/test1.csv" ]}



● Parameter descriptionFor details about the parameter configuration, see Table 7-19.


Parameter Mandatory

Type Description


desc Yes String Job description. Length range: 0 to 2,048characters.

queue_name No String Name of a queue. Length range: 1 to 128characters.


Number of CUs selected for a job.

manager_cu_number

No Integer

Number of CUs on the management nodeselected by the user for a job, whichcorresponds to the number of Flink jobmanagers. The default value is 1.

parallel_number

No Integer

Number of parallel operations selected for ajob.


Whether to enable the job log function.● true: indicates to enable the job log

function.● false: indicates to disable the job log

function.● Default value: false

obs_bucket No String OBS bucket where users are authorized tosave logs when log_enabled is set to true.

smn_topic No String SMN topic. If a job fails, the system willsend a message to users subscribed to theSMN topic.

main_class No String Job entry class.

entrypoint_args

No String Job entry class parameter.


No Boolean

Whether to enable the function of restartupon exceptions. The default value is false.



Parameter Mandatory

Type Description

entrypoint No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter is usedto customize the JAR file where the jobmain class is located.

dependency_jars

No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter is usedto customize other dependency packages.Example: myGroup/test.jar,myGroup/test1.jar.

dependency_files

No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter is usedto customize dependency files.Example: myGroup/test.cvs,myGroup/test1.csv.


{ "is_success": true, "message": "A Flink job is created successfully.", "job": { "job_id": 138, "status_name": "running", "status_desc": "Running" }}



Parameter Mandatory

Type Description

is_success No String Whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.


job No object Information about the job status.




Parameter Mandatory

Type Description

job_id Yes Long Job ID.

status_name No String Name of job status.

status_desc No String Status description.

Status Code

Table 7-22 describes status codes.



200 The custom Flink job is created successfully.


Error Code

If an error occurs when this API is invoked, the system does not return the resultsimilar to the preceding example, but returns the error code and error information.For details, see Error Code.


ErrorCode

Error Message

DLI.11016

The OBS bucket in the job request is not authorized to DLI. Grantthe OBS bucket permission to DLI.

7.5 Updating a Custom Flink Job

Function

This API is used to update custom jobs, which currently support the JAR formatand run in exclusive queues.

URI● URI format

PUT /v1.0/{project_id}/streaming/flink-jobs/{job_id}





Parameter Mandatory

Description


job_id Yes Job ID.


{ "name": "test1", "desc": "job for test", "job_type": "flink_jar_job", "queue_name": "testQueue", "manager_cu_number": 1, "cu_number": 2, "parallel_number": 1, "log_enabled": false, "main_class": "org.apache.flink.examples.streaming.JavaQueueStream", "restart_when_exception": false, "entrypoint": "FemaleInfoCollec.jar", "dependency_jars": [ "myGroup/test.jar", "myGroup/test1.jar" ], "dependency_files": [ "myGroup/test.csv", "myGroup/test1.csv" ]}


For details about the parameter configuration, see Table 7-25.


Parameter Mandatory

Type Description


desc Yes String Job description. Length range: 0 to 2,048characters.

queue_name No String Name of a queue. Length range: 1 to128 characters.


Number of CUs selected for a job. Thedefault value is 2.



Parameter Mandatory

Type Description

manager_cu_number

No Integer

Number of CUs on the managementnode selected by the user for a job,which corresponds to the number ofFlink job managers. The default value is1.

parallel_number

No Integer

Number of parallel operations selectedfor a job. The default value is 1.


Whether to enable the job log function.● true: indicates to enable the job log

function.● false: indicates to disable the job log

function.● Default value: false

obs_bucket No String OBS path where users are authorized tosave logs when log_enabled is set totrue.

smn_topic No String SMN topic. If a job fails, the system willsend a message to users subscribed tothe SMN topic.

main_class No String Job entry class.

entrypoint_args

No String Job entry class parameter.


No Boolean

Whether to enable the function ofrestart upon exceptions. The defaultvalue is false.

entrypoint No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter isused to customize the JAR file where thejob main class is located.

dependency_jars

No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter isused to customize other dependencypackages.Example: myGroup/test.jar,myGroup/test1.jar.



Parameter Mandatory

Type Description

dependency_files

No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter isused to customize dependency files.Example: myGroup/test.cvs,myGroup/test1.csv.


{ "is_success": true, "message": "The Flink job is updated successfully.", "job": { "update_time": 1516952770835 } }



Parameter

Mandatory

Type Description



job No object Information about job update.


Parameter Mandatory

Type Description

update_time No Long Job update time, expressed bymilliseconds






200 The custom Flink job is updated successfully.





DLI.12002 Failed to update the job. Cause: invalid resource /FemaleInfoCollec.jar, cannot find resource FemaleInfoCollec.jar.Check and try again.

7.6 Running Jobs in Batches

FunctionThis API is used to run jobs in batches.

URI● URI format

POST /v1.0/{project_id}/streaming/jobs/run● Parameter description


Parameter

Mandatory

Description



{ "job_ids": [ 131,



130, 138, 137 ], "resume_savepoint": true}



Parameter Mandatory

Type Description

job_ids Yes list[Long]

Batch job ID.

resume_savepoint

No Boolean Whether to restore a job fromthe latest savepoint.● If resume_savepoint is set to

true, the job is restored fromthe latest savepoint.

● If resume_savepoint is set tofalse, the job is startednormally, not from a specificsavepoint.

The default value is false.


[ { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." }, { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." }, { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." }, { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." }]





Parameter

Mandatory

Type Description

is_success

No Boolean Whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.

message

No String Message content.

Status Code




200 Jobs are successfully run in batches.


Error Code




DLI.13007 The job ID does not exist. Provide an available job ID, whichcan be obtained from the DLI job list.

7.7 Querying the Job List

Function

This API is used to query the list of the current user's jobs. You can set the job IDas the ID and query jobs whose IDs are greater than or less than the ID. You canalso query jobs in specific status, for example, in running status or other. Bydefault, all jobs are queried.



URI● URI format

GET /v1.0/{project_id}/streaming/jobs



Parameter Mandatory

Description



None



Parameter Mandatory

Type Description

job_type No String Job type.

limit No Integer Number of returned data records.The default value 10.

name No String Name of the job. Length range: 0 to57 characters.

offset No Integer Job offset.

order No String Sorting style of the query results.● asc: The query results are

displayed in ascending order.● desc: The query results are

displayed in the descending order.The default value is desc.

queue_name No String Name of a queue.

root_job_id No Integer Edge parent job ID, which is used toquery sub-jobs of a specified edgejob. If this parameter is not specified,all non-edge jobs and edge parentjobs, instead of edge sub-jobs, arequeried.



Parameter Mandatory

Type Description

show_detail No Boolean Whether to return job details. Thedefault value is false. If thisparameter is set to true, the jobdetails are returned. For details, seeQuerying Job Details.

show_job_type

No String Data type.● spark● flink● allThe default value is all.

status No String Job status code.Available job statuses are as follows:● Draft● Running● Idle● Completed● Stopped● Submitting● Submission failed● Running exception● Stopping● Stopping failed● Stopped due to arrears● Restoring (recharged jobs)● Creating the savepoint

user_name No String Username, which can be used as afilter.


{ "is_success": "true", "message": "Querying of the job list succeeds.", "job_list": { "total_count": 7, "jobs": [ { "job_id": 146, "user_id": "011c99a26ae84a1bb963a75e7637d3fd", "queue_name": "default_general", "project_id": "330e068af1334c9782f4226acc00a2e2", "name": "aaaaa", "desc": "",



"sql_body": "", "run_mode": "shared_cluster", "job_type": "flink_sql_job", "job_config": { "checkpoint_enabled": false, "checkpoint_interval": 10, "checkpoint_mode": "exactly_once", "log_enabled": false, "obs_bucket": null, "root_id": -1, "edge_group_ids": null, "graph_editor_enabled": false, "graph_editor_data": "", "manager_cu_number": 1, "executor_number": null, "executor_cu_number": null, "cu_number": 2, "parallel_number": 1, "smn_topic": null, "restart_when_exception": false, "idle_state_retention": 3600, "config_url": null, "udf_jar_url": null, "dirty_data_strategy": null, "entrypoint": null, "dependency_jars": null, "dependency_files": null }, "main_class": null, "entrypoint_args": null, "execution_graph": null, "savepoint_path": null, "status": "job_init", "status_desc": "", "create_time": 1578892414688, "update_time": 1578892414688, "start_time": null } ] }}



Parameter

Mandatory

Type Description

is_success



job_list No Array ofObject

Information about a job list. For details,see Table 7-38.



Table 7-38 Job_list parameters

Parameter

Mandatory

Type Description

total_count

No Integer Number of records in the queryresult.

jobs No Array ofObject

Information about a job. Fordetails, see Table 7-39.

Table 7-39 Jobs parameters

Parameter

Mandatory

Type Description

job_id No Long Job ID.

name No String Name of the job. Length range: 0 to 57characters.

desc No String Job description. Length range: 0 to 2,048characters.

username No String Username. This parameter is valid onlywhen show_detail is set to false.


status No String Job status.

status_desc

No String Description of job status.

create_time

No Long Time when a job is created.

start_time

No Long Time when a job is started. 0 indicates thata job has not been started.

duration No Integer Job running duration, expressed bymilliseconds. This parameter is valid onlywhen show_detail is set to false.

root_id No Integer Parent job ID. This parameter is valid onlywhen show_detail is set to false.

edge_group_ids

No String List of edge computing group IDs. Usecommas (,) to separate multiple IDs. Thisparameter is valid only when show_detail isset to false.

user_id No String ID of the user who creates the job. Thisparameter is valid only when show_detail isset to true.



Parameter

Mandatory

Type Description

project_id No String ID of the project to which a job belongs.This parameter is valid only whenshow_detail is set to true.

sql_body No String Stream SQL statement. This parameter isvalid only when show_detail is set to true.

run_mode No String Job running mode. This parameter is validonly when show_detail is set to true.● shared_cluster: indicates that the job is

running on a shared cluster.● exclusive_cluster: indicates that the job

is running on an exclusive cluster.● edge_node: indicates that the job is

running on an edge node.

job_config

No Array ofObject

Job configuration. This parameter is validonly when show_detail is set to true. Fordetails, see Table 7-40.

main_class

No String OBS path where users are authorized tosave the snapshot whencheckpoint_enabled is set to true. Thisparameter is valid only when show_detail isset to true.

args No String Job running parameter of the JAR file. Thisparameter is valid only when show_detail isset to true.

execution_graph

No String Job execution plan. This parameter is validonly when show_detail is set to true.

update_time

No Long Job update time. This parameter is validonly when show_detail is set to true.



Table 7-40 job_config parameters

Parameter Mandatory

Type Description

checkpoint_enabled

No Boolean

Whether to enable the automatic jobsnapshot function.● true: The automatic job snapshot

function is enabled.● false: The automatic job snapshot

function is disabled.The default value is false.

checkpoint_mode

No String Snapshot mode. There are two options:● 1: exactly_once, indicates that data is

processed only once.● 2: at_least_once, indicates that data is


checkpoint_interval

No Integer Snapshot interval, expressed by seconds.The default value is 10.


Whether to enable the log storagefunction. The default value is false.

obs_bucket No String Name of an OBS bucket.

smn_topic No String SMN topic name.

root_id No Integer Parent job ID.

edge_group_ids

No list[String]

List of edge computing group IDs that areseparated by commas (,).

manager_cu_number

No Integer Number of CUs of the management unit.The default value is 1.

cu_number No Integer Number of CUs selected by a user for ajob. This parameter is valid only whenshow_detail is set to true.● Minimum value: 2● Maximum value: 400The default value is 2.

parallel_number

No Integer Number of parallel jobs set by a user. Thisparameter is unique when show_detail isset to true.● Minimum value: 1● Maximum value: 2,000The default value is 1.



Parameter Mandatory

Type Description


No Boolean

Whether to enable the function of restartupon exceptions.


No Integer Expiration time.

udf_jar_url No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. The udf jar of theSQL job is specified by this parameter.

dirty_data_strategy

No String Dirty data policy of a job.2:obs-wan-wulan3/jobs: Save1: Trigger a job exception0: Ignore


dependency_jars

No List<string>

Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter isused to customize other dependencypackages.

dependency_files

No List<string>

Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter isused to customize dependency files.

executor_number

No Integer Number of compute nodes in a job.

executor_cu_number

No Integer Number of CUs in a compute node.

Status Code



200 Job list query succeeds.






ErrorCode

Error Message

DLI.23001 Failed to query the job list. The offset parameter is incorrect. Setthe offset parameter to a value greater than or equal to 0.

7.8 Querying Job Details

FunctionThis API is used to query details of a job. In the API for querying the job list, ifshow_detail is set to true, the job details are returned.

URI● URI format

GET /v1.0/{project_id}/streaming/jobs/{job_id}● Parameter description


Parameter

Mandatory

Description


job_id Yes Job ID.

RequestNone


{ "is_success": "true", "message": "Querying of the job details succeeds.", "job_detail": { "job_id": 104, "user_id": "011c99a26ae84a1bb963a75e7637d3fd",



"queue_name": "flinktest", "project_id": "330e068af1334c9782f4226acc00a2e2", "name": "jptest", "desc": "", "sql_body": "", "run_mode": "exclusive_cluster", "job_type": "flink_jar_job", "job_config": { "checkpoint_enabled": false, "checkpoint_interval": 10, "checkpoint_mode": "exactly_once", "log_enabled": false, "obs_bucket": null, "root_id": -1, "edge_group_ids": null, "graph_editor_enabled": false, "graph_editor_data": "", "manager_cu_number": 1, "executor_number": null, "executor_cu_number": null, "cu_number": 2, "parallel_number": 1, "smn_topic": null, "restart_when_exception": false, "idle_state_retention": 3600, "config_url": null, "udf_jar_url": null, "dirty_data_strategy": null, "entrypoint": "FemaleInfoCollection.jar", "dependency_jars": [ "FemaleInfoCollection.jar", "S3aBatchTest.jar" ], "dependency_files": [ "FemaleInfoCollection.jar", "ReadFromResource" ] }, "main_class": null, "entrypoint_args": null, "execution_graph": null, "savepoint_path": null, "status": "job_init", "status_desc": "", "create_time": 1578466221525, "update_time": 1578467395713, "start_time": null }}



Parameter

Mandatory

Type Description


message No String System prompt. If execution succeeds,the parameter setting may be left blank.

job_detail No Object Job details. For details, see Table 7-45.



Table 7-45 Job_detail parameters

Parameter Mandatory

Type Description


name No String Name of the job. Length range: 0 to57 characters.

desc No String Job description. Length range: 0 to2,048 characters.


status No String Job status.

status_desc No String Description of job status.

create_time No Long Time when a job is created.

start_time No Long Time when a job is started. 0 indicatesthat a job has not been started.

user_id No String ID of the user who creates the job.This parameter is valid only whenshow_detail is set to true.

queue_name

No String Name of a queue. Length range: 1 to128 characters.

project_id No String Stream SQL statement. Thisparameter is used only whenshow_detail is set to true.

sql_body No String Stream SQL statement. Thisparameter is used only whenshow_detail is set to true.

run_mode No String Job running mode. This parameter isvalid only when show_detail is set totrue.● shared_cluster: indicates that the

job is running on a shared cluster.● exclusive_cluster: indicates that

the job is running on an exclusivecluster.

● edge_node: indicates that the jobis running on an edge node.

job_config No Object Job configuration. This parameter isvalid only when show_detail is set totrue. For details, see Table 7-46.



Parameter Mandatory

Type Description

main_class No String OBS path where users are authorizedto save the snapshot whencheckpoint_enabled is set to true.This parameter is valid only whenshow_detail is set to true.

entrypoint_args

No String Job running parameter of the JARpackage. This parameter is used onlywhen show_detail is set to true.

execution_graph

No String Job execution plan. This parameter isused only when show_detail is set totrue.

update_time

No Long Job update time. This parameter isused only when show_detail is set totrue.

Table 7-46 job_config parameters

Parameter Mandatory

Type Description

checkpoint_enabled

No Boolean

Whether to enable the automatic jobsnapshot function.● true: The automatic job snapshot

function is enabled.● false: The automatic job snapshot

function is disabled.The default value is false.

checkpoint_mode

No String Snapshot mode. There are two options:● 1: exactly_once, indicates that data is

processed only once.● 2: at_least_once, indicates that data is


checkpoint_interval

No Integer Snapshot interval, expressed by seconds.The default value is 10.


Whether to enable the log storagefunction. The default value is false.

obs_bucket No String Name of an OBS bucket.

smn_topic No String SMN topic name.



Parameter Mandatory

Type Description

root_id No Integer Parent job ID.

edge_group_ids

No list[String]

List of edge computing group IDs that areseparated by commas (,).

manager_cu_number

No Integer Number of CUs of the management unit.The default value is 1.

cu_number No Integer Number of CUs selected by a user for ajob. This parameter is valid only whenshow_detail is set to true.● Minimum value: 2● Maximum value: 400The default value is 2.

parallel_number

No Integer Number of parallel jobs set by a user. Thisparameter is unique when show_detail isset to true.● Minimum value: 1● Maximum value: 2,000The default value is 1.


No Boolean

Whether to enable the function of restartupon exceptions.


No Integer Expiration time.

udf_jar_url No String Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. The udf jar of theSQL job is specified by this parameter.

dirty_data_strategy

No String Dirty data policy of a job.2:obs-wan-wulan3/jobs: Save1: Trigger a job exception0: Ignore


dependency_jars

No List<string>

Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter isused to customize other dependencypackages.



Parameter Mandatory

Type Description

dependency_files

No List<string>

Name of the resource package that hasbeen uploaded to the DLI resourcemanagement system. This parameter isused to customize dependency files.

executor_number

No Integer Number of compute nodes in a job.

executor_cu_number

No Integer Number of CUs in a compute node.

Status Code



200 Querying details of a job succeeds.





DLI.16001 The job ID does not exist. Provide an available job ID, which canbe obtained from the DLI job list.

7.9 Querying the Job Execution Plan

FunctionThis API is used to query the job execution plan.

URI● URI format

GET /v1.0/{project_id}/streaming/jobs/{job_id}/execute-graph





Parameter

Mandatory

Description

project_id


job_id Yes Job ID.

RequestNone


{ "is_success": "true", "message": "Querying the job execution graph succeeds.", "execute_graph": { "jid": "4e966f43f2c90b0e1bf3188ecf55504b", "name": "", "isStoppable": false, "state": "RUNNING", "start-time": 1578904488436, "end-time": -1, "duration": 516274, "now": 1578905004710, "timestamps": { "FAILING": 0, "SUSPENDING": 0, "FAILED": 0, "FINISHED": 0, "RESTARTING": 0, "RUNNING": 1578904488665, "CANCELED": 0, "SUSPENDED": 0, "CREATED": 1578904488436, "CANCELLING": 0, "RECONCILING": 0 }, "vertices": [ { "id": "bc764cd8ddf7a0cff126f51c16239658", "name": "Source: DIS_128_CAR_INFOS", "parallelism": 1, "status": "RUNNING", "start-time": 1578904505130, "end-time": -1, "duration": 499580, "tasks": { "SCHEDULED": 0, "RUNNING": 1, "RECONCILING": 0, "FINISHED": 0, "DEPLOYING": 0, "CREATED": 0, "CANCELING": 0, "FAILED": 0,



"CANCELED": 0 }, "metrics": { "read-bytes": 0, "read-bytes-complete": true, "write-bytes": 0, "write-bytes-complete": true, "read-records": 0, "read-records-complete": true, "write-records": 50, "write-records-complete": true }, "backpressure-level": "ok" }, { "id": "6cdc5bb954874d922eaee11a8e7b5dd5", "name": "Source: Custom Source -> select: (_UTF-16LE'1' AS CAR_ID, _UTF-16LE'lilei' AS CAR_OWNER, _UTF-16LE'bmw320i' AS CAR_BRAND, 28 AS CAR_PRICE) -> to: Tuple2 -> Flat Map", "parallelism": 1, "status": "FINISHED", "start-time": 1578904505138, "end-time": 1578904506009, "duration": 871, "tasks": { "SCHEDULED": 0, "RUNNING": 0, "RECONCILING": 0, "FINISHED": 1, "DEPLOYING": 0, "CREATED": 0, "CANCELING": 0, "FAILED": 0, "CANCELED": 0 }, "metrics": { "read-bytes": 0, "read-bytes-complete": true, "write-bytes": 0, "write-bytes-complete": true, "read-records": 0, "read-records-complete": true, "write-records": 1, "write-records-complete": true } }, { "id": "2963852293169ba90d9d1e7d6308db5c", "name": "Source: Custom Source -> select: (_UTF-16LE'2' AS CAR_ID, _UTF-16LE'hanmeimei' AS CAR_OWNER, _UTF-16LE'audia4' AS CAR_BRAND, 27 AS CAR_PRICE) -> to: Tuple2 -> Flat Map", "parallelism": 1, "status": "FINISHED", "start-time": 1578904505140, "end-time": 1578904506017, "duration": 877, "tasks": { "SCHEDULED": 0, "RUNNING": 0, "RECONCILING": 0, "FINISHED": 1, "DEPLOYING": 0, "CREATED": 0, "CANCELING": 0, "FAILED": 0, "CANCELED": 0 }, "metrics": { "read-bytes": 0, "read-bytes-complete": true,



"write-bytes": 0, "write-bytes-complete": true, "read-records": 0, "read-records-complete": true, "write-records": 1, "write-records-complete": true } }, { "id": "c9235a26195589826000b27f7d761f13", "name": "Map -> where: (AND(LIKE(CAR_BRAND, _UTF-16LE'audi%'), <(CAR_PRICE, 30))), select: (CAR_ID, CAR_OWNER, CAR_BRAND, CAR_PRICE) -> to: Tuple2 -> Flat Map", "parallelism": 1, "status": "RUNNING", "start-time": 1578904505141, "end-time": -1, "duration": 499569, "tasks": { "SCHEDULED": 0, "RUNNING": 1, "RECONCILING": 0, "FINISHED": 0, "DEPLOYING": 0, "CREATED": 0, "CANCELING": 0, "FAILED": 0, "CANCELED": 0 }, "metrics": { "read-bytes": 1495, "read-bytes-complete": true, "write-bytes": 0, "write-bytes-complete": true, "read-records": 50, "read-records-complete": true, "write-records": 49, "write-records-complete": true }, "backpressure-level": "ok" }, { "id": "15262ed2e480542a546ce9514170c8bf", "name": "Sink: AUDI_CHEAPER_THAN_30W", "parallelism": 1, "status": "RUNNING", "start-time": 1578904505276, "end-time": -1, "duration": 499434, "tasks": { "SCHEDULED": 0, "RUNNING": 1, "RECONCILING": 0, "FINISHED": 0, "DEPLOYING": 0, "CREATED": 0, "CANCELING": 0, "FAILED": 0, "CANCELED": 0 }, "metrics": { "read-bytes": 2695, "read-bytes-complete": true, "write-bytes": 0, "write-bytes-complete": true, "read-records": 49, "read-records-complete": true, "write-records": 0, "write-records-complete": true },



"backpressure-level": "ok" }, { "id": "6a739966c5220e4a0f256fe41d02793b", "name": "Sink: CAR_INFO_DATA", "parallelism": 1, "status": "FINISHED", "start-time": 1578904505277, "end-time": 1578904506843, "duration": 1566, "tasks": { "SCHEDULED": 0, "RUNNING": 0, "RECONCILING": 0, "FINISHED": 1, "DEPLOYING": 0, "CREATED": 0, "CANCELING": 0, "FAILED": 0, "CANCELED": 0 }, "metrics": { "read-bytes": 56, "read-bytes-complete": true, "write-bytes": 0, "write-bytes-complete": true, "read-records": 1, "read-records-complete": true, "write-records": 0, "write-records-complete": true } }, { "id": "ac1799ab18d143a12ca3550fb1dfc3a5", "name": "Sink: CAR_INFO_DATA", "parallelism": 1, "status": "FINISHED", "start-time": 1578904505279, "end-time": 1578904506711, "duration": 1432, "tasks": { "SCHEDULED": 0, "RUNNING": 0, "RECONCILING": 0, "FINISHED": 1, "DEPLOYING": 0, "CREATED": 0, "CANCELING": 0, "FAILED": 0, "CANCELED": 0 }, "metrics": { "read-bytes": 59, "read-bytes-complete": true, "write-bytes": 0, "write-bytes-complete": true, "read-records": 1, "read-records-complete": true, "write-records": 0, "write-records-complete": true } } ], "status-counts": { "SCHEDULED": 0, "RUNNING": 3, "RECONCILING": 0, "FINISHED": 4, "DEPLOYING": 0,



"CREATED": 0, "CANCELING": 0, "FAILED": 0, "CANCELED": 0 }, "jmx-metric-port": -1, "plan": { "jid": "4e966f43f2c90b0e1bf3188ecf55504b", "name": "", "nodes": [ { "id": "15262ed2e480542a546ce9514170c8bf", "parallelism": 1, "operator": "", "operator_strategy": "", "description": "Sink: AUDI_CHEAPER_THAN_30W", "inputs": [ { "num": 0, "id": "c9235a26195589826000b27f7d761f13", "ship_strategy": "FORWARD", "exchange": "pipelined_bounded" } ], "optimizer_properties": {} }, { "id": "c9235a26195589826000b27f7d761f13", "parallelism": 1, "operator": "", "operator_strategy": "", "description": "Map -> where: (AND(LIKE(CAR_BRAND, _UTF-16LE'audi%'), <(CAR_PRICE, 30))), select: (CAR_ID, CAR_OWNER, CAR_BRAND, CAR_PRICE) -> to: Tuple2 -> Flat Map", "inputs": [ { "num": 0, "id": "bc764cd8ddf7a0cff126f51c16239658", "ship_strategy": "FORWARD", "exchange": "pipelined_bounded" } ], "optimizer_properties": {} }, { "id": "bc764cd8ddf7a0cff126f51c16239658", "parallelism": 1, "operator": "", "operator_strategy": "", "description": "Source: DIS_128_CAR_INFOS", "optimizer_properties": {} }, { "id": "6a739966c5220e4a0f256fe41d02793b", "parallelism": 1, "operator": "", "operator_strategy": "", "description": "Sink: CAR_INFO_DATA", "inputs": [ { "num": 0, "id": "6cdc5bb954874d922eaee11a8e7b5dd5", "ship_strategy": "FORWARD", "exchange": "pipelined_bounded" } ], "optimizer_properties": {} }, {



"id": "6cdc5bb954874d922eaee11a8e7b5dd5", "parallelism": 1, "operator": "", "operator_strategy": "", "description": "Source: Custom Source -> select: (_UTF-16LE'1' AS CAR_ID, _UTF-16LE'lilei' AS CAR_OWNER, _UTF-16LE'bmw320i' AS CAR_BRAND, 28 AS CAR_PRICE) -> to: Tuple2 -> Flat Map", "optimizer_properties": {} }, { "id": "ac1799ab18d143a12ca3550fb1dfc3a5", "parallelism": 1, "operator": "", "operator_strategy": "", "description": "Sink: CAR_INFO_DATA", "inputs": [ { "num": 0, "id": "2963852293169ba90d9d1e7d6308db5c", "ship_strategy": "FORWARD", "exchange": "pipelined_bounded" } ], "optimizer_properties": {} }, { "id": "2963852293169ba90d9d1e7d6308db5c", "parallelism": 1, "operator": "", "operator_strategy": "", "description": "Source: Custom Source -> select: (_UTF-16LE'2' AS CAR_ID, _UTF-16LE'hanmeimei' AS CAR_OWNER, _UTF-16LE'audia4' AS CAR_BRAND, 27 AS CAR_PRICE) -> to: Tuple2 -> Flat Map", "optimizer_properties": {} } ] } }}



Parameter Mandatory

Type Description

is_success No Boolean Whether the request issuccessful.


execute_graph

No ShowJobPlanResp_plan

Response parameter forquerying a job plan.

Table 7-51 execute_graph parameters

Parameter Mandatory

Type Description

jid No String ID of a Flink job.



Parameter Mandatory

Type Description

name No String Name of a Flink job.

isStoppable

No Boolean Whether a job can be stopped.

state No String Execution status of a job.

start-time No Long Time when a job is started.

end-time No Long Time when a job is stopped.

duration No Long Running duration of a job.

Status Code



200 Querying the job execution plan succeeds.





DLI.14001 The job execution graph does not exist.

7.10 Querying Job Monitoring Information

FunctionThis API is used to query job monitoring information. You can query monitoringinformation about multiple jobs at the same time.

URI● URI format

POST /v1.0/{project_id}/streaming/jobs/metrics





Parameter

Mandatory

Description

project_id


RequestThe request parameter setting is in JSON format.

● Example request{ "job_ids": [298765, 298766]}



Parameter Mandatory

Type Description

job_ids Yes list[Long]

List of job IDs.


{ "is_success": true, "message": "Message content", "metrics": { "jobs": [ { "job_id": 0, "metrics": { "sources": [ { "name": "Source: KafKa_6070_KAFKA_SOURCE", "records": 0, "corrupted_records": 0 } ], "sinks": [ { "name": "Source: KafKa_6070_KAFKA_SOURCE", "records": 0, "corrupted_records": 0 } ], "total_read_rate": 100, "total_write_rate": 100 } }



] }}



Parameter Mandatory

Type Description


Indicates whether the request issuccessful.


payload No Object Information about a job list.

Table 7-57 payload parameters

Parameter Mandatory Type Description

jobs No Object Monitoring information aboutall jobs.

Table 7-58 jobs parameters

Parameter Mandatory

Type Description


metrics No Object All input and output monitoringinformation about a job.

Table 7-59 metrics parameters

Parameter Mandatory

Type Description

sources No Object All source streams. For details, seeTable 7-60.

sinks No Object All sink streams. For details, see Table7-60.

total_read_rate

No Double Total read rate.

total_write_rate

No Double Total write rate.



Table 7-60 source/sinks parameters

Parameter Mandatory

Type Description

name No String Name of the source or sink stream.

records No Long Total number of records.

corrupted_records

No Long Number of dirty data records.

Status Code



200 The query of job monitoring information succeeds.


Error Code




DLI.16001 The job ID does not exist. Provide an available job ID,which can be obtained from the DLI job list.

7.11 Querying the APIG Address of a Job

Function

This API is used to query the APIG address of a job.

URI● URI format

GET /v1.0/{project_id}/streaming/jobs/{job_id}/apig-sinks





Parameter

Mandatory

Description

project_id

Yes Project ID, which is used for resource isolation. Fordetails about how to obtain its value, seeObtaining a Project ID.

job_id Yes Job ID.

RequestNone


{ "is_success": "true", "message": "Querying of the APIG address of a job succeeds." "apig_sinks": { "sinks": [ { "app_id": "dc7a574571bb4749bdab401a004601c4", "name": "OVER_SPEED_WARNING", "url": "https://ae67828f8f0c4d42b419a1320252f1b7.apigw.cn-north-7.myhuaweicloud.com/154/OVER_SPEED_WARNING", "encode": "csv", "delimiter": ",", "attrs": [ { "name": "OVER_SPEED_MESSAGE", "type": "STRING" } ] } ] }}



Parameter Mandatory

Type Description

is_success Yes Boolean Indicates whether the request issuccessful.

message Yes String Message content.

apig_sinks No Object Information about the job audit log.



Table 7-65 apig_sinks parameters

Parameter Mandatory

Type Description

app_id No String Application ID output by the sinkstream of the job.

name No String Name of the sink stream of the job.

url No String APIG address of the sink stream ofthe job.

encode No String Format of the sink stream of thejob, for example, CSV.

delimiter No String Delimiter of the sink stream of thejob, for example, comma (,).

attrs No None Attribute information of the sinkstream of the job. For details, seeTable 7-66.

Table 7-66 attrs parameters

Parameter Mandatory

Type Description

name No String Attribute name of the sink stream ofthe job.

type No String Attribute type of the sink stream ofthe job.

Status Code



200 Querying of the APIG address of a job succeeds.


Error Code





ErrorCode

Error Message

DLI.11098

Failed to obtain the APIG address of a job. Cause: Not an existed sqljob or not running. Try again later.

7.12 Stopping Jobs in Batches

FunctionThis API is used to stop running jobs in batches.

URI● URI format

POST /v1.0/{project_id}/streaming/jobs/stop● Parameter description


Parameter

Mandatory

Description

project_id



{ "job_ids": [128, 137], "trigger_savepoint": false}



Parameter

Mandatory

Type Description

job_ids Yes list[Long] Job ID.



Parameter

Mandatory

Type Description

trigger_savepoint

No Boolean Whether to create a savepoint for a job tostore the job status information beforestopping it. The data type is Boolean.● If this parameter is set to true, a

savepoint is created.● If this parameter is set to false, no

savepoint is created. The default valueis false.


{ "is_success": "true", "message": "The request for stopping DLI jobs is delivered successfully."}



Parameter Mandatory

Type Description

is_success No Boolean Whether the request issuccessfully executed. Value trueindicates that the request issuccessfully executed.

message No String System prompt. If executionsucceeds, the parameter settingmay be left blank.

Status Code




200 The request of stopping a job is sent successfully.




Error Code




DLI.15002 A job in the current state cannot be stopped or the savepointcannot be triggered. For details, see the Data Lake Insight APIReference.

7.13 Deleting a Job

Function

This API is used to delete a job in any status.

URI● URI format

DELETE /v1.0/{project_id}/streaming/jobs/{job_id}● Parameter description


Parameter

Mandatory

Description

project_id


job_id Yes Job ID.

Request

None


{ "is_success": "true", "message": "The job is deleted successfully.",}





Parameter

Mandatory

Type Description


message No String System prompt. If execution succeeds,the parameter setting may be leftblank.




200 The job is deleted successfully.





DLI.17001 The job ID does not exist. Provide an available job ID, which can beobtained from the DLI job list.

7.14 Deleting Jobs in Batches

FunctionThis API is used to delete jobs in any status in batches.

URI● URI format

POST /v1.0/{project_id}/streaming/jobs/delete





Parameter

Mandatory

Description

project_id



{ "job_ids":[12,232]}



Parameter

Mandatory Type Description

job_ids Yes list[Long] Job ID.


{ "is_success": "true", "message": "The job is deleted successfully.",},{ "is_success": "true", "message": "The job is deleted successfully.",}



Parameter

Mandatory

Type Description

is_success








200 The job is deleted successfully.





DLI.17001 The job ID does not exist. Provide an available job ID, which can beobtained from the DLI job list.



8 APIs Related to Spark jobs

8.1 Session-related APIs

8.1.1 Creating a Session (Recommended)

FunctionThis API is used to create a session in a queue in the project.

URI● URI format

POST /v2.0/{project_id}/sessions● Parameter description


Parameter

Mandatory

Description

project_id


Data Lake InsightAPI Reference 8 APIs Related to Spark jobs


Request


Parameter Mandatory

Type Description

kind Yes String Session type. Only Spark parameters aresupported. Spark sessions support Scala jobs.

queue Yes String Queue name. Set this parameter to the nameof the created DLI queue.NOTE

● This parameter is compatible with thecluster_name parameter. That is, ifcluster_name is used to specify a queue, thequeue is still valid.

● The queue parameter is recommended. If bothqueue and cluster_name exist, the value ofqueue prevails.

cluster_name No String Queue name. Set this parameter to thecreated DLI queue name.

sc_type No String Compute resource type. Currently, resourcetypes A, B, and C are available. If thisparameter is not specified, the minimumconfiguration (type A) is used. For detailsabout resource types, see Table 8-5.

jars No Arrayofstrings

Name of the package that is of the JAR typeand has been uploaded to the DLI resourcemanagement system.

pyFiles No Arrayofstrings

Name of the package that is of the PyFiletype and has been uploaded to the DLIresource management system.

files No Arrayofstrings

Name of the package that is of the file typeand has been uploaded to the DLI resourcemanagement system.



Parameter Mandatory

Type Description

modules No Arrayofstrings

Name of the dependent system resourcemodule. You can view the module name usingthe API related to Querying All ResourcePackages.DLI provides dependencies for executingdatasource jobs. The following table lists thedependency modules corresponding todifferent services.● CloudTable/MRS HBase:

sys.datasource.hbase● CloudTable/MRS OpenTSDB:

sys.datasource.opentsdb● RDS MySQL: sys.datasource.rds● RDS Postgre: preset● DWS: preset● CSS: sys.datasource.css

resources No Arrayofstrings

JSON object list, including the name and typeof the JSON package that has been uploadedto the DLI resource management system.

groups No Arrayofstrings

JSON object list, including the package groupresource. For details about the format, see therequest example.

conf No Arrayofobjects

Session configuration items.

name No String Session name. The value contains a maximumof 128 characters.

heartbeatTimeoutInSecond

No Integer

Unit for the timeout interval for disabling asession after the session is idle for a longtime. The unit is second.

driverMemory No String Driver memory of the Spark application, forexample, 2 GB and 2048 MB. Thisconfiguration item replaces the defaultparameter in sc_type. The unit must beprovided. Otherwise, the startup fails.

driverCores No Integer

Number of CPU cores of the Spark applicationdriver. This configuration item replaces thedefault parameter in sc_type.



Parameter Mandatory

Type Description

executorMemory

No String Executor memory of the Spark application, forexample, 2 GB and 2048 MB. Thisconfiguration item replaces the defaultparameter in sc_type. The unit must beprovided. Otherwise, the startup fails.

executorCores No Integer

Number of CPU cores of each Executor in theSpark application. This configuration itemreplaces the default parameter in sc_type.

numExecutors No Integer

Number of Executors in a Spark application.This configuration item replaces the defaultparameter in sc_type.

Response

Table 8-3 Response parameter description

Parameter Mandatory

Type Description

id No String Session ID, which is a UUID string.

appId No String Application ID of a session.

name No String Name of a session.

owner No String User to which a session belongs.

proxyUser No String Proxy user (resource tenant) to which thesession belongs.

kind No String Session type. Only Spark parameters aresupported.

log No Array ofstrings

Session log information.

state No String Session status. For details, see Table 8-4.

appInfo No Array ofobjects

Details about an application.

sc_type No String Type of a compute resource. For details aboutresource types, see Table 8-5. If thecomputing resource type is customized, valueCUSTOMIZED is returned.

cluster_name

No String Queue to which a session belongs.



Parameter Mandatory

Type Description

queue No String Queue to which a session belongs.

create_time No Long Time when a session is set up. Unit: ms.

update_time

No Long Time when a session is updated. Unit: ms.

Table 8-4 Session status parameters


not_started Not started

starting A batch processing job is starting.

idle Idle

busy The statement is being executed.

shutting_down Stopping

error An error occurs.

dead A user has exited a session.

success A session successfully stops.

recovering Recovering

Table 8-5 Resource types

ResourceType

PhysicalResource

driverCores

executorCores

driverMemory

executorMemory

numExecutor

A 8 vCPUs,32-GBmemory

2 1 7 GB 4 GB 6

B 16vCPUs,64-GBmemory

2 2 7 GB 8 GB 7

C 32vCPUs,128-GBmemory

4 2 15 GB 8 GB 14




{ "kind": "spark", "cluster_name": "test_cluster", "sc_type": "A", "jars": [ "group.tesddws/gsjdbc4.jar" ], "resources": [ { "name": "group.tesddws/gsjdbc3.jar", "type": "jar" } ], "groups": [ { "name": "group.tesddws", "resources": [ { "name": "gsjdbc2.jar", "type": "jar" } ] } ]}

● Example response (successful request){ "id": "f28a0d0c-0a37-4bd4-8d1d-1416031e5151", "appId": "", "name": "", "owner": "", "proxyUser": "", "state": "starting", "kind": "spark", "appInfo": { "driverLogUrl": null, "sparkUiUrl": null }, "log": [ "stdout: ", "\nstderr: ", "\nYARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test_queue", "queue":"test_queue", "create_time": 1531906043036, "update_time": 0}


8.1.2 Creating a Session (Discarded)

Function

This API is used to create a session in a queue of a project.



URI● URI format

POST /v2.0/{project_id}/sessions● Parameter description


Parameter

Mandatory

Description

project_id


Request


Parameter Mandatory

Type Description

kind Yes String Session type. Only Spark parameters aresupported. Spark sessions support Scala jobs.

cluster_name Yes String Cluster name. Set this parameter to thecluster name of the DLI cluster that has beencreated.

sc_type No String Computing resource type. Currently, resourcetypes A, B, and C are available. If thisparameter is not specified, the minimumconfiguration (type A) is used. For detailsabout resource types, see Table 8-10.

jars No Array ofstrings

Name of the package that is of the JAR typeand has been uploaded to the DLI resourcemanagement system.

pyFiles No Array ofstrings


files No Array ofstrings

Name of the package that is of the file typeand has been uploaded to the DLI resourcemanagement system.

modules No Array ofstrings

Name of the dependent system resourcemodule. You can view the module name usingthe API related to Querying All ResourcePackages.



Parameter Mandatory

Type Description

resources No Array ofstrings

JSON object list, including the name and typeof the JSON package that has been uploadedto the DLI resource management system.

groups No Array ofstrings

JSON object list, including the user groupresource. For details about the format, see therequest example.


Session configuration items.

name No String Session name. The value contains a maximumof 128 characters.

heartbeatTimeoutInSecond

No Integer Unit for the timeout interval for disabling asession after the session is idle for a longtime. The unit is second.

driverMemory No String Driver memory of the Spark application, forexample, 2 GB and 2048 MB. Thisconfiguration item replaces the defaultparameter in sc_type. The unit must beprovided. Otherwise, the startup fails.

driverCores No Integer Number of CPU cores of the Spark applicationdriver. This configuration item replaces thedefault parameter in sc_type.

executorMemory

No String Executor memory of the Spark application, forexample, 2 GB and 2048 MB. Thisconfiguration item replaces the defaultparameter in sc_type. The unit must beprovided. Otherwise, the startup fails.

executorCores No Integer Number of CPU cores of each Executor in theSpark application. This configuration itemreplaces the default parameter in sc_type.

numExecutors No Integer Number of Executors in a Spark application.This configuration item replaces the defaultparameter in sc_type.



Response


Parameter Mandatory

Type Description

id No String Session ID, which is a UUID string.

appId No String Application ID of a session.

name No String Name of a session.

owner No String User to which a session belongs.

proxyUser No String Proxy user (resource tenant) to which thesession belongs.

kind No String Session type. Only Spark parameters aresupported.

log No Arrayofstrings

Session log information.

state No String Session status. For details, see Table 8-9.

appInfo No Arrayofobjects

Details about an application.

sc_type No String Type of a computing resource. For detailsabout resource types, see Table 8-10. If thecomputing resource type is customized,value CUSTOMIZED is returned.

cluster_name

No String Cluster where the session resides.

create_time No Long Time when a session is set up. Unit: ms.

update_time No Long Time when a session is updated. Unit: ms.

Table 8-9 Session status parameters


not_started Not started

starting Starting

idle Idle

busy The statement is being executed.

shutting_down Stopping




error An error occurs.

dead A user has exited a session.

success A session successfully stops.

recovering Recovering


ResourceType

PhysicalResource

driverCores

executorCores

driverMemory

executorMemory

numExecutor


2 1 7 GB 4 GB 6


2 2 7 GB 8 GB 7


4 2 15 GB 8 GB 14


{ "kind": "spark", "cluster_name": "test_cluster", "sc_type": "A", "jars": [ "group.tesddws/gsjdbc4.jar" ], "resources": [ { "name": "group.tesddws/gsjdbc3.jar", "type": "jar" } ], "groups": [ { "name": "group.tesddws", "resources": [ { "name": "gsjdbc2.jar", "type": "jar" } ] } ]}



● Example response (successful request){ "id": "f28a0d0c-0a37-4bd4-8d1d-1416031e5151", "appId": null, "name": null, "owner": null, "proxyUser": null, "state": "starting", "kind": "spark", "appInfo": { "driverLogUrl": null, "sparkUiUrl": null }, "log": [ "stdout: ", "\nstderr: ", "\nYARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test_cluster", "create_time": 1531906043036, "update_time": 0}


8.1.3 Canceling a Session

Function

This API is used to cancel a session in a project.

URI● URI format

DELETE /v2.0/{project_id}/sessions/{session_id}● Parameter description


Parameter

Mandatory

Description

project_id


session_id

Yes Session ID, which is returned when a session is created.

Request

None



Response


Parameter Mandatory

Type Description

msg No String If the operation is successful, valuedeleted is returned.


None

● Example response (successful request){ "msg": "deleted"}


8.1.4 Viewing the Session List (Recommended)

FunctionThis API is used to view the list of sessions in a queue in the project.

URI● URI format

GET /v2.0/{project_id}/sessions● Parameter description


Parameter

Mandatory

Description

project_id




Request


Parameter

Mandatory

Type Description

cluster_name

No String Name of a DLI queue. If this parameter is leftunspecified, all sessions in the current project arereturned.

queue No String Queue name. Set this parameter to the name ofthe created DLI queue.NOTE

You only need to specify either cluster_name orqueue. The two parameters cannot be both empty.

from No Integer ID of the start session.

size No Integer Number of obtained sessions.

Response


Parameter

Mandatory

Type Description


total No Integer Total number of sessions.

sessions No Array ofobjects

Session information. For details aboutthe parameters, see Table 8-8 inCreating a Session (Discarded).

create_time

No Long Time when a session is set up. Thetimestamp is expressed inmilliseconds.


None

● Example response (successful request){ "from": 0, "total": 1, "sessions": [ { "id": "f28a0d0c-0a37-4bd4-8d1d-1416031e5151", "appId": "application_1521602145695_0044", "name": null, "owner": null,



"proxyUser": null, "state": "idle", "kind": "spark", "appInfo": { "driverLogUrl": "https://xxx-cluster-75-0006:26010/node/containerlogs/container_1521602145695_0044_01_000001/mls", "sparkUiUrl": "https://xxx-cluster-75-0008:26001/proxy/application_1521602145695_0044/" }, "log": [ "\t ApplicationMaster RPC port: 0", "\t queue: default", "\t queue user: mls", "\t start time: 1521685195140", "\t final status: UNDEFINED", "\t tracking URL: https://xxx-cluster-75-0008:26001/proxy/application_1521602145695_0044/", "\t user: mls | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "2018-03-22 10:20:08,305 | INFO | pool-1-thread-1 | Shutdown hook called | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "2018-03-22 10:20:08,307 | INFO | pool-1-thread-1 | Deleting directory /tmp/spark-43a8dcab-e315-4c09-bb2d-7b2dab71d588 | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "\nYARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test_cluster", "create_time": 1531906043036 } ]}


8.1.5 Viewing the Session List (Discarded)

Function

This API is used to view the list of sessions in a cluster in the project.

URI● URI format

GET /v2.0/{project_id}/sessions



Parameter

Mandatory

Description

project_id




Request


Parameter

Mandatory

Type Description

cluster_name

No String Name of a DLI cluster, which is used to identify acluster. If this parameter is left unspecified, allsessions in the current project are returned.


size No Integer Number of obtained sessions.

Response


Parameter

Mandatory

Type Description


total No Integer Total number of sessions.

sessions

No Array ofobjects

Session information. For details about theparameters, see Table 8-8 in Creating a Session(Discarded).

create_time

No Long Time when a session is set up. The timestamp isexpressed in milliseconds.


{}

● Example response (successful request){ "from": 0, "total": 1, "sessions": [ { "id": "f28a0d0c-0a37-4bd4-8d1d-1416031e5151", "appId": "application_1521602145695_0044", "name": null, "owner": null, "proxyUser": null, "state": "idle", "kind": "spark", "appInfo": { "driverLogUrl": "https://xxx-cluster-75-0006:26010/node/containerlogs/container_1521602145695_0044_01_000001/mls", "sparkUiUrl": "https://xxx-cluster-75-0008:26001/proxy/application_1521602145695_0044/"



}, "log": [ "\t ApplicationMaster RPC port: 0", "\t queue: default", "\t queue user: mls", "\t start time: 1521685195140", "\t final status: UNDEFINED", "\t tracking URL: https://xxx-cluster-75-0008:26001/proxy/application_1521602145695_0044/", "\t user: mls | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "2018-03-22 10:20:08,305 | INFO | pool-1-thread-1 | Shutdown hook called | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "2018-03-22 10:20:08,307 | INFO | pool-1-thread-1 | Deleting directory /tmp/spark-43a8dcab-e315-4c09-bb2d-7b2dab71d588 | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "\nYARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test_cluster", "create_time": 1531906043036 } ]}


8.1.6 Viewing the Session Details

Function

This API is used to view details about a session in a project.

URI● URI format

GET /v2.0/{project_id}/sessions/{session_id}● Parameter description


Parameter

Mandatory

Description

project_id


session_id


Request

None



Response

Response parameters

Table 8-8 in Creating a Session (Discarded) describes the response parameters.


None

● Example response (successful request){ "id": "f28a0d0c-0a37-4bd4-8d1d-1416031e5151", "appId": "application_1521602145695_0044", "name": null, "owner": null, "proxyUser": null, "state": "idle", "kind": "spark", "appInfo": { "driverLogUrl": "https://xxx-cluster-75-0006:26010/node/containerlogs/container_1521602145695_0044_01_000001/mls", "sparkUiUrl": "https://xxx-cluster-75-0008:26001/proxy/application_1521602145695_0044/" }, "log": [ "\t ApplicationMaster RPC port: 0", "\t queue: default", "\t queue user: mls", "\t start time: 1521685195140", "\t final status: UNDEFINED", "\t tracking URL: https://xxx-cluster-75-0008:26001/proxy/application_1521602145695_0044/", "\t user: mls | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "2018-03-22 10:20:08,305 | INFO | pool-1-thread-1 | Shutdown hook called | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "2018-03-22 10:20:08,307 | INFO | pool-1-thread-1 | Deleting directory /tmp/spark-43a8dcab-e315-4c09-bb2d-7b2dab71d588 | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)", "\nYARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test_cluster", "create_time": 1531906043036}


8.1.7 Checking Status of a Session

Function

This API is used to view the status of sessions in a queue in the project.

URI● URI format

GET /v2.0/{project_id}/sessions/{session_id}/state




Table 8-20 URI parameter description

Parameter

Mandatory

Description

project_id


session_id


RequestNone

ResponseTable 8-8 in Creating a Session (Discarded) describes the response parameters.


None

● Example response (successful request){ "id": "726c31dd-a9ef-4a35-83ce-85a083ab68a1", "state": "starting"}


8.1.8 Viewing Session Logs

FunctionThis API is used to view session logs in a project.

URI● URI format

GET /v2.0/{project_id}/sessions/{session_id}/log● Parameter description




Parameter

Mandatory

Description

project_id


session_id


Request


Parameter

Mandatory

Type Description


size No Integer Number of returned sessions.

Response


Parameter

Mandatory

Type Description

id No String Session ID.

from No Integer Number of the start row in which logsare displayed.

total No Integer Total number of rows in the logs.

log No Array of strings Session log.


None

● Example response (successful request){ "id": "0e094f78-5760-43ed-a062-1fb3aed121d2", "from": 0, "total": 89, "log": [ "stdout: ", "\nstderr: ", "2018-03-26 10:08:07,104 | WARN | main | Unable to load native- hadoop library for your



platform... using builtin-java classes w here applicable | org.apache.hadoop.util.NativeCodeLoader.<clini t>(NativeCodeLoader.java:62)", "2018-03-26 10:08:07,408 | INFO | main | Login successful for u ser [email protected] using keytab file user.keytab | org.apa che.hadoop.security.UserGroupInformation.loginUserFromKeytab(Use rGroupInformation.java:1008)", "Warning: Skip remote jar hdfs://hacluster/user/c8f7a880fe7e4bba 866582c8111eb989/3efffb4f-40e9-455e-8b5a-a23b4d355e46_luxo-router-1.1.1.jar.", "2018-03-26 10:08:08,681 | INFO | main | Attempting to login to the Kerberos using principal: [email protected] and keytab: /opt/DLI/mlsResource/user.keytab | org. ... ]}


8.2 Statement-related APIs

8.2.1 Creating a Statement

FunctionThis API is used to create a statement in a session of a project.

URI● URI format

POST /v2.0/{project_id}/sessions/{session_id}/statements● Parameter description


Parameter

Mandatory

Description

project_id


session_id




Request


Parameter

Mandatory

Type Description

code Yes String Code snippet executed in the statement.Currently, only the Scala and Python codes aresupported. The code can contain a maximum of1024 characters.

Response


Parameter

Mandatory

Type Description

id No Integer Statement ID, which is a UUID characterstring

code No String Statement

state No Array of strings Statement execution status. For details, seeTable 8-27.

output No Array of objects Statement execution result. For detailsabout the parameters, see Table 8-28.

progress

No Double Statement execution progress.● 0: Indicates that the execution starts.● 1: Indicates that the execution is

complete.

Table 8-27 Statement state parameter description

Parameter Type Description

waiting String A statement is queuing and is not executed yet.

running String The statement is being executed.

available String A statement has been executed.

error String A statement fails to be executed.

cancelling String Execution of a statement is being canceled.




cancelled String Execution of a statement has been canceled.

Table 8-28 Statement output parameter description


status string Statement execution status

execution_count

int Number of statement execution times

data json Output result. The JSON format is supported. TheOLV data format is used if DLI interconnects withMLS.


{ "code": "val a=1"}

● Example response (successful request){ "id": 1, "code": "val a=1", "state": "waiting", "output": null, "progress": 0}


8.2.2 Canceling Execution of a Statement

Function

This API is used to cancel the execution of a statement in a session.

URI● URI format

POST /v2.0/{project_id}/sessions/{session_id}/statements/{statement_id}/cancel





Parameter

Mandatory

Description

project_id


session_id


statement_id

Yes Statement ID, which is returned when a statement iscreated.

Request

None

Response


Parameter Mandatory

Type Description

msg No String If the operation is successful, valuecanceled is returned.


None

● Example response (successful request){"msg": "canceled"}


8.2.3 Querying List of Statements

Function

This API is used to view the list of statements of a session in a project.



URI● URI format

GET /v2.0/{project_id}/sessions/{session_id}/statements● Parameter description


Parameter

Mandatory

Description

project_id


session_id

Yes Session ID, which is returned when a session iscreated.

Request

None

Response


Parameter Mandatory

Type Description

total_statements

No Integer Total number of statements

statements No Array ofObjects

Statement list. For details, see Table 8-26 inCreating a Statement.

state No String Execution status. For details, see Table 8-27in Creating a Statement.

output No Struct Execution result. For details, see Table 8-28in Creating a Statement.

progress No Double Statement execution progress.● 0: Indicates that the execution starts.● 1: Indicates that the execution is

complete.


None



● Example response (successful request){ "total_statements": 1, "statements": [ { "id": 0, "code": "val a =0 ", "state": "available", "output": { "status": "ok", "execution_count": 0, "data": { "text/plain": "a: Int = 0" } }, "progress": 1 } ]}


8.2.4 View Information About Statements

Function

This API is used to view information about the statements of a session in a project.

URI● URI format

GET /v2.0/{project_id}/sessions/{session_id}/statements/{statement_id}● Parameter description


Parameter

Mandatory

Description


session_id


statement_id

Yes Statement ID, which is returned when a statement iscreated.

Request

None



ResponseTable 8-26 in Creating a Statement describes the response parameters.


None

● Example response (successful request){ "id": 2, "code": "val a =0 ", "state": "waiting", "output": null, "progress": 0}


8.3 Batch Processing-related APIs

8.3.1 Creating a Batch Processing Job

FunctionThis API is used to create a batch processing job in a queue.

URI● URI format

POST /v2.0/{project_id}/batches● Parameter description


Parameter

Mandatory

Description

project_id




Request


Parameter Mandatory

Type Description

file Yes String Name of the package that is of the JARtype and has been uploaded to the DLIresource management system.

className Yes String Java/Spark main class of the batchprocessing job.

queue Yes String Queue name. Set this parameter to thename of the created DLI queue.NOTE

● This parameter is compatible with thecluster_name parameter. That is, ifcluster_name is used to specify a queue,the queue is still valid.

● The queue parameter is recommended. Ifboth queue and cluster_name exist, thevalue of queue prevails.

cluster_name

No String Queue name. Set this parameter to thecreated DLI queue name.

args No Array ofstrings

Parameters passed to the main class.

sc_type No String Compute resource type. Currently, resourcetypes A, B, and C are available. If thisparameter is not specified, the minimumconfiguration (type A) is used. For detailsabout resource types, see Table 8-36.

jars No Array ofstrings

Name of the package that is of the JARtype and has been uploaded to the DLIresource management system.

pyFiles No Array ofstrings


files No Array ofstrings

Name of the package that is of the filetype and has been uploaded to the DLIresource management system.



Parameter Mandatory

Type Description

modules No Array ofstrings

Name of the dependent system resourcemodule. You can view the module nameusing the API related to Querying AllResource Packages.DLI provides dependencies for executingdatasource jobs. The following table liststhe dependency modules corresponding todifferent services.● CloudTable/MRS HBase:

sys.datasource.hbase● CloudTable/MRS OpenTSDB:

sys.datasource.opentsdb● RDS MySQL: sys.datasource.rds● RDS Postgre: preset● DWS: preset● CSS: sys.datasource.css


JSON object list, including the name andtype of the JSON package that has beenuploaded to the queue.

groups No Array ofstrings

JSON object list, including the packagegroup resource. For details about theformat, see the request example.


Batch configuration item. For details, seeSpark Configuration.

name No String Batch processing task name. The valuecontains a maximum of 128 characters.

driverMemory

No String Driver memory of the Spark application,for example, 2 GB and 2048 MB. Thisconfiguration item replaces the defaultparameter in sc_type. The unit must beprovided. Otherwise, the startup fails.

driverCores No Integer Number of CPU cores of the Sparkapplication driver. This configuration itemreplaces the default parameter in sc_type.

executorMemory

No String Executor memory of the Spark application,for example, 2 GB and 2048 MB. Thisconfiguration item replaces the defaultparameter in sc_type. The unit must beprovided. Otherwise, the startup fails.



https://spark.apache.org/docs/latest/configuration.html

Parameter Mandatory

Type Description

executorCores

No Integer Number of CPU cores of each Executor inthe Spark application. This configurationitem replaces the default parameter insc_type.

numExecutors

No Integer Number of Executors in a Sparkapplication. This configuration itemreplaces the default parameter in sc_type.

obs_bucket No String OBS bucket for storing the Spark jobs. Setthis parameter when you need to savejobs.

autoRecovery

No Boolean Whether to enable the retry function. Ifenabled, Spark jobs will be automaticallyretried after an exception occurs. Thedefault value is false.

maxRetryTimes

No Integer Maximum retry times. The maximum valueis 100, and the default value is 20.


ResourceType

PhysicalResource

driverCores

executorCores

driverMemory

executorMemory

numExecutor


2 1 7 GB 4 GB 6


2 2 7 GB 8 GB 7


4 2 15 GB 8 GB 14



Response


Parameter

Mandatory

Type Description

id No String ID of a batch processing job.

state No String Status of a batch processing job. For details,see Table 8-38.

appId No Array ofstrings

Back-end application ID of a batchprocessing job.


Detailed information about a batchprocessing job.


Last 10 records of the current batchprocessing job.

sc_type No String Type of a computing resource. If thecomputing resource type is customized,value CUSTOMIZED is returned.

cluster_name

No String Queue where a batch processing job islocated.

create_time

No Long Time when a batch processing job iscreated. The timestamp is expressed inmilliseconds.

Table 8-38 Batch processing job statuses


starting String The batch processing job is being started.

running String The batch processing job is executing a task.

dead String The batch processing job has exited.

success String The batch processing job is successfully executed.

recovering String The batch processing job is being restored.


{ "file": "batchTest/spark-examples_2.11-2.1.0.luxor.jar", "className": "org.apache.spark.examples.SparkPi", "sc_type": "A", "jars": ["demo-1.0.0.jar"], "files": ["count.txt"],



"resources":[ {"name": "groupTest/testJar.jar", "type": "jar"}, {"name": "kafka-clients-0.10.0.0.jar", "type": "jar"}], "groups": [ {"name": "groupTestJar", "resources": [{"name": "testJar.jar", "type": "jar"}, {"name": "testJar1.jar", "type": "jar"}]}, {"name": "batchTest", "resources": [{"name": "luxor.jar", "type": "jar"}]}], "cluster_name": " test", "name": "TestDemo4"}

The batchTest/spark-examples_2.11-2.1.0.luxor.jar file has been uploaded throughAPI involved in Uploading a Resource Package.

● Example response (successful request){ "id": "80ceaaff-3cfc-4162-a56f-70031ea4fa91", "state": "starting", "appId": null, "appInfo": { "driverLogUrl": null, "sparkUiUrl": null }, "log": [ "stdout: ", "\nstderr: ", "\nYARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test", "create_time": 1531906043036}


8.3.2 Canceling a Batch Processing Job

Function

This API is used to cancel a batch processing job.

URI● URI format

DELETE /v2.0/{project_id}/batches/{batch_id}● Parameter description


Parameter

Mandatory

Description

project_id




Parameter

Mandatory

Description

batch_id

Yes ID of a batch processing job.

Request

None

Response


Parameter

Mandatory Type Description

msg No String If the batch processing job issuccessfully cancelled, valuedeleted is returned.


None

● Example response (successful request){ "msg": "deleted"}


8.3.3 Obtaining the List of Batch Processing Jobs

Function

This API is used to obtain the list of batch processing jobs in a queue of a project.

URI● URI format

GET /v2.0/{project_id}/batches





Parameter



Request


Parameter

Mandatory

Type Description

cluster_name

No String DLI queue name. If this parameter is left blank, thenames of all batch processing jobs in the currentproject are obtained. You are advised to specify thisparameter, instead of leaving it blank.

from No Integer

Index number of the start batch processing job. Bydefault, the index number starts from 0.

size No Integer

Number of batch processing jobs to be queried

Response


Parameter

Mandatory

Type Description

from No Integer Index number of the start batch processing job.

total No Integer Total number of batch processing jobs.

sessions No Array ofobjects

Batch job information. For details, see Table8-37 in Creating a Batch Processing Job.

create_time

No Long Time when a batch processing job is created.


None



● Example response (successful request){ "from": 0, "total": 1, "sessions": [ { "id": "178fa687-2e8a-41ed-a439-b00de60bb176", "state": "dead", "appId": null, "appInfo": { "driverLogUrl": null, "sparkUiUrl": null }, "log": [ "stdout: ", "stderr: ", "YARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test", "create_time": 1531906043036 } ]}


8.3.4 Querying Batch Job Details

Function

This API is used to query details about a batch processing job based on the job ID.

URI● URI format

GET /v2.0/{project_id}/batches/{batch_id}● Parameter description


Parameter


project_id


batch_id Yes ID of a batch processing job.

Request

None



Response


Parameter

Mandatory

Type Description


appId No Array ofstrings

Back-end application ID of a batchprocessing job.

name No String Name of a batch processing job.

owner No String Owner of a batch processing job.

proxyUser No String Proxy user (resource tenant) to which abatch processing job belongs.

state No String Status of a batch processing job. Fordetails, see Table 8-38 in Creating aBatch Processing Job.

kind No String Type of a batch processing job. OnlySpark parameters are supported.


Detailed information about a batchprocessing job.

log No list of strings Last 10 records of the current batchprocessing job.

sc_type No String Type of a computing resource. If thecomputing resource type is customized,value CUSTOMIZED is returned.

cluster_name

No String Queue where a batch processing job islocated.

queue No String Queue where a batch processing job islocated.

create_time

No Long Time when a batch processing job iscreated. The timestamp is expressed inmilliseconds.

update_time

No Long Time when a batch processing job isupdated. The timestamp is expressed inmilliseconds.


None

● Example response (successful request){ "id": "0a324461-d9d9-45da-a52a-3b3c7a3d809e",



"appId": "", "name": "", "owner": "", "proxyUser": "", "state": "starting", "kind":"", "appInfo": { "driverLogUrl": null, "sparkUiUrl": null }, "log": [ "stdout: ", "stderr: ", "YARN Diagnostics: " ], "sc_type": "A", "cluster_name": "test", "queue": "test", "create_time": 1531906043036, "update_time": 1531906043036}


8.3.5 Querying a Batch Job Status

Function

This API is used to obtain the execution status of a batch processing job.

URI● URI format

GET /v2.0/{project_id}/batches/{batch_id}/state



Parameter

Mandatory

Description



Request

None



Response


Parameter

Mandatory

Type Description

id No String ID of a batch processing job, which is in the universalunique identifier (UUID) format.

state

No String Status of a batch processing job. For details, see Table8-38 in Creating a Batch Processing Job.


None

● Example response (successful request){"id":"0a324461-d9d9-45da-a52a-3b3c7a3d809e","state":"Success"}


8.3.6 Querying Batch Job Logs

FunctionThis API is used to query the back-end logs of batch processing jobs.

URI● URI format

GET /v2.0/{project_id}/batches/{batch_id}/log● Parameter description


Parameter

Mandatory

Description

project_id





Request


Parameter

Mandatory

Type Description

from

No Integer

Number of bytes in the start log. By default, the last 1MB log is displayed. If the size of a log file is less than 1MB, the log file is displayed from 0 bytes.

size No Integer

Size of a query log.

type No String If type is set to driver, the Spark Driver log is generated.

index

No Integer

When a submitted job is retried, multiple driver logs aregenerated. This parameter specifies the index number ofthe specified driver log. The default value is 0. Thisparameter must be used together with the typeparameter.

Response


Parameter

Mandatory

Type Description


from No String Start index of a log.

total No Long Total number of records in a log.


Log of the current batch processing job.


None

● Example response (successful request){ "id": "0a324461-d9d9-45da-a52a-3b3c7a3d809e", "from": 0, "total": 3, "log": [ "stdout: ", "stderr: ", "YARN Diagnostics: "



]}


8.4 Resource Package-related APIs

8.4.1 Uploading a Resource Package

FunctionThis API is used to upload a resource package to a project.

If a resource package with the same name as the existing package is uploaded, the newresource package will overwrite the old one.

URI● URI format

POST /v2.0/{project_id}/resources● Parameter description


Parameter

Mandatory

Description


RequestTable 8-52 and Table 8-53 describe the request parameters.



Table 8-52 Table parameters

Parameter

Mandatory

Type Description

paths Yes Array ofstrings

List of OBS object paths. The OBS object pathrefers to the OBS object URL.NOTE

The length of the uploaded resource package name(including the file name extension) cannot exceed128 characters.

kind Yes String Resource type. For details, see Table 8-53.



jar JAR file

pyFile Python file

file User file

modelFile User AI model file

Response


Parameter Mandatory

Type Description

resource_names

No List ofString

List of names of uploaded resourcepackages.


{ "paths": ["https://xkftest.obs.huawei.com/kmeans-2.0.0.jar"], "kind": "jar"}

● Example response (successful request){ "resource_names": [ "kmeans-2.0.0.jar" ]}




8.4.2 Deleting Resource Packages

FunctionThis API is used to delete resource packages in a project.

URI● URI format

DELETE /v2.0/{project_id}/resources/{resource_name}● Parameter description


Parameter

Mandatory

Description


resource_name

Yes Name of the resource package returned when theresource package is uploaded

RequestNone

Response● Code 200 is returned if you successfully delete a resource package.● Code 404 is returned if you initiate a request to delete a resource package

that does not exist.


None

● Example response (successful request)None




8.4.3 Querying All Resource Packages

Function

This API is used to view the resource package list of a project, including the built-in resource modules and resource packages.

URI● URI format

GET /v2.0/{project_id}/resources● Parameter description


Parameter

Mandatory

Description

project_id


Request

None

Response


Parameter

Mandatory

Type Description

resources

No Array ofobjects

List of names of uploaded user resources.For details about resources, see Table 8-58.

modules

No Array ofobjects

List of built-in resource modules. For detailsabout the modules, see Table 8-59.

Table 8-58 Resource details


create_time Long UNIX time when a resource package is uploaded.The timestamp is expressed in milliseconds.

update_time Long UNIX time when the uploaded resource package isuploaded. The timestamp is expressed inmilliseconds.




resource_type String Resource type.

resource_name

String Resource name.

status String ● Value UPLOADING indicates that the packagegroup is being uploaded.

● Value READY indicates that the resourcepackage has been uploaded.

● Value FAILED indicates that the resourcepackage fails to be uploaded.

underlying_name

String Name of the resource package in the queue.

Table 8-59 Details about resource modules


module_name

String Module name.

module_type String Module type. For details about the module type,see Table 8-53.

status String ● Value UPLOADING indicates that the modulepackage is being uploaded.

● Value READY indicates that the modulepackage has been uploaded.

● Value FAILED indicates that the modulepackage fails to be uploaded.

description String Module description.

resources Array ofstrings

List of names of resource packages contained inthe module.

create_time Long UNIX time when a module package is uploaded.

update_time Long UNIX time when a module package is updated.


None

● Example response (successful request){ "resources": [ { "create_time": 1521532893736, "update_time": 1521552364503,



"resource_type": "jar", "resource_name": "luxor-router-1.1.1.jar", "status": "READY", "underlying_name": "3efffb4f-40e9-455e-8b5a-a23b4d355e46_luxor-router-1.1.1.jar" } ], "modules": [ { "module_name": "sys.dli.gatk", "module_type": "jar", "status": "READY", "description": "jar dependencies required for genome analysis", "resources": [ "gatk-package-4.0.4-spark.jar", "tika-core-1.18.jar", "s3fs-2.2.2.jar" ], "create_time": 1521532893736, "update_time": 1521552364503 } ]}


8.4.4 Querying a Specified Resource Package

Function

This API is used to view the resource packages in a project.

URI● URI format

GET /v2.0/{project_id}/resources/{resource_name}



Parameter Mandatory

Description


resource_name Yes Name of the resource package returned when itis uploaded.

Request

None



ResponseTable 8-58 describes the response parameters.


None

● Example response (successful request){ "create_time": 1522055409139, "update_time": 1522228350501, "resource_type": "jar", "resource_name": "luxor-ommanager-dist.tar.gz", "status": "uploading", "underlying_name": "7885d26e-c532-40f3-a755-c82c442f19b8_luxor-ommanager-dist.tar.gz"}


8.5 Package Group-related APIs

8.5.1 Uploading a Package Group

FunctionThis API is used to upload a package group to a project.

URI● URI format

POST /v2.0/{project_id}/resources● Parameter description


Parameter Mandatory

Description




Request


Parameter

Mandatory

Type Description


List of OBS object paths. The OBS object pathrefers to the OBS object URL.

kind Yes String File type of a package group.NOTE

If the same group of packages to be uploaded containsdifferent file types, select file as the type of the file tobe uploaded.

group

Yes String Name of the group to be created.

Response

Response parameters


Parameter Mandatory

Type Description

group_name

No String Group name.

status No String Status of a package group to be uploaded.


List of names of resource packagescontained in the group.

create_time No Long UNIX timestamp when a package group isuploaded.

update_time

No Long UNIX timestamp when a package group isupdated.


{ "paths": [ "https://xkftest.obs.huawei.com/txr_test/jars/spark-sdv-app.jar", "https://xkftest.obs.huawei.com/txr_test/jars/wordcount", "https://xkftest.obs.huawei.com/txr_test/jars/wordcount.py" ], "kind": "jar",



"group": "gatk"}

● Example response (successful request){ "group_name": "gatk", "status": "READY", "resources": [ "spark-sdv-app.jar", "wordcount", "wordcount.py" ], "details": [ { "create_time": 0, "update_time": 0, "resource_type": "jar", "resource_name": "spark-sdv-app.jar", "status": "READY", "underlying_name": "987e208d-d46e-4475-a8c0-a62f0275750b_spark-sdv-app.jar" }, { "create_time": 0, "update_time": 0, "resource_type": "jar", "resource_name": "wordcount", "status": "READY", "underlying_name": "987e208d-d46e-4475-a8c0-a62f0275750b_wordcount" }, { "create_time": 0, "update_time": 0, "resource_type": "jar", "resource_name": "wordcount.py", "status": "READY", "underlying_name": "987e208d-d46e-4475-a8c0-a62f0275750b_wordcount.py" } ], "create_time": 1551334579654, "update_time": 1551345369070}

8.5.2 Querying Package Group List

Function

This API is used to query all resources in a project, including groups.

URI● URI format

GET /v2.0/{project_id}/resources● Parameter description


Parameter

Mandatory

Description

project_id




Request

None

Response

Response parameters

Table 8-65 describes the response parameters.


Parameter

Mandatory

Type Description

resources

No Array ofobjects

List of names of uploaded user resources. Fordetails about resources, see Table 8-66.

modules

No Array ofobjects

List of built-in resource groups. For detailsabout the groups, see Table 8-67.

groups No Array ofobjects

Uploaded package groups of a user.

Table 8-66 Resource parameters


create_time Long UNIX timestamp when a resource package isuploaded.

update_time Long UNIX timestamp when the uploaded resource packageis uploaded.

resource_type String Resource type.

resource_name

String Resource name.

status String ● Value UPLOADING indicates that the resourcepackage is being uploaded.

● Value READY indicates that the resource packagehas been uploaded.

● Value FAILED indicates that the resource packagefails to be uploaded.

underlying_name

String Name of the resource package in the queue.



Table 8-67 Resource group parameters


group_name String Group name.

status String ● Value UPLOADING indicates that the packagegroup is being uploaded.

● Value READY indicates that the package grouphas been uploaded.

● Value FAILED indicates that the package groupfails to be uploaded.

resources List ofString

List of names of resource packages contained inthe group.

details List ofJson

Resource package details.

create_time Long UNIX timestamp when a package group isuploaded.

update_time Long UNIX timestamp when a package group is updated.


None

● Example response (successful request){ "resources": [ { "create_time": 1521532893736, "update_time": 1521552364503, "resource_type": "jar", "resource_name": "luxor-router-1.1.1.jar", "status": "READY", "underlying_name": "3efffb4f-40e9-455e-8b5a-a23b4d355e46_luxor-router-1.1.1.jar" } ], "groups": [ { "group_name": "groupTest", "status": "READY", "resources": [ "part-00000-9dfc17b1-2feb-45c5-b81d-bff533d6ed13.csv.gz", "person.csv" ], "details": [ { "create_time": 1547090015132, "update_time": 1547090015132, "resource_type": "jar", "resource_name": "part-00000-9dfc17b1-2feb-45c5-b81d-bff533d6ed13.csv.gz", "status": "READY", "underlying_name": "db50c4dc-7187-4eb9-a5d0-73ba8102ea5e_part-00000-9dfc17b1-2feb-45c5-b81d-bff533d6ed13.csv.gz" }, { "create_time": 1547091098668, "update_time": 1547091098668,



"resource_type": "file", "resource_name": "person.csv", "status": "READY", "underlying_name": "a4243a8c-bca6-4e77-a968-1f3b00217474_person.csv" } ], "create_time": 1547090015131, "update_time": 1547091098666 } ], "modules": [ { "module_name": "gatk", "status": "READY", "resources": [ "gatk.jar", "tika-core-1.18.jar", "s3fs-2.2.2.jar" ], "create_time": 1521532893736, "update_time": 1521552364503 } ]}

8.5.3 Uploading a JAR Package Group

Function

This API is used to upload a group of JAR packages to a project.

When a resource group with the same name is uploaded, the new group overwrites the oldgroup.

URI● URI format

POST /v2.0/{project_id}/resources/jars



Parameter


project_id


Request





Parameter

Mandatory

Type Description


List of OBS object paths. The OBS object pathrefers to the OBS object URL.

group Yes String Name of a package group.

ResponseResponse parameters


Parameter Mandatory

Type Description

module_name No String Name of a resource group.

module_type No String Type of a resource group. For details,see Table 8-71.

status No String ● Value UPLOADING indicates thatthe package group is beinguploaded.

● Value READY indicates that thepackage group has been uploaded.

● Value FAILED indicates that thepackage group fails to be uploaded.

description No String Description of a resource group.


List of resource package namescontained in the resource group.

create_time No Long The UNIX timestamp uploaded by theresource group.

update_time No Long The UNIX timestamp updated by theresource group.

Table 8-71 Resource group types


jar JAR file

pyFile Python file

file User file




{ "paths": [ "https://xkftest.obs.huawei.com/spark.jar", "https://xkftest.obs.huawei.com/tika-core-1.18.jar", "https://xkftest.obs.huawei.com/s3fs-2.2.2.jar" ], "group": "gatk"}

● Example response (successful request){ { "group_name": "gatk", "status": "READY", "resources": [ "gatk.jar", "tika-core-1.18.jar", "s3fs-2.2.2.jar" ], "create_time": 1521532893736, "update_time": 1521552364503 }

8.5.4 Uploading a PyFile Package Group

Function

This API is used to upload a group of PyFile packages to a project.

When a group with the same name as the PyFile package is uploaded, the new groupoverwrites the old group.

URI● URI format

POST /v2.0/{project_id}/resources/pyfiles



Parameter Mandatory

Description


Request





Parameter Mandatory

Type Description


List of OBS object paths. The OBS objectpath refers to the OBS object URL.


Response

Response parameters

For details about the response parameters, see Table 8-70.


{ "paths": [ "https://xkftest.obs.huawei.com/spark.py" ], "group": " gatk"}

● Example response (successful request){ "group_name": "gatk", "status": "READY", "resources": [ "spark.py " ], "create_time": 1521532893736, "update_time": 1521552364503}

8.5.5 Uploading a File Package Group

Function

This API is used to upload a group of File packages to a project.

When the File package group with the same name is uploaded, the new group overwritesthe old group.

URI● URI format

POST /v2.0/{project_id}/resources/files





Parameter

Mandatory

Description

project_id


RequestTable 8-75 describes the request parameters.


Parameter

Mandatory

Type Description


List of OBS object paths. The OBS object path refersto the OBS object URL.


ResponseResponse parameters



{ "paths": [ https: //xkftest.obs.huawei.com/spark.py, "https://xkftest.obs.huawei.com/spark.msi" ]"group": " gatk"}

● Example response (successful request){ "group_name": "gatk", "status": "READY", "resources": [ "spark.py ", " spark.msi" ], "create_time": 1521532893736, "update_time": 1521552364503}



8.5.6 querying Resource Packages in a Group

Function

This API is used to query resource information of a package group in a Project.

URI● URI format

GET /v2.0/{project_id}/resources/{resource_name}?group={group}



Parameter

Mandatory

Description

project_id


resource_name

No Name of the resource package that is uploaded.

group Yes Name of the package group returned when theresource package is uploaded.

Request

None

Response

Response parameters



{}

● Example response (successful request){ "create_time": 1522055409139, "update_time": 1522228350501, "resource_type": "jar", "resource_name": "luxor-ommanager-dist.tar.gz", "status": "uploading", "underlying_name": "7885d26e-c532-40f3-a755-c82c442f19b8_luxor-ommanager-dist.tar.gz"}



8.5.7 Deleting a Resource Package from a Group

Function

This API is used to delete resource information of a package group in a Project.

URI● URI format

DELETE /v2.0/{project_id}/resources/{resource_name}?group={group}



Parameter

Mandatory

Description

project_id


group Yes Name of the package group returned when theresource package is uploaded.

Request

None

Response● Code 200 is returned if you successfully delete a resource package.

● Code 404 is returned if you initiate a request to delete a resource packagethat does not exist.


{}

● Example response (successful request){}


8.6 Cluster-related APIs



8.6.1 Creating a Cluster (Discarded)

FunctionThis API is used to create a cluster. The cluster will be bound to specified computeresources.

URI● URI format

POST /v2.0/{project_id}/clusters● Parameter description


Parameter

Mandatory

Description

project_id


RequestIn addition to the common request header parameters involved in cluster creation,the special request header parameters described in Table 8-79 are also involved.

Table 8-79 Header description

Parameter Mandatory

Description

X-Auth-Token Yes User token obtained from IAM.

Accept Yes The default value is application/json.

Content-Type Yes The value is specified to application/json.

charset Yes The encoding format is specified to UTF8.





Parameter Mandatory

Type Description

cluster_name Yes String Name of a new cluster. The name cancontain only digits, letters, and underscores(_), but cannot contain only digits or startwith an underscore (_).

description No String Description of a cluster.


Number of computing units (units to bebilled) bound to a cluster. You can select 16CUs, 64 CUs, or 256 CUs. If you need a largerquota, contact the DLI service administrator.

cidr_in_vpc No String VPC CIDR

cidr_in_mgntsubnet

No String CIDR of the management subnet

cidr_in_subnet No String Special subnet CIDR

Response● Return code

Code 201 is returned if the operation is successful.● Response parameter

None


{ "cluster_name": "cluster1", "description": "test cluster", "cu_count": 16}

● Example response (successful request){}


8.6.2 Deleting a Cluster



8.6.3 Querying Information of a Specified Cluster (Discarded)

Function

This API is used to query information about a specified cluster, including the name,description, and number of bound computing resources.

URI● URI format

GET /v2.0/{project_id}/clusters/{cluster_name}● Parameter description


Parameter Mandatory

Description


cluster_name

Yes Cluster name

Request

None


Code 200 is returned if the operation is successful.● Response parameter

Table 8-82 describes the response parameters.


Parameter Mandatory

Type Description

create_time Yes Long Time when a cluster is created Thetimestamp is expressed in milliseconds.


cu_count Yes Integer Number of CUs that are bound to acluster.

owner Yes String User who creates the cluster.



Parameter Mandatory

Type Description

cluster_name

Yes String Cluster name

status Yes String Cluster status. For details, see Table8-83.

resource_mode

Yes Integer Cluster type.● 0: Shared cluster● 1: Dedicated cluster

Table 8-83 Cluster status


CREATING The cluster is being created and is unavailable now.

AVAILABLE The cluster is available.

SUSPENDED

The cluster is suspended because it is not used for a longtime. You can reactivate the cluster by using the Creating aBatch Processing Job interface.

RECOVERING

The cluster is being recovered from the suspending state. Thecluster is available again after its status changes toAVAILABLE.


{}

● Example response (successful request){ "create_time": 1508143955000, "cu_count": 16, "description": "test", "owner": "tenant1", "cluster_name": "cluster1", "status": "AVAILABLE", "resource_mode": 0 }




8.6.4 Querying All Cluster Information (Discarded)

Function

This API is used to list all clusters under the project.

URI● URI format

GET/v2.0/{project_id}/clusters



Parameter

Mandatory

Description

project_id


Request

None


Code 200 is returned if the operation is successful.

● Response parameter


Parameter Mandatory

Type Description

create_time Yes Long Time when a cluster is created


cu_count Yes Integer Number of CUs that are bound to acluster.

owner Yes String User who creates the cluster.

cluster_name

Yes String Cluster name

status Yes String Cluster status. For details, see Table8-83.



Parameter Mandatory

Type Description

resource_mode

Yes Integer Cluster type.● 0: Shared cluster● 1: Dedicated cluster


{}

● Example response (successful request){ "clusters": [ { "create_time": 1508143955000, "cu_count": 16, "description": "test", "owner": "tenant1", "cluster_name": "cluster1", "status": "AVAILABLE", "resource_mode": 0 } ] }




9 APIs Related to Stream Job Templates

9.1 Creating a Template

FunctionThis API is used to create a user template for the DLI service. A maximum of 100user templates can be created.

URI● URI format

POST /v1.0/{project_id}/streaming/job-templates● Parameter description


Parameter

Mandatory

Description

project_id



{ "name": "simple_stream_sql", "desc": "Example of quick start", "sql_body": "select * from source_table"}


Data Lake InsightAPI Reference 9 APIs Related to Stream Job Templates



Parameter

Mandatory

Type Description

name Yes String Template name. Length range: 0 to 57 characters.

desc No String Template description. Length range: 0 to 2,048characters.

sql_body

No String Stream SQL statement, which includes at least thefollowing three parts: source, query, and sink.Length range: 0 to 2,048 characters.


{ "is_success": true, "message": "A template is created successfully.", "template": { "template_id": 0, "name": "IoT_example", "desc": "Example of quick start", "create_time": 1516952710040 }}



Parameter Mandatory

Type Description

is_success No Boolean Indicates whether the request issuccessful.


template No Object Information about job update.For details, see Table 9-4.

Table 9-4 template parameters

Parameter Mandatory

Type Description

template_id No Int Template ID.

name No String Template name.

desc No String Template description.



Parameter Mandatory

Type Description

create_time No Long Time when the template iscreated.




200 A template is created successfully.





DLI.18006 The template name cannot be left blank. Specify atemplate name that is not left blank.

9.2 Updating a Template

FunctionThis API is used to update existing templates in DLI.

URI● URI format

PUT /v1.0/{project_id}/streaming/job-templates/{template_id}● Parameter description




Parameter

Mandatory

Description

project_id


template_id

Yes Template ID.

RequestSubmit the request for modifying a template.

● Example request{ "name": "simple_stream_sql", "desc": "Example of quick start", "sql_body": "select * from source_table"}



Parameter

Mandatory

Type

Description

name

No String

Template name. Length range: 0 to 57 characters.

desc No String

Template description. Length range: 0 to 2,048characters.

sql_body

No String

Stream SQL statement, which includes at least thefollowing three parts: source, query, and sink. Lengthrange: 0 to 1024x1024 characters.


{ "is_success": "true", "message": "The template is updated successfully.",}





Parameter

Mandatory

Type Description

is_success

No Boolean

Whether the request is successfully executed.Value true indicates that the request issuccessfully executed.

message





200 A template is updated successfully.





DLI.1205 The template to be updated does not exist. Select anexisting template.

9.3 Deleting a Template

FunctionThis API is used to delete a template. A template used by jobs can also be deleted.

URI● URI format

DELETE /v1.0/{project_id}/streaming/job-templates/{template_id}





Parameter

Mandatory

Description

project_id


template_id

Yes Template ID.

RequestNone


{ "is_success": "true", "message": "The template is deleted successfully.", "template": { "template_id": 2 }}



Parameter Mandatory

Type Description

is_success No Boolean Whether the response is successful. Value true indicates success.


template No Object Information about the template to be deleted.

Table 9-14 template parameters

Parameter Mandatory

Type Description

template_id No Long Template ID.






200 A template is deleted successfully.


Error Code




DLI.18005 The template does not exist. Select an existingtemplate ID.

9.4 Querying the Template List

Function

This API is used to query the job template list. Currently, only custom templatescan be queried.

URI● URI format

GET /v1.0/{project_id}/streaming/job-templates● Parameter description


Parameter

Mandatory

Description



None





Parameter

Mandatory

Type Description

offset No Long Job offset.

limit No Integer

Number of returned data records. The default value10.

order No String Sorting style of the query results.● asc: The query results are displayed in ascending

order.● desc: The query results are displayed in the

descending order.The default value is desc.



{ "is_success": "true", "message": "The template list is obtained successfully.", "template_list": { "total_count": 2, "templates": [ { "template_id": 2, "name": "updatetest", "desc": "Example of quick start", "create_time": 1578748092000, "update_time": 1578748092000, "sql_body": "select * from source_table" }, { "template_id": 1, "name": "we", "desc": "qwe", "create_time": 1577951045000, "update_time": 1577951045000, "sql_body": "" } ] }}





Parameter Mandatory

Type Description

is_success No Boolean Indicates whether the request issuccessful.


template_list No Object Information about the templatelist. For details, see Table 9-20.

Table 9-20 template_list parameters

Parameter Mandatory

Type Description

total_count No Integer Total number of templates.

templates No Object Detailed information about atemplate.

Table 9-21 templates parameters

Parameter Mandatory

Type Description

template_id No Integer Template ID.


desc No String Template description.

create_time No Long Time when the template iscreated.

update_time No Long Time when the template isupdated.

sql_body No String Stream SQL statement, whichincludes at least the followingthree parts: source, query, andsink.






200 Template list query succeeds.




ErrorCode

Error Message

DLI.1206 Incorrect parameters for requesting the template list. Use correctparameters for requesting the template list. The values of offsetand limit must be greater than or equal to 0.



10 APIs Related to Basic DatasourceConnections

10.1 Creating a Basic Datasource Connection

FunctionThis API is used to create a basic datasource connection with other services.

● For details about how to use the console interface, see Basic Datasource Connectionsin the Data Lake Insight User Guide.

● Datasource connections cannot be created for the default queue.

URI● URI format

POST /v2.0/{project_id}/datasource-connection● Parameter description


Parameter

Mandatory

Description


Data Lake InsightAPI Reference 10 APIs Related to Basic Datasource Connections


Request


Parameter Mandatory

Type Description

name Yes String Connection name

service Yes String The name of the connected service. Currently,the value can be CloudTable.OpenTSDB,CloudTable, MRS.OpenTSDB, DWS, RDS, or CSS.NOTE

The value is case insensitive.

security_group_id

Yes String The ID of the security group, where the servicewith the connection to be created belongs.

network_id Yes String The subnet ID of the to-be-connected service.

url Yes String URL used for accessing the service by externalsystems. The value can contain a maximum of512 characters. For details about how to obtainthe URL, see Table 10-3.



Table 10-3 URL obtaining method of each service

ServiceName

Obtaining Method

CloudTable

● HBase

● OpenTSDB

DWS



ServiceName

Obtaining Method

RDS

CSS

MRS




201● Response parameters


Parameter Mandatory

Type Description

is_success No Boolean Indicates whether the request issuccessfully executed. Value trueindicates that the request issuccessfully executed.

message No String System prompt. If execution succeeds,the parameter setting may be leftblank.

connection_id No String Connection ID. Identifies the UUID ofa datasource connection.


{ "name": "test1", "security_group_id": "a07ab107-a4ad-4835-ad09-08a2f2efccdb", "network_id": "e83ba67d-e812-42d3-908d-6ea0b2cc5ecb", "url": "cloudtable-8636-zk2-ldQklrNa.mycloudtable.com:2181,cloudtable-8636-zk3-N9EEH4mI.mycloudtable.com:2181,cloudtable-8636-zk1-78Lt0l8L.mycloudtable.com:2181", "service": "ct"}

● Example response (successful request){ "is_success": true, "message": "", "connection_id": "68233202-63bf-4f27-a99b-fa5f5a2dbae1"}


10.2 Deleting a Basic Datasource Connection

FunctionThis API is used to delete a created basic datasource connection.

The connection that is being created cannot be deleted.



URI● URI format

DELETE /v2.0/{project_id}/datasource-connection/{connection_id}● Parameter description


Parameter Mandatory

Description

project_id Yes Project ID, which is used for resourceisolation. For details about how to obtainits value, see Obtaining a Project ID.

connection_id Yes Connection ID. Identifies the UUID of adatasource connection.

Request

None

Response


Parameter

Mandatory

Type Description

is_success


message No String System message. Value Deleted indicates thatthe operation is successful.


None

● Example response (successful request){ "is_success": true, "message": "Deleted"}




10.3 Querying a Basic Datasource Connection List

FunctionThis API is used to query the list of created basic datasource connections.

URI● URI format

GET /v2.0/{project_id}/datasource-connection● Parameter description


Parameter

Mandatory

Description


RequestNone

Response


Parameter

Mandatory

Type Description



connections

No Array ofobjects

Datasource connection information list. Fordetails, see Table 10-10.


None

● Example response (successful request){ "is_success": true,



"message": "", "connections": [ { "is_success": true, "message": "", "connection_id": "75419c22-7a31-42e4-a3f3-102f6e6895c9", "destination": "cloudtable-c172-zk3-WBFt4fr2.mycloudtable.com:2181,cloudtable-c172-zk2-LHZEWNau.mycloudtable.com:2181,cloudtable-c172-zk1-cvwY6lMh.mycloudtable.com:2181", "state": "DELETED", "process": 0, "name": "test", "connection_url": "", "cluster_name": "", "service": "CloudTable", "create_time": 1549092947256 } ] }


10.4 Querying a Basic Datasource Connection

FunctionThis API is used to query a created basic datasource connection.

URI● URI format

GET /v2.0/{project_id}/datasource-connection/{connection_id}● Parameter description


Parameter Mandatory

Description


connection_id

Yes Connection ID. Identifies the UUID of a datasourceconnection.

RequestNone



Response


Parameter Mandatory

Type Description




connection_id

No String Connection ID. Identifies the UUID of adatasource connection.

destination

No String Queue access address entered by the user duringconnection creation.

state No String Connection status. For details about the statuscode, see Table 10-11.

process No Double

Progress of the datasource connection that isbeing created. The value ranges from 0.0 to 1.0,indicating 0% to 100%.

name No String User-defined connection name.

connection_url

No String Connection URL used to create a datasourceassociation table.

cluster_name

No String Name of the Serverless Spark queue. For thedatasource connection established in SQL queuemode, this field is left blank.

service No String The peer service (CloudTable/CloudTable.OpenTSDB/MRS.OpenTSDB/DWS/RDS/CSS) specified by theuser during connection creation.

create_time

No Long Time when a link is created. The time isconverted to a UNIX timestamp.

Table 10-11 Connection Status

Parameter Description Description

CREATING Creating The datasourceconnection is beingcreated.




ACTIVE Active The datasourceconnection has beencreated, and theconnection to thedestination address isnormal.

FAILED Failed Failed to create adatasource connection.

DELETED Deleted The datasourceconnection has beendeleted.


None

● Example response (successful request){ "is_success": true, "message": "", "connection_id":"378eb1e6-521d-41a4-a41c-bbb58f34a6d6", "destination":"cloudtable-cf82-zk3-pa6HnHpf.cloudtable.com:2181,cloudtable-cf82-zk2-weBkIrjI.cloudtable.com:2181,cloudtable-cf82-zk1-WY09px9l.cloudtable.com:2181", "state": "CREATING", "process": 0, "name": "op", "connection_url": "", "cluster_name": "", "service":"CloudTable.OPENTSDB", "create_time": 1566180327674 }




11 APIs Related to the EnhancedDatasource Connections

11.1 Creating an Enhanced Datasource Connection

FunctionThis API is used to create an enhanced datasource connection with other services.

● For details about how to use the console interface, see Enhanced DatasourceConnections in the Data Lake Insight User Guide.

● Datasource connections cannot be created for the default queue.

URI● URI format

POST /v2.0/{project_id}/datasource/enhanced-connections● Parameter description


Parameter Mandatory

Description


Data Lake InsightAPI Reference

11 APIs Related to the Enhanced DatasourceConnections


Request


Parameter Mandatory

Type Description

name Yes String Name of the connection. The value consistsof 64 characters, including digits, letters, andunderscores (_).

dest_vpc_id

Yes String The ID of the service VPC to be connected.For details, see Creating a VPC in VirtualPrivate Cloud API Reference.

dest_network_id

Yes String The subnet ID of the to-be-connected service.For details, see Creating a Subnet in VirtualPrivate Cloud API Reference.

queues No Array ofstrings

List of queue names that are available fordatasource connections.NOTE

Only yearly/monthly queues and dedicated queuesare allowed to use the enhanced datasourceconnections.

hosts No Array ofobjects

The user-defined host information. Amaximum of 20,000 records are supported.For details, see hosts request parameters.

Table 11-3 hosts request parameters

Parameter Mandatory

Type Description

name No String The user-defined host name. The value canconsist of 128 characters, including digits,letters, underscores (_), hyphens (-), andperiods (.). It must start with a letter.

ip No String The IPv4 address of the host.


201● Response parameter




https://support.huaweicloud.com/en-us/api-vpc/vpc_api01_0001.html

https://support.huaweicloud.com/en-us/api-vpc/vpc_subnet01_0001.html



is_success Boolean Indicates whether the request is successfullyexecuted. Value true indicates that the requestis successfully executed.

message String System prompt. If execution succeeds, theparameter setting may be left blank.

connection_id String Connection ID. Identifies the UUID of adatasource connection.


{ "name": "test", "dest_vpc_id": "22094d8f-c310-4621-913d-4c4d655d8495", "dest_network_id": "78f2562a-36e4-4b39-95b9-f5aab22e1281", "queues": [ "q1", "q2" ], "hosts": [ { "ip":"192.168.0.1", "name":"ecs-97f8-0001" }, { "ip":"192.168.0.2", "name":"ecs-97f8-0002" } ]}

● Example response (successful request){ "is_success": true, "message": "", "connection_id": "2a620c33-5609-40c9-affd-2b6453071b0f"}


11.2 Deleting an Enhanced Datasource Connection

Function

This API is used to delete an enhanced datasource connection.

The connection that is being created cannot be deleted.




URI● URI format

DELETE /v2.0/{project_id}/datasource/enhanced-connections/{connection_id}● Parameter description


Parameter Mandatory

Description



RequestNone

Response



is_success Boolean Indicates whether the request is successfullyexecuted. Value true indicates that the request issuccessfully executed.

message String System message. Value Deleted indicates thatthe operation is successful.


{ "is_success": true, "message": "Deleted"}





11.3 Querying an Enhanced Datasource Connection List

Function

This API is used to query the list of created enhanced datasource connections.

URI● URI format

GET /v2.0/{project_id}/datasource/enhanced-connections?limit={limit}&offset={offset}&status={status}&name={name}



Parameter Mandatory

Description


limit No The maximum number of connections to bequeried. The default value is 100.

offset No The offset of the query result. The default value is0. Note that the connections are sorted by creationtime.

status No Connection status. The options are as follows:● Active: The connection has been activated.● DELETED: The connection has been deleted.

name No Connection name

Request

None

Response









connections Array ofobjects

Datasource connection information list. For details,see Table 11-10.

count Integer Number of returned datasource connections.


{ "is_success": true, "message": "", "count": 1, "connections": [ { "name": "withvpc", "id": "4c693ecc-bab8-4113-a838-129cedc9a563", "available_queue_info": [ { "status": "ACTIVE", "name": "resource_mode_1", "peer_id": "d2ae6628-fa37-4e04-806d-c59c497492d1", "err_msg": "", "update_time": 1566889577861 } ], "dest_vpc_id": "22094d8f-c310-4621-913d-4c4d655d8495", "dest_network_id": "78f2562a-36e4-4b39-95b9-f5aab22e1281", "create_time": 1566888011125, "status": "ACTIVE" } ]}


11.4 Querying an Enhanced Datasource Connection

Function

This API is used to query the created enhanced datasource connections.

URI● URI format

GET /v2.0/{project_id}/datasource/enhanced-connections/{connection_id}






Parameter Mandatory

Description



Request

None

Response



is_success Boolean Indicates whether the request is successfullyexecuted. Value true indicates that therequest is successfully executed.


id String Connection ID. Identifies the UUID of adatasource connection.

name String User-defined connection name.

status String Connection status. The options are as follows:● Active: The connection has been activated.● DELETED: The connection has been

deleted.

available_queue_info

Array ofobjects

For details about how to create a datasourceconnection for each queue, see Table 11-11.

dest_vpc_id String The VPC ID of the connected service. Fordetails, see Querying VPC Details in VirtualPrivate Cloud API Reference.

dest_network_id String Subnet ID of the connected service. Fordetails, see Querying Subnet Details inVirtual Private Cloud API Reference.

create_time Long Time when a link is created. The time isconverted to a UNIX timestamp.




https://support.huaweicloud.com/en-us/api-vpc/vpc_api01_0002.html

https://support.huaweicloud.com/en-us/api-vpc/vpc_subnet01_0002.html


hosts Array ofobjects

User-defined host information. For details, seehosts parameter description.

Table 11-11 available_queue_info parameter description


peer_id String ID of a datasource connection.

status String Connection status. For details about the status code,see Table 11-13.

name String Name of a queue.

err_msg String Detailed error message when the status is FAILED.

update_time Long Time when the available queue list was updated.

Table 11-12 hosts parameter description


name String The user-defined host name.

ip String The IPv4 address of the host.

Table 11-13 Connection Status


CREATING Creating The datasource connection is beingcreated.

ACTIVE Active The datasource connection has beencreated, and the connection to thedestination address is normal.

FAILED Failed Failed to create a datasourceconnection.


{ "is_success": true, "message": "", "name": "withvpc", "id": "4c693ecc-bab8-4113-a838-129cedc9a563",




"available_queue_info": [ { "status": "ACTIVE", "name": "resource_mode_1", "peer_id": "d2ae6628-fa37-4e04-806d-c59c497492d1", "err_msg": "", "update_time": 1566889577861 } ], "dest_vpc_id": "22094d8f-c310-4621-913d-4c4d655d8495", "dest_network_id": "78f2562a-36e4-4b39-95b9-f5aab22e1281", "create_time": 1566888011125, "status": "ACTIVE", "hosts": [ { "ip":"192.168.0.1", "name":"ecs-97f8-0001" }, { "ip":"192.168.0.2", "name":"ecs-97f8-0002" } ]}


11.5 Binding a Queue

Function

This API is used to bind a queue to a created enhanced datasource connection.

URI● URI format

POST /v2.0/{project_id}/datasource/enhanced-connections/{connection_id}/associate-queue



Parameter Mandatory

Description


connection_id

Yes Connection ID. Identifies the UUID of adatasource connection.




Request


Parameter Mandatory

Type Description

queues Yes List List of queue names that are available fordatasource connections.

Response


Parameter

Type Description




{ "queues": [ "q1", "q2" ]}

● Example response (successful request){ "is_success": true, "message": "associated peer connection for queues: {q1,q2}."}


11.6 Unbinding a Queue

Function

This API is used to unbind a queue from an enhanced datasource connection.




URI● URI format

POST /v2.0/{project_id}/datasource/enhanced-connections/{connection_id}/disassociate-queue



Parameter Mandatory

Description


connection_id


Request


Parameter Mandatory

Type Description

queues Yes List List of queue names that are available fordatasource connections.

Response


Parameter

Type Description




{ "queues": [ "q1",




"q2" ]}

● Example response (successful request){ "is_success": true, "message": "Disassociated peer connection for queues:{q1,q2}."}


11.7 Modifying the Host Information

Function

This API is used to modify the host information of a connected datasource. Onlyfull overwriting is supported.

URI● URI format

PUT /v2.0/{project_id}/datasource/enhanced-connections/{connection_id}● Parameter description


Parameter Mandatory

Description


connection_id


Request


Parameter Mandatory

Type Description

hosts Yes Array ofobjects

The user-defined host information. Amaximum of 20,000 records are supported.For details, see hosts request parameters.If this parameter is left blank, allconfigured host information will be deleted.




Table 11-22 hosts request parameters

Parameter Mandatory

Type Description

name No String The user-defined host name. The value canconsist of 128 characters, including digits,letters, underscores (_), hyphens (-), andperiods (.). It must start with a letter.

ip No String The IPv4 address of the host.

Response


Parameter

Type Description




{ "hosts": [ { "ip":"192.168.0.1", "name":"ecs-97f8-0001" }, { "ip":"192.168.0.2", "name":"ecs-97f8-0002" } ]}









12 Permissions Policies and SupportedActions

This chapter describes fine-grained permissions management for your DLI. If yourHUAWEI CLOUD account does not need individual IAM users, then you may skipover this chapter.

By default, new IAM users do not have permissions assigned. You need to add theusers to one or more groups, and attach permissions policies or roles to thesegroups. The users then inherit permissions from the groups to which they areadded. After authorization, the user can perform specified operations on MRSbased on the permissions. For more information about policy syntax and examplepolicies, see Roles and Policies.

Type: There are roles and policies.● Roles: A type of coarse-grained authorization mechanism that defines

permissions related to user responsibilities. This mechanism provides only alimited number of service-level roles for authorization. When using roles togrant permissions, you need to also assign other roles on which thepermissions depend to take effect. However, roles are not an ideal choice forfine-grained authorization and secure access control.

● Policies: A type of fine-grained authorization mechanism that definespermissions required to perform operations on specific cloud resources undercertain conditions. This mechanism allows for more flexible policy-basedauthorization, meeting requirements for secure access control. For example,you can grant DLI users only the permissions for managing a certain type ofcloud servers.

Policy-based authorization is useful if you want to allow or deny the access to an API.

A HUAWEI CLOUD account has all of the permissions required to call all APIs, butIAM users must have the required permissions specifically assigned. Thepermissions required for calling an API are determined by the actions supported bythe API. Only users who have been granted permissions allowing the actions cancall the API successfully. For example, if an IAM user needs to create buckets usingan API, the user must have been granted permissions that allow thedli:queue:create_queue action.

Data Lake InsightAPI Reference 12 Permissions Policies and Supported Actions




Supported ActionsOperations supported by policies are specific to APIs. The following are commonconcepts related to policies:

● Permissions: defined by actions in a custom policy.● Actions: added to a custom policy to control permissions for specific

operations.● Authorization Scope: A custom policy can be applied to IAM projects or

enterprise projects or both. Policies that contain actions supporting both IAMand enterprise projects can be assigned to user groups and take effect in bothIAM and Enterprise Management. Policies that only contain actionssupporting IAM projects can be assigned to user groups and only take effectfor IAM. Such policies will not take effect if they are assigned to user groupsin Enterprise Project. For details about the differences between IAM andenterprise projects, see Differences Between IAM and EnterpriseManagement.

● APIs: RESTful APIs that can be called in a custom policy.

DLI supports the following actions that can be defined in custom policies:

Table 12-1 Actions

Permission

Actions AuthorizationScope

APIs

Creating aQueue

dli:queue:create_queue

Supported:ProjectNot supported:none

POST /v1.0/{project_id}/queues

Deleting aQueue

dli:queue:drop_queue


DELETE /v1.0/{project_id}/queues/{queue_name}

Submitting a Job

dli:queue:submit_job


POST /v1.0/{project_id}/jobs/submit-job

Cancelinga job

dli:queue:cancel_job


DELETE /v1.0/{project_id}/jobs/{job_id}

ViewingQueuePermissions of OtherUsers

dli:queue:show_privileges


GET /v1.0/{project_id}/queues/{queue_name}/users



https://support.huaweicloud.com/en-us/iam_faq/iam_01_0101.html

https://support.huaweicloud.com/en-us/iam_faq/iam_01_0101.html

Permission


APIs

Creating aDatabase

dli:database:create_database


POST /v1.0/{project_id}/databases

Deleting aDatabase

dli:database:drop_database


DELETE /v1.0/{project_id}/databases/{database_name}

Explainingthe SQLStatementas anExecutionPlan

dli:database:explain


POST /v1.0/{project_id}/jobs/submit-job, SQLstatement invoking

Creating aTable

dli:database:create_table


POST /v1.0/{project_id}/databases/{database_name}/tables

Creating aView

database:create_view



Creating aFunction

dli:database:create_function



Describinga Function

dli:database:describe_function



Deleting aFunction

dli:database:drop_function



Displayinga Function

dli:database:show_functions





Permission


APIs

Creating arole

dli:database:create_role



Deleting arole

dli:database:drop_role



Displayinga Role

dli:database:show_roles



DisplayingAll Roles

dli:database:show_all_roles



Binding aRole

dli:database:grant_role



Unbindingthe Role

dli:database:revoke_role



DisplayingtheBindingRelationshipsBetweenAll Rolesand Users

dli:database:show_users



ViewingDatabasePermissions of OtherUsers

dli:database:show_privileges


GET /v1.0/{project_id}/databases/{database_name}/users

Deleting aTable

dli:table:drop_table


DELETE /v1.0/{project_id}/databases/{database_name}/tables/{table_name}



Permission


APIs

DisplayingTableStructure

dli:table:describe_table


GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}

Queryinga Table

dli:table:select Supported:ProjectNot supported:none

GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/previewPOST /v1.0/{project_id}/jobs/submit-job, SQLstatement invoking

Displayingthe TableCreationStatement

dli:table:show_create_table


GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/preview

DisplayingAllPartitions

dli:table:show_partitions



SettingTableConfiguration

dli:table:alter_table_set_properties



Adding aColumn

dli:table:alter_table_add_columns



AddingPartitionsto thePartitioned Table

dli:table:alter_table_add_partition



Renaminga TablePartition

dli:table:alter_table_rename_partition



DeletingPartitionsfrom aPartitioned Table

dli:table:alter_table_drop_partition





Permission


APIs

RestoringTablePartitions

dli:table:alter_table_recover_partition



Renaminga Table

dli:table:alter_table_rename



SettingthePartitionPath

dli:table:alter_table_set_location



InsertingData to aTable

dli:table:insert_into_table


POST /v1.0/{project_id}/jobs/submit-job, statementinvoking

Overwriting a Table

dli:table:insert_overwrite_table



ViewingTablePermissions of OtherUsers

dli:table:show_privileges


GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/users/{user_name}GET /v1.0/{project_id}/databases/{database_name}/tables/{table_name}/users

Modifyinga View

dli:table:alter_view Supported:ProjectNot supported:none


Queryinga Column

dli:column:select Supported:ProjectNot supported:none


Queryingthe FlinkJob List

dli:jobs:list_job Supported:ProjectNot supported:none

GET /v1.0/:x_project_id/jobs



Permission


APIs

QueryingFlink JobDetails

dli:jobs:get_job Supported:ProjectNot supported:none

GET /v1.0/:x_project_id/job/:job_id

Creating aFlink Job

dli:jobs:create_job Supported:ProjectNot supported:none

POST /v1.0/:x_project_id/sql_job

Updatinga Flink Job

dli:jobs:update_job Supported:ProjectNot supported:none

PATCH /v1.0/:x_project_id/sql_job

Deleting aFlink Job

dli:jobs:delete_job Supported:ProjectNot supported:none

DELETE /v1.0/:x_project_id/job/:job_id

Starting aFlink Job

dli:queue:submit_job


POST v1.0/:x_project_id/job/:job_id/run

Stopping aFlink Job

dli:queue:cancel_job


POST /v1.0/:x_project_id/job/:job_id/stop



13 Appendix

13.1 Status CodesTable 13-1 describes status codes.


StatusCode

Message Description

100 Continue The client should continue with its request.This interim response is used to inform the client thatpart of the request has been received and has not yetbeen rejected by the server.

101 SwitchingProtocols

The protocol should be switched. The protocol canonly be switched to a newer protocol.For example, the current HTTP protocol is switchedto a later version of HTTP.

200 Success The request has been fulfilled. This indicates that theserver has provided the requested web page.

201 Created The request is successful and the server has created anew resource.

202 Accepted The request has been accepted, but the processinghas not been completed.

203 Non-AuthoritativeInformation

Unauthorized information. The request is successful.

Data Lake InsightAPI Reference 13 Appendix


StatusCode

Message Description

204 NoContent The server has successfully processed the request, butdoes not return any content.The status code is returned in response to an HTTPOPTIONS request.

205 Reset Content The server has successfully processed the request, butdoes not return any content.

206 Partial Content The server has successfully processed the partial GETrequest.

300 Multiple Choices There are multiple options for the location of therequested resource. The response contains a list ofresource characteristics and addresses from which auser terminal (such as a browser) can choose themost appropriate one.

301 MovedPermanently

The requested resource has been assigned a newpermanent URI, and the new URI is contained in theresponse.

302 Found The requested resource resides temporarily under adifferent URI.

303 See Other The response to the request can be found under adifferent URI,and should be retrieved using a GET or POSTmethod.

304 Not Modified The requested resource has not been modified. Insuch a case, there is no need to retransmit theresource since the client still has a previously-downloaded copy.

305 Use Proxy The requested resource is available only through aproxy.

306 Unused The HTTP status code is no longer used.

400 BadRequest Invalid request.The client should not repeat the request withoutmodifications.

401 Unauthorized This status code is returned after the client providesthe authentication information, indicating that theauthentication information is incorrect or invalid.

402 PaymentRequired

This status code is reserved for future use.



StatusCode

Message Description

403 Forbidden The server has received the request and understoodit, but the server is refusing to respond to it.The client should modify the request instead of re-initiating it.

404 NotFound The requested resource cannot be found.The client should not repeat the request withoutmodifications.

405 MethodNotAllowed

A request method is not supported for the requestedresource.The client should not repeat the request withoutmodifications.

406 Not Acceptable The server could not fulfill the request according tothe content characteristics of the request.

407 ProxyAuthenticationRequired

This code is similar to 401, but indicates that theclient must first authenticate itself with the proxy.

408 Request Time-out

The server has timed out waiting for the request.The client may repeat the request withoutmodifications at any time later.

409 Conflict The request could not be processed due to a conflictin the request.This status code indicates that the resource that theclient is attempting to create already exists, or thatthe request has failed to be processed because of theupdate of the conflict request.

410 Gone The requested resource cannot be found.The status code indicates that the requested resourcehas been deleted permanently.

411 Length Required The server is refusing to process the request withouta defined Content-Length.

412 PreconditionFailed

The server does not meet one of the preconditionsthat the requester puts on the request.

413 Request EntityToo Large

The server is refusing to process a request becausethe request entity is too large for the server toprocess. The server may disable the connection toprevent the client from sending requestsconsecutively. If the server is only temporarily unableto process the request, the response will contain aRetry-After header field.



StatusCode

Message Description

414 Request-URI TooLarge

The Request-URI is too long for the server to process.

415 UnsupportedMedia Type

The server does not support the media type in therequest.

416 Requested rangenot satisfiable

The requested range is invalid.

417 ExpectationFailed

The server has failed to meet the requirements of theExpect request-header field.

422 UnprocessableEntity

The request was well-formed but was unable to befollowed due to semantic errors.

429 TooManyRequests

The client sends excessive requests to the serverwithin a given time (exceeding the limit on theaccess frequency of the client), or the server receivesexcessive requests within a given time (beyond itsprocessing capability). In this case, the client shouldresend the request after the time specified in theRetry-After header of the response has elapsed.

500 InternalServerEr-ror

The server is able to receive the request but unableto understand it.

501 NotImplemented

The server does not support the requested function.

502 Bad Gateway The server was acting as a gateway or proxy andreceived an invalid request from the remote server.

503 ServiceUnavaila-ble

The requested service is invalid.The client should not repeat the request withoutmodifications.

504 ServerTimeout The request cannot be fulfilled within a given time.This status code is returned to the client only whenthe Timeout parameter is specified in the request.

505 HTTP Version notsupported

The server does not support the HTTP protocolversion used in the request.

13.2 Error CodeIf an error occurs in API calling, no result is returned. Identify the cause of errorbased on the error codes of each API. If an error occurs in API calling, HTTP statuscode 4xx or 5xx is returned. The response body contains the specific error code



and information. If you are unable to identify the cause of an error, contacttechnical personnel and provide the error code so that we can help you solve theproblem as soon as possible.

Format of an Error Response Body

If an error occurs during API calling, the system returns an error code and amessage to you. The following shows the format of an error response body:

{ "error_msg": "The format of message is error", "error_code": "AS.0001"}

In the preceding information, error_code is an error code, and error_msgdescribes the error.

Table 13-2 Exceptions

Parameter Parameter Type Description

error_code String Error code. For details,see Table 13-3.

error_msg String Error details.

Error Code Description


Status Code Error Code Error Message

400 DLI.0001 Parameter check errors occur.

400 DLI.0002 The object does not exist.

400 DLI.0003 SQL permission verificationfails.

400 DLI.0004 SQL syntax parsing errorsoccur.

400 DLI.0005 SQL semantics parsing errorsoccur.

400 DLI.0006 The object exists.

400 DLI.0007 Feature not supported.

400 DLI.0008 Metadata errors occur.

400 DLI.0009 System restrictions

400 DLI.0011 The file permission check fails.



Status Code Error Code Error Message

400 DLI.0012 Resource objects areunavailable.

401 DLI.0013 User authentication errorsoccur.

401 DLI.0014 Service authentication errorsoccur.

400 DLI.0016 The identity and role areincorrect.

400 DLI.0018 Data conversion errors occur.

400 DLI.0019 The task times out.

400 DLI.0100 The result expires.

404 DLI.0023 No related resources werefound.

500 DLI.0999 Server-side errors occur.

ExampleIf no queue named testqueue exists, the following error message is displayedwhen you submit a job submission request:

{ "error_code": "DLI.0008", "error_msg": "There is no queue named testqueue"}

13.3 Obtaining a Project ID

Obtaining a Project ID by Calling an APIYou can obtain a project ID by calling the API used to Querying ProjectInformation.

The API for obtaining a project ID is GET https://{Endpoint}/v3/projects/.{Endpoint} indicates the endpoint of IAM, which can be obtained from Regionsand Endpoints.

The following is an example response. The value of id under projects is the projectID.

{ "projects": [ { "domain_id": "65382450e8f64ac0870cd180d14e684b", "is_domain": false, "parent_id": "65382450e8f64ac0870cd180d14e684b", "name": "cn-north-4", "description": "",





https://developer.huaweicloud.com/en-us/endpoint

https://developer.huaweicloud.com/en-us/endpoint

"links": { "next": null, "previous": null, "self": "https://www.example.com/v3/projects/a4a5d4098fb4474fa22cd05f897d6b99" }, "id": "a4a5d4098fb4474fa22cd05f897d6b99", "enabled": true } ], "links": { "next": null, "previous": null, "self": "https://www.example.com/v3/projects" }}

Obtaining a Project ID from the Management ConsoleA project ID is required for some URIs when an API is called.

To obtain a project ID, perform the following operations:

1. Log in to the management console.2. Hover the mouse over the username and select Basic Information.3. On the displayed page, click Manage in Security Credentials.

On the Projects tab of the API Credentials page, view project IDs.

Figure 13-1 Obtaining a Project ID

If there are multiple projects in one region, expand Region and viewsubproject IDs in the Project ID column.

13.4 Obtaining an Account IDAn account ID (domain-id) is required for some URLs when an API is called. Toobtain an account ID, perform the following operations:

1. Log in to the management console.2. Click the username and select My Credentials from the drop-down list.

On the My Credentials page, view the Account ID.Figure 1-1 Obtaining an account ID





A Change History

Table A-1 Change history

Released On Description

2020-2-21 This issue is the twentieth officialrelease.Added the following sections:4.5- Restarting, Scaling Out, andScaling In Queues4.6- Viewing Details of a Queue5.2.8- Obtaining the Partition List9- APIs Related to Stream Jobs10- APIs related to Stream jobs

2020-1-22 This issue is the nineteenth officialrelease.Added the following sections:11- Permissions Policies and SupportedActions

2019-8-16 This is the eighteenth official release.Added the following sections:4.4- APIs Related to EnhancedDatasource Connections

2019-5-15 This issue is the seventeenth officialrelease.● Deleted the following sections:

5.8-APIs Related to Gene JobSubmission

Data Lake InsightAPI Reference A Change History



2018-12-13 This issue is the sixteenth officialrelease.● Added the following sections:

5.9-APIs Related to DatasourceConnection of DLI5.8.11-Submitting a CNVkit job5.8.12-Querying a CNVkit Job

2018-8-28 This issue is the fifteenth officialrelease.● Added the following sections:

5.8-APIs Related to Gene JobSubmission

2018-7-26 This issue is the fourteenth officialrelease.● Added the following sections:

5.10.1-Interconnecting withCloudTable HBase

2018-5-31 This issue is the thirteenth officialrelease.● Added the following sections:

5.7-APIs Related to Spark JobSubmission

2018-4-25 This issue is the twelfth official release.● Added the following sections:

5.2.13-Checking SQL Syntax5.4.4-Previewing Table Content

2018-4-19 This issue is the eleventh officialrelease.● Changed the service name from

UQuery to DLI.

2018-2-6 This issue is the tenth official release,which incorporates the followingchanges:● Added the following sections:

5.1.2-Creating a Queue.5.1.3-Deleting a Queue6.3-Error Code

● Modified the parameters andexamples and added the errordescription in each section in 5-APIs.




2018-01-22 This issue is the ninth official release,which incorporates the followingchanges:● Revised the document based on the

template.● Added the following section:

1-Before You Start● Adjusted all sections in 5-APIs.● Changed "Common Header" to

"Common Parameters".● Added the following sections:

5.2.1-Submitting a SQL Job(Recommended)5.2.7-Querying the Job ExecutionResult (Recommended)5.2.11-Canceling a Job(Recommended)5.4.1-Querying All Tables(Recommended)

2017-12-28 This issue is the eighth official release,which incorporates the followingchanges:● Modified the document as follows:

– 2. API Overview– 4.1-Service Usage– 5-All sections in 5-APIs

2017-11-30 This issue is the seventh officialrelease, which incorporates thefollowing changes:● Modified the document as follows:

– 5.2.3-Importing Data– 5.2.4-Exporting Data– 5.2.6-Querying Job Details– 5.2.9-Exporting Query Results– 5.5.1-Granting Users with the

Queue Usage Permission– 5.5.3-Granting Users with the

Data Usage Permission– 5.5.5-Querying Table Users




2017-11-22 This issue is the sixth official release,which incorporates the followingchanges:● Modified the document as follows:

– 3.1-Obtaining RequestAuthentication Information

– 3.2-Obtaining a Project ID– 6.1-Common Request Parameter– 5.2.3-Importing Data– 5.2.4-Exporting Data– 5.2.5-Querying Job Status– 5.2.6-Querying Job Details– 5.2.10-Querying All Jobs– 5.3.2-Querying All Databases– 5.4.2-Querying All Tables

(Deprecated)– 5.4.5-Creating a Table– 5.5-Permission-related APIs

2017-11-03 This issue is the fifth official release,which incorporates the followingchange:● Modified the document as follows:

– 4-API

2017-10-28 This issue is the fourth official release.

2017-09-30 This issue is the third official release.

2017-09-12 This issue is the second official release.● Modified the document as follows:

– Changed user_id to user_name.– Changed queue_id to

queue_name.● Deleted the following sections:

– 4.2.1-Adding a Queue– 4.2.3-Deleting a Queue

2017-08-25 This issue is the first commercialrelease.



Date post:	09-May-2020
Category:	Documents
Upload:	others
View:	8 times
Download:	0 times

support.huaweicloud.com · 12 Permissions Policies and Supported Actions This chapter describes...

Documents