Taking advantage of the built-in developer tools – Couchbase Connect 2016

©2016 Couchbase Inc. 1

The Couchbase Connect16 mobile appTake our in-app survey!


Taking Advantage of Built-In Developer Tools

Examining 2012 Presidential Campaign Finance Data with Couchbase

©2016 Couchbase Inc. 3©2016 Couchbase Inc.

Roadmap

• Importing Data with cbimport• Exploring Data with Query Workbench• Power query features with Couchbase Query Shell (cbq)


Importing DataMike Wiederhold, Senior Software

Engineer, Couchbase


What is cbimport?

• Allows importing of JSON and CSV/TSV data• Ability to read data locally or from an HTTP URL• Provides key generation capabilities based on the data being imported


Importing JSON data

• Unlike CSV data JSON datasets can come in a variety of formats• Some datasets have a JSON object on each line• Some have a list of JSON objects• Some are Couchbase specific formats

Object Per Line:

{ “field1”: “value1”}{ “field2”: “value2”}{ “field3”: “value3”}{ “field4”: “value4”}{ “field5”: “value5”}{ “field6”: “value6”}

List of Objects:

[ { “field1”: “value1”}, { “field2”: “value2”}, { “field3”: “value3”}, { “field4”: “value4”}, { “field5”: “value5”}, { “field6”: “value6”},]

Folder containing objects:

/root doc1.json doc2.json doc3.json doc4.json doc5.json doc6.json


Importing CSV/TSV Data

• Allows importing of JSON and CSV/TSV data• Separator is configurable so custom separated files also work

Comma Separated Files

field1,field2,field3value1,value2,value3value1,value2,value3value1,value2,value3value1,value2,value3value1,value2,value3

Tab Separated Files

field1 field2 field3value1 value2 value3value1 value2 value3value1 value2 value3value1 value2 value3value1 value2 value3


Using Key Generators

• Ability to generate keys based on the value of fields as well as custom generators

Given the document:

{ "name": "alice", "age": 40}

Given the key generator:

key::%name%::#MONO_INCR#

Result Key: key::alice::1


Query WorkbenchEben Haber, Senior Software Engineer &

User Advocate, Couchase

©2016 Couchbase Inc.©2016 Couchbase Inc.

Query Workbench – new UI feature in 4.5• cbq limitations:

• Hard to edit long queries• Output hard to read• Little info on buckets,

document structure

• Query Workbench:• Multi-line query editor

with syntax coloring, auto-completion

• Query output shown as JSON, table, or tree

• List of buckets shown, with bucket schema inferencing


Query Workbench

Enter N1QL queries here

View previous queries/results

See results as JSON, table, or tree

View buckets, schemas here


Schemas?• Couchbase does not enforce schemas, but documents

usually share common structure(s)

• The Query Workbench includes schema inferencing, which uses a random sample of documents to identify common fields/schemas

• E.g., for beer-sample, it finds two distinct ‘flavors’ of schema

• When you get fields in the Bucket Analysis panel, those fields become available for auto-complete

• Hover the mouse over a field name to see sample values for that field as a tool tip


Autocompletion• The query editor can autocomplete N1QL keywords, bucket names, and even field

names when you click ‘tab’ in the middle of a word.


Data Display• Reading/comparing multiple JSON docs can be hard, so data can also be shown

as table, tree:

• N.B. hovering over cell in table/tree view will show path to that value


Other Features• File I/O: Can save queries or query results to a file

• Query History: Past queries are stored in the browser’s local storage, so queries you have run remain visible across sessions (but not query results)

• Can run schema inference as a query to get more details, e.g.,• infer `beer-sample` with {“sample_size” : 2000}• This runs the inferencer on a sample of 2000 documents, and will show how many

documents are in which flavor, how frequently each field appears, shows multiple sample values for each field, etc.

• Schema inference can also highlight dirty data – fields with multiple types, etc.


Query Workbench Limitations

• Workbench times out for queries taking > 5 minutes• use cbq in for very long running queries)

• Displaying large query results can be slow in the web browser• JSON view about 10x faster than table, tree views• If rendering expected to take > 5 seconds, “are you sure” link appears• Tip: even if results not shown, you can still save them to a file

©2016 Couchbase Inc.

DEMO


CBQ - Command line shell for N1QL

Isha Kandaswamy, Software Developer, Couchbase


CBQ- New CLI for N1QL in 4.5

• What is cbq ?• Command line shell that provides an interface to the Query engine allowing you to

run queries interactively. Additionally one can also issue scripting commands.• Shell commands start with a ' \’

• When cbq starts up, it is given a set of command line options. If no options are present then it assumes default values for expected options.

• It is important to note that all shell commands are case insensitive. Command line options on the other hand are case sensitive.

• The set of command line options and shell commands can be classified into the following

Connection Management Security ManagementSession Management Query Management Shell/ Server Management Scripting support

©2016 Couchbase Inc.20©2016 Couchbase Inc.

CBQ commands and options• Connection Management

• Manage connections between cbq and the server.• Connect to the cluster : Auto discover query services and round robin amongst them.• Connect to a specific query service directly. • Don’t connect to any endpoint• Switch between connections

• Security Management • Provide mechanism for authentication.

• Role based access : Provide username and password when bringing up the shell.• Protected bucket access : Pass single/ multi-bucket credentials on a per query basis. (using

query rest api parameter creds)• Secure connections


CBQ commands and options• Session Management

• Set user preferences and manage different shell sessions.• Echo statements. • Scroll through history, reverse search, partial query auto lookup.• Set/ Unset session and server level parameter settings.• Alias

• Query Management • Manage interaction of the shell with the query service• Cancel/ Abort running queries.• Ability to set positional and Named parameters.• Prepared and Named Prepared statements. • Shell comments (# and --)


CBQ commands and options• Shell/ Server Management

• Convey information about the shell/cluster endpoint• Display versions for the shell and server and display copyright information.• Shortcut-keys support• Edit commands saved in the history

• Scripting support • Interpret and run single/multiple shell and N1QL commands in cbq – Multi-line input and

Batch mode• Read commands from a file and execute them• Redirect output of queries to a file.• Piping commands into cbq• Exit on error• Display specific parts of the results of a query


• In order to handle Query Rest API/Named parameters with the query service and allow for better scripting, cbq allows 3 kinds of parameters.

• Every parameter is of the type <prefix><name>. The prefix is used to distinguish between each type. • The values for these parameters can be set using the \SET command and unset using the \UNSET

command.\SET –args [1, "Step 1”] \UNSET –args

• All settings via the Rest API are settable via the shell in the form of Query Parameters prefixed by - .

Prefix Parameter type Example- Query parameters \SET –creds beer-sample:pass;

No prefix

$

Session parametersPre-defined parameters

User-defined parameters

\SET histfile .new_cbqhistory;

\SET $temp 5;

-$ Named parameters \SET -$rate 9.5;

CBQ – Parameters


Shell Commands :

Command Args Description Example

\SET Parameter Value

Resets the topmost value of the stack for the input parameter with the input value. cbq> \SET -args ["1", "Ok"] ;

\SET none Display the values of all the parameters for the current session.

cbq>\SET;Query Parameters : Parameter name : args Value : [["1","Ok"]]

Named Parameters :

User Defined Session Parameters :

Predefined Session Parameters : Parameter name : histfile Value : [".cbq_history"]

\UNSET ParameterDeletes/resets the stack for the input parameter.

Pops the while stack for input value and then deletes it.

cbq> \UNSET -args;

CBQ – Parameter Configuration


Named and Positional Parameters • In order to define these, we use the \SET command.

• For each named parameter, use the prefix -$. \SET -$date "1-1-2014";

• For positional parameters we set the -args query parameter. \SET -args [9.5, "1-1-2014"];

Prepared Statements – How to • Set the named and positional parameters that are present in the prepared statement.

• PREPARE <N1QL Query>; (Prepared statement) OR • PREPARE <name> from <N1QL query>; (Named Prepare statement)

• EXECUTE <name>;

CBQ – Query Management – NAMED and POSITIONAL Query Parameters


#Create a prepared statement to see which advertising companies made a lot of money

prepare advAmount FROM SELECT cand_nm,sum(TONUMBER(disb_amt)) as total FROM `expenditures` WHERE cand_nm = $candidate AND recipient_nm = $1 AND disb_amt >= $2 GROUP BY cand_nm;

\SET -$candidate "Romney, Mitt"; # Set named parameter candidate for candidate name\SET -args ["AMERICAN RAMBLER PRODUCTIONS LLC" ,500]; #Set positional parameters

execute advAmount; # Execute prepared statement advAmount

\SET -$candidate "Obama, Barack" # Change candidate to Obama, Barack\SET -args ["GMMB INC.", 500];

execute advAmount; # Run prepared statement airinfo with new candidate name

CBQ – Query Management – NAMED and POSITIONAL Query Parameters


DEMO


Isha KandaswamySoftware [email protected]


Thank You!


The Couchbase Connect16 mobile appTake our in-app survey!


Share your opinion on Couchbase

1. Go here: http://gtnr.it/2eRxYWn

2. Create a profile

3. Provide feedback (~15 minutes)

http://gtnr.it/2eRxYWn



Date post:	15-Feb-2017
Category:	Software
Upload:	couchbase
View:	116 times
Download:	0 times