002 Ugm2013 Whats New Final

What‘s new…

Bernd Wiswedel

KNIME.com AG, Zurich, Switzerland

Two feature releases last year: 2.6 & 2.7

Documented in Changelog, “What‘s new

summary” and as video on YouTube

What‘s new page on knime.org

KNIMETV Youtube Channel

Outline

Illustrative examples

• Swiss Survival Analysis

• KNIME Forum Analysis

• (Next Best Offer)

New Features in 2.6 & 2.7

Outline






Swiss Survival Analysis

• Survival Analysis / Actuarial Tables

• Using population and deaths data to predict

longevity

• Creating the tables

• Investigating the tables

• Creating customer tables for:

• Overall

• Personal

• Historical

• Forecasting

• Make it easy to use for the non-expert!

Outline






KNIME Forum Analysis

Learn something about the KNIME forum:

http://tech.knime.org/forum





Challenges:

• Get data into KNIME

• Extract simple statistics (how many posts,

response time, response length)

• Classify topics and detect topic shifts

• Identify content and users




Challenges:






Forum Analysis – Get Data

Two alternatives:

• Connect to underlying database, read

content

Doable but complicated:

7+ tables need to be read,

prepared and joined

Forum Analysis – Get Data

Two alternatives:

• Connect to underlying database, read

content

complicated and not generic

• Crawl the web page, parse html

• Use XML parser & Palladian’s html retriever

nodes

Forum Analysis – Structure of forum

Several Categories, “KNIME General”,

“KNIME Reporting”, “Palladian”, …

(~20 in total)


Discussion threads on several sub-pages


Each thread consists of an initial post

and a variable number of comments

Forum Analysis – Crawler Flow







Discussion threads on several sub-pages




Input for all subsequent workflows!




Challenges:






Forum Analysis – Simple Statistics


Input table from crawler workflow


Meta nodes perform simple

preprocessing, e.g. average number

of active users per month


Many different reporting nodes with

different statistics. Reporting

extension to generate PDF, DOC, …


Number of active users per year


An active user is an user with at

least one comment or one post in

that year.

Number of posts per year


Numbers are just posts (new

discussion threads), not comments

Number of posts per month and year


Big increase early 2011.

Coincidentally, Simon Richards

(richards99) joined

Who comments/answers on posts?


Response time


Number of comments per post





Challenges:






Forum Analysis – Classify Posts

• Use text mining to classify forum post into

categories such as ‘io’, ‘manipulation’,

‘mining’, …

• No training set available

(mis-)use KNIME node description

• See evolution of discussion topics over the

years


Want to classify forum post (only

first post, no comments)…


… using KNIME node description text

as labeled training set


Reads node descriptions from xml

dumps (generated with KNIME

command line tool)

Uses forum data input file and

prepares with text mining tools


Unzips an archive with all xml files

into temp location


XML files read with loop and

preprocessed (header and footer

removed)


Description is converted into KNIME

text document, from which

(stemmed) terms are extracted



Training data extracted. Learning

attributes are keyword

occurrences; target is document

category


Training data extracted. Learning

attributes are keyword

occurrences; target is document

category

Verify model by splitting data

into train/test.

Using random forest classifier to

address high dimensionality of

small (and sparse) data set


… continuing with main input branch

(Input table from crawler workflow)


Preprocessing similar to before,

extracting date, author, title, …


Extracting attribute table using the

keywords from the node description

(training) data.


Remainder of the workflow ranks

the prediction and prepares for the

report.


Hot topics have always been

manipulation and mining … tasks

that KNIME is very good at.

Note also increase of ‘flowcontrol’

over the years and low ‘r’ traffic

(separate forum category, not part

of this data set)




Challenges:






Forum Analysis – Content & Users

• Look at individual categories (KNIME

General, Developer, Reporting, …)

• Learn what is discussed

• See who is contributing


Input are all discussions

in one forum category…


Output is a multi page

report with tag cloud and

user connection graph

Combines KNIME’s text and

network mining extensions



Input table from crawler workflow


Main loop over all ~20 categories


General statistics per category

User network analysis

Text analytics


Text analysis: Forum posts converted

to documents and tagged (persons,

node names, node categories)


Terms fed into tag cloud, colors

represent persons (‘kilian’), nodes

(‘bow creator’), node categories

(‘xml’), …


Network analysis:

User connections

(content ignored)


Network analysis: Ignore topics, only

look at user relation ships. Network

nodes represent users, connections

represent (directed) relationships

between users


Network analysis: Very simple

user graph, visualized with

standard KNIME graph viewer


Data collected and send

to reporting extension


Multi page pdf output for

different forum categories


Text Mining forum category


RDKit (community

chemistry extension)


KNIME Users – not

dominated by any

particular users




Challenges:






Reviewing all workflows

• All workflows rely on the same input data

• Requires re-run of “Crawler” workflow and

updating parameters in analysis flow

What do all these flows have in common?

They all require the “Crawler” data

Reviewing all workflows

• All workflows rely on the same input data

• Requires re-run of “Crawler” workflow and

updating parameters in analysis flow

• Better: Use meta node and share it between

all instances



Now use it in all the

analysis flows

Nice … but now all workflows

fetch the data each time they

execute!

Let’s add a cache option.

Quickform Node defining a switch:

-Get data from web or

-use cached file (lives on server)

Meta Node Templates

• Meta nodes as isolated functional unit

• Shared on KNIME Server (or teamspace) for

use in other workflows or by other users

• Quickforms to expose relevant parameters

in meta node dialog or in wizard execution

• Can also be used on the KNIME server…

Knime Web Portal

Knime Web Portal

Knime Web Portal

Knime Web Portal

Outline






NBO as a typical Project Collect training

data from

multiple sources:

- DB tables

- text files

- excel files

- SAS files

- binary tables

- map files

Define File Paths

and Parameters

Train and evaluate a number of

prediction algorithms to predict

variable Target

Retrieve old model

that has been decently working so

far

Compare

performances

and choose

best model

Recalculate

predictions

based on

best model

and save

Save

best

model

Read

current

data

NBO as an Example

Collect Training

Data from

multiple Sources

Select best

prediction model

Apply best

model to

score data

Select files and

define parameters

Build a

report

NBO Report

KNIME Server Training 109

Mean Error in %

Mean Error in %

e-mail

notification

[email protected]

Global

Flow

Variables

Quickform dialogs

Execution

Wizard File Upload

Quickforms

Value

Selection

Quickform

Integer Input

Quickform „Workflow

Stopped“ light

Status

“Workflow

Running” icon

“Workflow

Running” light

KNIME User Training 115

Errors and Warnings

Report

Export report as

Results of past

Executions

Outline






New Features in 2.6 & 2.7 - Highlights

• Enhanced database functionality

• File Handling node collection

• More flexible R integration

• Streaming API

• Better (Java) scripting support

• Hypothesis testing nodes

• UI Changes

• Database update and delete

Enhanced DB functionality

• New type support: Boolean and Blobs

Enhanced DB functionality

• Set of nodes to read, (un)zip, copy, move,

convert, … files

• Add notion of unique resource identifier

(URI) and mime types

Used in 3rd party extensions

• Nodes to up and

download files:

ssh, http, ftp, …

File Handling Nodes

• Collection of Nodes to extract statistical measures

• Different t-tests

• Anova

• (Crosstab)

Hypotheses Testing Nodes

• Before KNIME 2.7:

• With KNIME 2.7:

Flexible R integration

• Enhanced functionality:

• define multiple outputs at once

• Script templates

• Better editor

• Syntax highlighting

• Auto completion

Scripting – Java Snippet & friends

Enhanced programming interface in KNIME

enabling nodes to be streamed and

distributed.

Streaming API

KNIME Explorer replaces “Workflow Projects”

KNIME UI Changes

Customizable Node repository

(getting from 1500+ nodes to <100)

KNIME UI Changes

…

Tons more …

Summary

Discussed KNIME Usage Examples

check “Examples” Server for even more

New functionality constantly added, thanks to

community, partners and customers

And more is coming…

Date post:	12-Nov-2014
Category:	Documents
Upload:	nilmarnascimento7833
View:	24 times
Download:	3 times

002 Ugm2013 Whats New Final

Documents