Date post: | 03-Mar-2017 |
Category: |
Data & Analytics |
Upload: | akin-kazakci |
View: | 36 times |
Download: | 1 times |
Akın Kazakçı
MINES ParisTech PSL Research University, CGS-I3 UMR 9217
Mehdi Cherti, Balázs Kégl
CNRS, Université Paris-SaclayCentre for Data Science
{mehdi.cherti, balazskegl}@gmail.com
*Runco, M. A., and Jaeger, G. J. 2012. The standard definition of creativity. Creativity Research Journal 24(1):92–96.
Creativity is a process by which novel and valuable ideas are
produced*
Relationship between value and novelty is critical
Empirical evidence suggests that, for genuinely novel objects, it’s hard to determine the value.
Especially, if the object is not there yet.
So, how to guide the generative process?
Relationship between value and novelty is critical - yet understudied
For instance, no design theory mentions explicitly “value” as a chief construct of its ontology (FBS, PSI, GDT, CK, CDP…)
The field of economics which deals explicitly with “value” does not consider generative processes.
In computational or psychometric studies of creativity either the value or the objects are known beforehand.
The “value of novelty” is a hard problem
- and it’s the blindspot of creativity research.*
* Kazakci, A. "Conceptive artificial intelligence: Insights from design theory.” International Design Conference, Dubrovnik 2014
Kazakci, A. Cherti, M, Kégl, B., “Digits that are not: Generating new types through deep generative nets” ICCC, Paris 2016
The Unknown Value Problem (UVP)
Assume you have a set of (existing) objects. Assume you know their value.
0
1
V(x)
x ∈ Sx1 x3x2 x4
The Unknown Value Problem (UVP)
0
1
V(x)
x ∈ Sx1 x3x2 x4
y* = V(x*)
Let S° = {x1, x2, x3, x4} ∈ S. How can we build an x* such that:Problem 1: y* = V(x*) is different than V(xi), for xi ∈ S°?Problem 2: y* = V(x*) is bigger than V(xi), for xi ∈ S°?
Bounding-box and Out-of-the-box
0
1
V(x)
x ∈ℝn
S1 ∈ S
Let S1 be the “bounding-box” that can be generated from S° by some generative process G: S1 = G(S°).
If we can model V(x) explicitly (that is, we have a knowledge model about value), then, for any xi ∈ S1, we know that the value will be within [Vmin, Vmax] with some distribution p(V(S1)) of value.
p(V(S1))
Vmax
Vmin
The magnitude of novelty can be defined as the shift in the posterior distribution (after generation). Note that, as we keep generating points, this magnitude vanishes, since we’ll approximate uniform distribution.
Bounding-box and Out-of-the-box
0
1
V(x)
x ∈S
If we want to go out-of-the-box, we have two problems:- We have to be able to generate x*- We no longer have a value function that can guide the generation of x*
x*
The difficulty in generating valuable novelty
64 pixels
64 pixels
In a 64x64 pixel space, the probability of generating this (or any) image by random selections of 0 or 1 for each pixel is:
2-4096 ≈ 0
In this space, images with any structure (under any meaningful definition of structure) is a very small subset of all the possible images.
An example: GAs without fitness (value) function
Successive panels in the sequence show populations of images generated by cross-over and mutation operators from the previous population.
Without a value function guiding the exploration of the unknown, everything become noise in just a few iterations.
Value function barrier• In most cases, the value function is an exogenous factor and
its extension over the novelty is not considered.
• For example, in evolutionary computational creativity systems, the value function is fixed and predetermined
• These fitness functions reflects system designer’s preference for novelty - not the machine’s.
• We want to try & get rid of the hard-coded value functions; let the system develop its own (Kazakçı, 2016)
A program to go beyond the value barrier
Any theory of creativity or design that does not explain how an agent can build its own value function for novelty
is incomplete
A program to go beyond the value barrier
We adopt a computational approach to study mechanisms by which such value functions may be built.
- Our main tools are deep generative neural nets
- DGNNs learn a hierarchy of transformations, given a set of objects
- These transformations are robust with respect to novelty
A program to go beyond the value barrier
1. Given a set of objects, can we generate new objects that are not white noise? Digits that are not: Generating new types through deep generative nets, ICCC, Paris 2016
2. What is a good generative model for novelty generation? Out-of-class novelty generation: an experimental foundation. ICLR 2017 (submitted, see openreview.net)
3. How does an agent can develop its own value function?(work in progress)
We adopt a computational approach to study mechanisms by which such value functions may be built.
Lear
nFo
rce
Fixa
te
Can we generate new images that are significantly different than existing images - yet still not white noise?
Auto-associative neural netsa.k.a auto-encoders
Learning to disassemble
Learning to build
- Auto-encoders have existed for long time (Kramer 1991)
- Deep variants are more recent (Hinton, Salakhutdinov, 2006; Bengio 2009)
- A deep auto-encoder learns successive transformations that decompose and then recompose a set of training objects
- The depth allows learning a hierarchy of transformations
The experimental setup
- We trained sparse convolutional auto-encoder (3c1d) on 70000 images of handwritten digits (MNIST) of size 28x28
- Training objective is to minimize the reconstruction error
Generating new symbols
- We use an iterative method to build images the system has never seen:- Start with a random image x0 = r, - and force the network to construct (i.e. interpret)- xk = f(xk-1), until convergence
- Our method is inspired by Bengio et al. (2013)- By contrast to them, we do not constrain the net to generate only
known types (we do not consider unknown symbols as spurious)
A map of the unknown
Coloured clusters are original MNIST digits (classes from 0 to 9)
The gray dots are newly generated objects
New objects form new clusters
Using a clustering algorithm, we recover coherent sets of new symbols
A distance preserving projection of digits to a two-dimensional space
(van der Maaten and Hinton 2008)
Suppose we ask a thousand children, who only know how to write digits, to write something different.
How would you evaluate these children on their ability to come up with coherent, and even meaningful symbols?
Please replace “children” in the above two sentences with “generative models”.
What is a good generative model for novelty generation?
Idea: Simulate the unknown
Unknown can be simulated by held-out classes, just like future data is simulated by held-out data in traditional machine learning
Train on known objects and types, test on types known to the experimenter but unknown to the model
Train on digits, test on letters
~1000 DGNNs: Autoencoders (sparse, contractive, denoising) and GANs, with a wide variety of hyperparameter values
+ a referee model: a discriminator between letters and digits (36 classes)
What are the models that are better at generating novelty?
Two measures: - Out of class count: The number of times a model has generated an image - evaluated as a letter by the discriminator
- Objectness (Salimans, 2016): Posterior entropy to distinguish noise from structure
Combining the two objectives
Yields images that are not digits, not random noise and convincing a specialist discriminator that they are letters.
Reviewer: “For the "novel" samples in Fig. 3, they are clearly digits.”
Us: We kindly ask the reviewer to classify the symbols on the following panel into one of the digit categories 0-9. They are from Figure 3.
Thank you
Mehdi Cherti, Balázs Kégl {mehdi.cherti, balazskegl}@gmail.com
Akın Kazakçı [email protected]