Creativity¶

In addition to correctness we also desire a model to have creativity.

Consider a model that generates a new population by sampling randomly from the training population. This model would have excellent or even perfect correctness. However this model would be unable to generate up-sampled populations of sequences without sampling the same data multiple times. More generally this model would be unable to generate sequences not seen in the training data, which in practice is only a small sample of the true population.

We therefore also measure and value creativity, which we define as a combination of diversity and novelty.

In [1]:

import random

import pandas as pd

from caveat.evaluate.features import creativity

import random import pandas as pd from caveat.evaluate.features import creativity

In [2]:

# create some data
raw = pd.read_csv("data/synthetic_schedules.csv")


def down_sample(df, p):
    n_samples = int(len(df.pid.unique()) * p)
    sample_ids = random.sample(list(df.pid.unique()), n_samples)
    sampled = df[df.pid.isin(sample_ids)]
    return sampled


observed = down_sample(raw, 0.2)

a = down_sample(observed, 0.5)
b = down_sample(raw, 0.2)
synthetic = {"a": a, "b": b}

# create some data raw = pd.read_csv("data/synthetic_schedules.csv") def down_sample(df, p): n_samples = int(len(df.pid.unique()) * p) sample_ids = random.sample(list(df.pid.unique()), n_samples) sampled = df[df.pid.isin(sample_ids)] return sampled observed = down_sample(raw, 0.2) a = down_sample(observed, 0.5) b = down_sample(raw, 0.2) synthetic = {"a": a, "b": b}

In [3]:

observed_hash = creativity.hash_population(observed)
a_hash = creativity.hash_population(a)
b_hash = creativity.hash_population(b)

observed_hash = creativity.hash_population(observed) a_hash = creativity.hash_population(a) b_hash = creativity.hash_population(b)

Diversity¶

We consider the diversity of the populations of sequences by counting the number of unique sequences as a proportion of the total number of sequences. We can compare the diversity of two populations in this manner. We consider higher diversity to be good. Ideally as diverse or more so than the training population.

In [4]:

print(
    f"Observed population of size {len(observed)} has diversity of {creativity.diversity(observed, observed_hash)}"
)
print(
    f"Synthetic population A of size {len(a)} has diversity of {creativity.diversity(a, a_hash)}"
)
print(
    f"Synthetic population B of size {len(b)} has diversity of {creativity.diversity(b, b_hash)}"
)

print( f"Observed population of size {len(observed)} has diversity of {creativity.diversity(observed, observed_hash)}" ) print( f"Synthetic population A of size {len(a)} has diversity of {creativity.diversity(a, a_hash)}" ) print( f"Synthetic population B of size {len(b)} has diversity of {creativity.diversity(b, b_hash)}" )

Observed population of size 838 has diversity of 0.735
Synthetic population A of size 419 has diversity of 0.78
Synthetic population B of size 865 has diversity of 0.695

In all cases we have high diversity. Note that that the size of a population also has an impact on diversity. we expect it to be easier to generate a diverse population if it is smaller.

Novelty¶

We consider the novelty of a model as how well it can generate sequences not observed in the training population. We measure novelty by counting the number of unique sequences not seen in the training population as a proportion of the total number of sequences in the population. By only considering unique sequences we are are effectively combining our measure of diversity with novelty, and we more generally refer to this metric as creativity.

In [5]:

print(
    f"Synthetic population A of size {len(a)} has novelty of {creativity.novelty(observed_hash, a_hash)}"
)
print(
    f"Synthetic population B of size {len(b)} has novelty of {creativity.novelty(observed_hash, b_hash)}"
)

print( f"Synthetic population A of size {len(a)} has novelty of {creativity.novelty(observed_hash, a_hash)}" ) print( f"Synthetic population B of size {len(b)} has novelty of {creativity.novelty(observed_hash, b_hash)}" )

Synthetic population A of size 419 has novelty of 0.0
Synthetic population B of size 865 has novelty of 0.5611510791366906

In our example, population A is sampled from the observed population and therefore has 0 novelty. Population B is sampled from the same larger population (intended to represent the "real" population) as the observed so has some novelty.