What is data analysis? and what is temporal data analysis.

There is a great amount of variation in defining what is a data analysis. According to Wikipedia, data analysis is a process: inspecting, cleansing, transforming, and modelling. And out of this process, we'll learn some knowledge and insights from the data, and make some decisions.

How about temporal data analysis? We still go through the same process, but the data dealt with is time-indexed data. The decisions we're going to make are somehow related to time.

But in order to conduct this process and analyse the data, we need tools. There are many tools out there. We have to carefully choose the good ones to lubricate this process as smoothly as possible. This set of tools, we call them "tidy tools".

Before introducing you tidy tools, I'd like to show you an example of what messy tools are.

This is a slide that I took from last year's Applied forecasting unit taught by Rob. I'm going say this is the messy tools' example. (Sorry, Rob)

It's about fitting three regressions on one data set but with different time trends as regressors. The first two code blocks are straightforward: the first one is linear trend and then forecast 10 steps ahead; the second is exponential trend by taking a log transformation on the response variable but using lambda = 0 and then forecast.

But the third one looks a bit horrified, many new functions have been introduced. The actual idea is simple, fitting a piecewise linear trend. Clearly this tslm() doesn't incorporate the piecewise case. So both the lecturer and the students get distracted from the modelling idea but have to understand how to implement the algorithm first. More importantly, this piece of code is very ad hoc. A few tweaks needed for a different data set, and there's possibility to invite mistakes while tweaking.

This makes me wonder if students should become programmers first before they can become forecasters.

Definitely not!

They do have to know how to program to do data analysis. But they don't have to be programmers. Particularly it's not the purpose of that unit.

Now I hope you get a sense of what are messy tools. In terms of sensation, messy tools are smelly.

Contrasting to messy tools, they are tidy tools.

This year, Rob is going to teach the same materials but use a different set of tools. You can see the code is clean and succinct.

Most importantly, it helps to communicate the modelling idea in a very clear way. We are going to model this data with three linear regressions. It also addresses the difference between piecewise and a linear trend is by adding knots. Once we're done with model fitting, we forecast them as a whole.

What has been abstracted away is computational implementations and put more emphasis on ideas and expressions. It makes teaching easier and more fun. The door for introducing mistakes is closed. This set of tools are "tidy tools". (Good job, Rob)

This is my conceptualisation about "tidy tools":

1. Tidy tools are accessible. It means anyone can run this bit of code, anywhere anytime, to produce identical results. There shouldn't be a paywall to prevent users from using it. The source code can be inspected by others as well.
2. Tidy tools are expressive. Tidy tools are code, and code is text. We can express our ideas in text and we can frame our data analysis problems through code. Code is documentation, documenting how we approach the problem. We can easily share plain text across different platforms and groups of people. This goes beyond what point and click software can do. Yes, it is R code. But I believe someone who never used R before, they should be able to quickly get what's happening here. This is the power of expressiveness.
3. Tidy tools are pipeable. Because the input format is consistent with the output format. We can chain these operations together to build a fluent pipeline and read more naturally. Modelling and then forecasting. It makes possible to have fluent workflow.

Data analysis is a process, particularly it's an iterative process.

The tidyverse is a collection of R packages to help with this process quick and tidy.

Why it's important? Before deploying the final model into the production stage, we need to repeat this process many many times. If this model doesn't work, we make some tweaks, and then fit another model. We want this iteration as rapid as possible.

In addition to tidy tools, what's underlying this fluent workflow is tidy data.

The "tidy" stage doesn't mean data cleaning, but refers to making the data in a right shape, a uniform shape that is shared by these three stages in this circle.

With the tidyverse, we have such a fluent pipeline to handle data analysis. But when you have time series data and want to do time series analysis ...

This is what you experience. Lots of you in this department primarily focus on the statistical models. But I have been working on the rest of the process to support modelling.

Small data lives in csv files, big data stays in database. But most time, the form of the data at the "import" stage deviates a lot from the current time series objects that are designed for modelling. When doing transformation and visualisation, different data shapes are expected. We do not have a uniform shape for lubricating this process for time series. There's no formal guidance how to reshape the data for modelling purpose.

Screaming out "WAT" is the natural response to this workflow. Let me share my "WAT" moment.

When I was doing my honours project years agos, I needed to compute a bunch of time series features on yahoo web traffic data.

The script shows I put multiple csvs into a big time series matrix. But it took me a while to understand what past me was doing.

When I revisit this code, I started to realise there's something fundamentally wrong here. Multiple observational units and multiple measurements have been put into one object, which isn't appropriate for the traditional time series structure.

So my work is going to smooth out those rough edges, provides a conceptual framework for temporal data analysis, and fill the gap of tidy data in temporal context. We're going to call this new data structure---"tsibble".

You know I'm serious about this work, bc it has a logo.

Next I'm going to demonstrate this whole data process with tsibble and showcase the value of it.

Data analysis cannot live w/o data.

I'm going to share with you two data stories simultaneously.

The LHS is daily users visiting my personal website . I own this domain and the website since 2012. I have a long data history. The RHS is on-time performance for domestic airlines in the States. One personal, one national. One small data, and one big data.

How the data looks like?

My data is quite simple, with 1000 obs and 2 variables. The Date column provides the time context. and how many visitors.

This data starts from the day when I started my PhD until yesterday. It has a sad beginning that nobody visited my website in the early days.

By contrast, airline data contains 5M obs and 22 variables. The sched_dep_datetime column gives the date-times. Other variables include the meta info describing the flights and on-time performance metrics. It is an event data set.

They already arrived in the tidy shape, with clearly defined observations and variables. Now we need to declare them as temporal data using tsibble.

Tsibble is a multi-faceted product.

First of all, tsibble represents tidy temporal data.

A tsibble has to be tidy data first. Data is organised in the common rectangular form: each row is an observation; each column holds a variable. But temporal data is more of a contextual data object. So on top of the tidy data, column variables require more meanings: index for time support, key for defining identities over time, and the rest are measurements.

Let me elaborate index and key more.

The index variable consists of time indices that provide a contextual basis for temporal data.

A variety of time representations exist in the wild, the common ones you saw before, dates and date-times, as well as intradays and intervals from one time point to another. Tsibble supports all of them, as long as they can be ordered from past to future.

Unlike other time series objects, tsibble respects time zones, will not ignore them and not convert them to UTC.

Temporal data can be regular and irregularly spaced in time. That's why we say tsibble is temporal data instead of time series.

And the key can consist of empty, one or more variables. It is typically known for analysts. Each observational unit in the key, only allows unique time indices.

These features make tsibble a modern re-imagining temporal data.

Can these two data sets meet the tsibble definition in the first instance?

Since the web traffic is a univariate series, just about me. So the key is empty. And the index is Date.

Now it's a tsibble object with 1 day interval.

How about the airline data?

From the passenger's point of view, we're interested in flight number at its scheduled departure time. Also it's irregular interval.

After declaring key and index. Oops! We got an error.

The error says: ... and suggests ...

Another way to interpret this error message is that this data may have data quality issue.

We need to search for those duplicates and fix them.

As you can see, we've got two problematic entries. The flight number NK630 has been scheduled twice at the same departure time. They have identical arrival time, departure delay, arrival delay, and carrier. But they are different in tail number, origin and destination. After researching this flight number and going through the whole database, my decision is to remove the highlighted entry: the flight from LAX to DEN, bc this flight doesn't operate on this route.

Tsibble is strict on tidy data.

Once we remove the duplicated observation, we are all good.

Irregular time space is marked by exclamation mark. and we've got 22 thousands of flights or series in the table.

When creating a tsibble object, we're also going through the tidy stage. Tsibble complains early when the data involves duplicates, like the flights data. We need to find the duplicates and fix them. We now end up with a properly-defined temporal data.

Tsibble permits time gaps. It is recommended to check time gaps before transformation and modelling. Bc you are likely to encounter errors if gaps in time. Luckily no gaps in these two data sets.

Tsibble provides formal organisation on how to tidy your temporal data.

Tsibble as a data object, it's also the first time series object that can directly talk to grammar of graphics, without coercion.

The grammar of graphics defines plots as a functional mapping from data points to graphical elements. The plot is initialised through mapping variables from data space to graphical space. Variable Date is mapped to x axis and Users mapped to y axis. But I haven't put the visual elements on the canvas yet. So it's just empty.

Instead of a line chart, I add an area geom on the canvas with specified colours.

This plot tells my phd life was quiet in the first year. Since May 2017, this website started to attract more or less traffic. It was the time I started to deliver workshops and present to a wider audience. The year of 2018 saw many spikes, because I talked, talked, and talked a lot. It was fun to look at how my phd performs over three years.

Tsibble is not only a data abstraction, but also a DSL for wrangling temporal data.

Like we can make a plot layer by layer with grammar of graphics, we can chain operation by operation using tsibble to make a complete analysis. Bc the grammar is declarative.

When I actually develop tsibble, I started to appreciate the tidyverse team. Lots of amazing design thinking behind it. A fluent and consistent API gives an intuitive pipeline but it takes lots of time to refine the idea. For example, naminng a thing is difficult.

In order to master a language, we need to learn vocabularies first.

The tsibble dictionary divides into 4 major blocks.

1. Vector functions deals with 1 dimentional vector, mostly time vector
2. Unlike vector functions, table verbs take a data table and returns a data table
3. It is necessary to have commonly-used rolling window operations
4. Other functions to make tsibble extensible to support new index type and data class.

These functions are written and organised independently to achieve greater flexibility. so users can interweave any of these functions to solve a particular problem. This slide gives you a fraction of vocabularies.

To check out the full list, here it is.

If you're familiar with the tidyverse, the learning curve for tsibble is very gentle. Bc tsibble leverages the tidyverse verbs but in a slightly different way.

I'm constrasting how the same verb works differently for a normal data table and a tsibble.

For example, selecting columns from a normal data table, you get a table with that columns.

However, selecting tsibble's columns, it automatically selects the index variable. it always keeps the time context around and will not drop it. Without index and key, a tsibble will no longer be a tsibble.

Summarising a table, it reduces to a single value. But summarising a tsibble, it computes the values over time. For this case, it returns itself.

We use the tidyverse verbs to avoid introducing new verbs for less cognitive burden on users. But we give those verbs more contextual meaning so they behave more intuitively in the temporal context.

We gradually build up our vocabularies in order to write a sentence and futher compose prose.

For example, this sentence begins with the data. And an action performed on this data is an adverb plus a verb. This describes this action is aggregating daily data to annual data.

Yearly data makes my story look promising. Visitors doubled every year. The first quarter of this year is almost as the sumup of 2017.

This is a brief sentence with the web traffic.

A paragraph for airline traffic is made up of two adv and two verbs. Selecting three airports: JFK, Hawaii, and LAX from hundreds of them. and then get a summary of daily percentage delay for each airport.

The airport in Hawaii, is low in delay with better on-time performance all year around.

Once you internalise this set of grammar, all sorts of transformation come handy.

The third aspect of tsibble is adopting FP for rolling windows.

When doing rolling window analysis, we cannot avoid for loops. But tsibble's rolling family keeps users away from for loops. and help concentrate on what you're going to achieve instead of how you're going to achieve.

A toy example: computing moving averages for window size 2 and median for 3

In order to compute the mean and median, I have to do a bit of setup for the loop. But the only part we're interested is the action: mean and median.

The length of output is not stable, depending on the the choice window size. This is for loop

What I mean by FP? It abstracts away the for-loop, and focus on what you want to do by supplying a function. And it's length-stable and type-stable.

3 types of rolling window

1. sliding: moves a fixed length window through the data with overlapping
2. tiling: moves a fixed length window through the data w/o overlapping
3. stretching: fix an initial window and expand to include more observations.

The animation shows three different rolling averages.

it takes an arbitrary function: simple complex

time series CV: easier to compute forecasting accuracy by comparing rolling forecast with rolling data

...

The only limit for how to use this family for window-related analysis is your imagination.

Last year, I said tsibble was going to be the future of time series. This year, I would say the future is around the corner. A few dependent packages have been developed around the tsibble structure, including the new fable package for forecasting to replace the forecast package. Rob has already started teaching this set of cutting-edge tools in his class this semester. The next generation of forecasters will be more productive.

What's my contribution is:

Tsibble provides a uniform and standardized temporal data structure for downstream analytic tasks. You can easily transform, visualise and model your temporal data, and smoothly shifts between them for rapid iteration to gain data insights.

Tsibble also provides a descriptive grammar that you can express your temporal data analysis problem easily.

Lemme show you a final example to conclude this part by chaining these 3 parts together.

I'm interested in my website traffic performance in the rest of the year.

3 paragraphs here:

1. summarising daily to monthly data, shorten this long series
2. fit an ETS and forecast 9 months ahead
3. bind historical data and forecast together and aggregate to yearly data.

Xkcd man tells I've got 3000 users to achieve, so pressure is on.

I hope this pipeline looks intuitive to me.

Tsibble forms the second chapter, and it is submitted to JCGS.

One step backward is the first chapter on calendar-based graphics. It is tentatively accepted by JCGS, asking for revision and resubmission.

The calendar plot is inspired by the actual calendar. You mark your daily schedule on this calendar, and we draw glyphs instead. Interestingly, the function that I wrote for the calendar plot also makes this physical copy of the calendar for our numbats group. So it returns to the calendar.

The hard work is to layout the panels in the calendar format.

1. is to determine what's the weekday that the first day of the month starts, and use modulus arithimetis to figure out the rest. The calendar conforms to 5 rows by 7 columns. The days that exceed the 5th week are wrapped to the first top rows.

Once the layout is setup, the plot drops in.

User cases by others

1. a runner

One step forward is exploring missing patterns for time series. Many works have been done on missing data, but few for time series. With the support from tsibble, we can better handle temporal missing values from the data-centric point of view.

We have grouped types of missings into 4 groups.

It's generally difficult to measure the research impact. But I've got 2 R pkgs associated with each chapter. Using the pkg metrics could be good proxy.

Finally I'd like to share with you some of my thoughts on organising a thesis. I learned this term RC at this year's rstudio conference.

1. is a container contains the document and its computations (i.e. text, code, data,...) making code and data available is equally important as text and figures.
2. provides a way for distributing, managing and updating the collection.

In this peerj paper, Ben Marwick discusses RC provides a standard and easily recognisable way for organising the digital materials of a research project for better transparency and reproducibility.

I think it's a brilliant idea. I've been practicing it for the thesis.

It turns out the R pkg structure is well suited for organising research project, if you're using R for analysis.