d3 & Hierarchical Datasets

# Visualizing<br>Hierarchical<br>Data in D3<br>_Efficiently_
data is central<br>
comprehend its structure first<br>
the viz comes second

???

Who am I?
Why this topic? (Text Analytics)
When you understand your data, magic happens.

---

# Preface

### _Pick your projects_ — Information Arbitrage
### _Know your tools_ — Excellence = skillful application of great tools
### _Tell your story_ — Context, Challenge, _Solution_, Resolution

---

# Our Goal

.right-column-wide[
<iframe width="400" height="500" frameborder="0" src="viz/tree-json-example.html"></iframe>
]

Simplify<br>hierarchical data<br>visualizations<br>by perceiving:

* our data
* our viz options
* _d3.stratify()_

???

epiphany!
To this date, I've built with carefully crafted json structured data.
Now I build with tabular data and d3.stratify.
But I had to really understand the "catches" before it became easy.

Today, we'll talk about what I learned and what you need to build great viz's with hierarchical data without spending all of your time structuring your data.

---

.context-footer[Curran Kelleher @ bl.ocks.org/curran ([1356 lines of data](https://raw.githubusercontent.com/curran/data/gh-pages/un/placeHierarchy/countryHierarchy.json))]

???

[https://bl.ocks.org/curran/1dd7ab046a4ed32380b21e81a38447aa](Kelleher)

---

.context-footer[Kerry Rodden @ bl.ocks.org/kerryrodden ([9350 lines of data](https://gist.githubusercontent.com/kerryrodden/766f8f6d31f645c39f488a0befa1e3c8/raw/9fc86efac379f228749b2de3537acd629c0867c8/visit-sequences.csv))]

???

[https://bl.ocks.org/kerryrodden/766f8f6d31f645c39f488a0befa1e3c8](Rodden)

---
name: d3-examples

???

[https://bl.ocks.org/mbostock/raw/4063582/](Bostock)

---

???

[http://bl.ocks.org/aaizemberg/9e7c7c7762a58c1cd107af5f9235825b](Aizemberg)

---

name: d3-examples
.context[<iframe width="125" height="125" frameborder="0" src="viz/icon.html"></iframe>]
.context-footer[Other layout options: Cluster, Tree, Force.  Asthetics by Peter Cook @ d3indepth.com.]
# Hierarchies: Illustrated

.right-column[
<div class="rotate">Sunburst</div>
<iframe style="display: block;" width="400" height="225" frameborder="0" src="viz/sunburst-json-basic.html"></iframe><br>
<div class="rotate">RadialTree</div>
<iframe style="display: block;" width="400" height="225" frameborder="0" src="viz/radialtree-json-basic.html"></iframe>
]

.left-column[
<div class="rotate">Pack</div>
<iframe width="400" height="225" frameborder="0" src="viz/pack-json-basic.html"></iframe><br>
<div class="rotate">Treemap</div>
<iframe width="400" height="225" frameborder="0" src="viz/treemap-json-basic.html"></iframe>
]

???

Circle pack. Sunburst. Treemap. Radial tree.

Radial tree is a little pathetic...

---

_plural_ __hierarchies__

<mark> DATA </mark><br>
a pyramid-like organization of ideas, people, items...
* with a _single_, top-level item called the __root__.
* every item has _one higher_ and _zero or more lower_ neighbors.
* higher levels have greater inclusion, influence, or breadth.

---

# D3-Friendly JSON Hierarchy

``` json
{
  "name": "North America",
  "children": [
    { "name": "Canada" },
    { "name": "Mexico" },
    { "name": "United States",
      "children": [
        { "name": "California" },
        { "name": "Indiana",
          "children": [
            { "name": "Fishers" },
            { "name": "Carmel" },
            { "name": "Indianapolis" }
          ]
        }
      ]
    }
  ]
}
```

]

.right-column.small.gray[
<p> </p>
__JSON Objects__<br>
`{ "name":"John", "age":30, "car":null }`
* surrounded by curly braces {}
* written in key/value pairs
* keys must be strings; valid value types<sup>1</sup>
* keys and values are separated by a colon
* each key/value pair is separated by a comma

__JSON Arrays__<br>
`[ "Ford", "BMW", "Fiat" ]`
* almost the same as arrays in JavaScript
* valid value types<sup>1</sup>

]

---

# Hierarchy Examples

_Goal_: Think of your own dataset.

__Data Examples__
* _Company_: CEO ⇢ EVP ⇢ SVP ⇢ VP ⇢ DIR ⇢ MGR
* _Filesystem_: Folder ⇢ Subfolder ⇢ File
* _Book_: Dictionary ⇢ Letter ⇢ Word ⇢ Part-of-speech ⇢ Definitions ⇢ Examples
* _Family Tree_: Grandparents ⇢ Parents ⇢ Kids (this one kinda' falls apart...)

__Decision Making__
* _MECE_: "mutually exclusive, collectively exhaustive" in consulting
* _Taxonomy_: from science to risk management, we like orderly structures
* _Decision Tree_: machine learning technique in data science

???

* Start thinking of your idea.
* Not everything that seems hierarchical is: family tree
* MECE is actually a good way to divide up complex problems and a litmus test for true hierarchy.

---

## Let's Make a Viz

---

# Sticky Note Dataset (Hierarchical)

_Goal_: When we're done, you'll have ~6 items divided into ~3 categories all written on sticky notes.
.right-column[<img width="400" src="images/engineering.jpg">]

1. __Pick a <mark>topic</mark>__ (cars, coding, ice cream, stores, shoes, vacations, bucket list, learning agenda, your projects). Don't overthink it.
2. __Divide your topic__ into 3-ish <mark>categories</mark> (love, hate, envy; done, probably will do, too chicken; I came, I saw, I conquered; past, present future).
3. __Add <mark>items</mark> to each category__
4. __Record on sticky notes__:

* <mark class='mark-pink'>Pink</mark> = _Topic_ (the _Root_ of your hierarchy) [n=1]
    * <mark class='mark-blue'>Blue</mark> = _Categories_ (how you divided things up) [n=~3]
    * <mark class='mark-yellow'>Yellow</mark> = _Items_ (the details) [n=~6]

---

# My Sticky Notes & JSON

.right-column[
``` json
{
  "id": "cars",
  "children": [{
    "id": "owned",
    "children": [
      {"id": "pilot", "size": 40},
      {"id": "325ci", "size": 20},
      {"id": "accord", "size": 2}]
  }, {
    "id": "traded",
    "children": [
      {"id": "chevette", "size": 1}]
  }, {
    "id": "learned",
    "children": [
      {"id": "odyssey", "size": 20},
      {"id": "maxima", "size": 5}]
  }]
}
```

]

---

.context[<iframe width="175" height="125" frameborder="0" src="viz/tree-icon.html"></iframe>]
.context-footer[[bl.ocks.org/denjn5](https://bl.ocks.org/denjn5)]

# Tree in 62 Lines (Part 1)

``` html
<svg>
    <g></g>
</svg>

// Prepare our physical space
    var g = d3.select('svg').attr('width', vWidth).attr('height', vHeight)
        .select('g').attr('transform', 'translate(20,20)');

// Get the data from our CSV file
    d3.json('data.json', function(error, vData) {
        if (error) throw error;

drawViz(vData);
    });
</script>
```

---

.context[<iframe width="175" height="125" frameborder="0" src="viz/tree-icon.html"></iframe>]
.context-footer[[bl.ocks.org/denjn5](https://bl.ocks.org/denjn5)]

# Tree in 62 Lines (Part 2)

``` html
<script>
    function drawViz(vData) {
        // Declare d3 layout
        var vLayout = d3.tree().size([vHeight * 0.9, vWidth * 0.8]);

// Layout + Data
        var vRoot = d3.hierarchy(vData);
        var vNodes = vRoot.descendants();
        var vLinks = vLayout(vRoot).links();

// Draw on screen
        g.selectAll('path').data(vLinks).enter().append('path')
            .attr('d', d3.linkHorizontal()
            .x(function(d) { return d.y; })
            .y(function(d) { return d.x; }));

g.selectAll('circle').data(vNodes).enter().append('circle')
            .style('r', 10)
            .attr('transform', function (d) { return 'translate(' + d.y + ',' + d.x + ')'; });
    }
</script>
```

---

## Our Data Doesn't Look Like That...

---
.context[`d3.stratify()(myData);`]

# An Easier Way!?

_Let's re-think our sticky note exercise._ This time we'll think of our dat as a set of links in tabular form (e.g., .csv) with very light syntax requirements (vs. our verbose json-hierarchy model).

__Why?__
* a tabular style often matches our source data. (e.g., employee table's are often recursive)
* new records? easy
* no JSON-syntax typos

__Why Not?__
* large datasets are still complex
* errors can be difficult to hunt down
* the "imaginary" link from null to root

---

# Sticky Note Dataset (Tabular / Links)

__Goal__: Our original dataset (~6 items divided into ~3 categories all written on sticky notes), but laid out as source-target pairs (~10 lines of data).

__Create new sticky notes...__
* 1 __topic__ note for each _category_ + 1 (e.g., 4 sticky's that say "cars").
* 1 __category__ note for each of its child _items_ + 1.
* Column Header stickies: "ID", "ParentID".

__Think links!__ Start with the _root_ / _topic_ node. Then branch out to _categories_. Finally do _items_. Each stage of our tree is a simple pair of nodes. Lay out...
1. Column Headers (Put ID first to ensure that we only create 1 row per ID--since each ID must be unique.)
2. <mark class='mark-pink'>Topic</mark> row: 1 row for the _topic_ (it's the ID; the parentID is _blank_). This marks the root.
3. <mark class='mark-blue'>Category</mark> rows: 1 row per _category_ (it's the ID; the parentID is the _topic_).
4. <mark class='mark-yellow'>Item</mark> rows: 1 row per _item_ (it's the ID; the parentID is the _category_).

---

# d3.stratify() Hints
1. 1st row -> column headers (keep column headers & data in the same order)
2. 2nd row -> [Root],[null] (Our root; don't forget the comma)
3. Each ID must be _unique_
4. We can have only 1 Root
5. No blank lines
6. No spaces after commas

``` txt
id,parentId
cars,
owned,cars
pilot,owned
325ci,owned
accord,owned
traded,cars
chevette,traded
learned,cars
odyssey,learned
maxima,learned
```
]

.right-column[<iframe width="350" height="600" frameborder="0" src="viz/tree-sticky-notes-animate.html"></iframe>
]

---

# Updated Tree

.left-column-wide[
__OLD: From JSON__
``` javascript
d3.json('data-car.json',
        function(error, vData) {
    if (error) throw error;

drawTree(vData);
});
```

__NEW: From CSV; then stratify__
``` javascript
d3.csv('data-car.csv',
        function(error, vCsvData) {
    if (error) throw error;

vData = d3.stratify()(vCsvData);
    drawTree(vData);
});
```
]

???

Changes:
1. `d3.json('data-car.json',` becomes `d3.csv('data-car.csv',`
2. We changed the initial data variable name to `vCsvData`
3. Add this line: `vData = d3.stratify()(vCsvData);`

---

# Columns have Different Names?

__New Data Set__
``` txt
emp,mgr
simpsons,
marge,simpsons
homer,simpsons
bart,marge
...
```

]

.right-column-wide[
__Good Column Names (Prev Code)__
``` javascript
d3.csv('data-car.csv',
    function(error, vCsvData) {
  if (error) throw error;

vData = d3.stratify()
    (vCsvData);
  drawTree(vData);
});
```

__Other Column Names (Updated Code)__

``` javascript
d3.csv('data-car.csv',
    function(error, vCsvData) {
  if (error) throw error;

vData = d3.stratify()
    .id(function(d) { return d.emp; })
    .parentId(function(d) { return d.mgr; })
    (vCsvData);
  drawTree(vData);
});
```
]

---

# Want to Add Nodes?

_Open up your CSV and create new links._
* Each row -> unique `ID`
* `parentID` -> an existing `ID` from another row

???

Options
* Update data-car.csv (trailers are heavy; left turns are dangerous)
* Repoint tree-csv-example.html to data-car-plus.csv
[break it; then fix it]

---

# Want to Add Metadata?

.right-column-wide[
<iframe width="600" height="300" frameborder="0" src="viz/tree-csv-example-color.html"></iframe><br>

]

Add a new column header, and the appropriate data for each row.

``` txt
id,parentId,idColor
cars,,pink
owned,cars,lightblue
traded,cars,lightblue
learned,cars,lightblue
pilot,owned,yellow
325ci,owned,yellow
accord,owned,yellow
chevette,traded,yellow
odyssey,learned,yellow
maxima,learned,yellow
trailers are heavy,odyssey
left turns can be dangerous,maxima
```
<br>

``` javascript
g.selectAll('circle').data(vNodes).enter().append('circle')
    .style('fill', function(d) { return d.data.data.idColor; })
    .attr('transform', function (d) {
        return 'translate(' + d.y + ',' + d.x + ')'; });
```

---

# Keep in Mind

__For viz's that need size,__ add that as a column in your data, but refer to it as `d.data.size` now.
<br> <br>

__d3.stratify() works for tabular data from the range of d3 data helpers__
``` json
var vJSONData = [
  {"emp": "simpsons", "mgr": null},
  {"emp": "marge", "mgr": "simpsons" },
  {"emp": "homer", "mgr": "simpsons" },
  {"emp": "bart", "mgr": "marge" }
];
```

---

## Cool Extras

---

# Hierarchical Data -> Force Diagram
.left-column[
_Want to create a force diagram, but your data is set up hierarchically?_

`node.links()` returns an array of links for from your hierarchical dataset, where each link is an object that defines source and target properties. The source of each link is the parent node, and the target is a child node.

]

.right-column[
__Hierarchical Data & Layout__
``` javascript
var vRoot = d3.hierarchy(vData);
var vNodes = vRoot.descendants();
var vLinks = vLayout(vRoot).links();
```

<br>
__Hierarchical Data; Force Layout__
``` javascript
var vRoot = d3.hierarchy(vData);
var vNodes = vRoot.descendants();
var vLinks = vRoot.links();
```
]
.context-footer[Docs: [github.com/d3/d3-hierarchy#node_links](https://github.com/d3/d3-hierarchy#node_links)]

---

# d3.linkHorizontal() - Draw Links Simply

__Manually Draw a Link with SVG Path__: Requires setting your `<path>` 'd' attribute to a complex concatenation of source, target, and curves:
``` javascript
g.selectAll('path').data(vLinks).enter().append('path')
    .attr('d', function(d) {
        return 'M' + d.y + ',' + d.x
            + 'C' + (d.parent.y + 100) + ',' + d.x
            + ' ' + (d.parent.y + 100) + ',' + d.parent.x
            + ' ' + d.parent.y + ',' + d.parent.x;
    });

```

__D3's Shape `.linkHorizontal()`__ returns a link generator. To visualize links in a horizontal tree diagram rooted on the left edge of the display:
``` javascript
g.selectAll('path').data(vLinks).enter().append('path')
    .attr('d', d3.linkHorizontal()
    .x(function(d) { return d.y; })
    .y(function(d) { return d.x; }));
```

.context-footer[Docs: [github.com/d3/d3-shape/#links](https://github.com/d3/d3-shape/#links) (linkHorizontal, linkVertical, linkRadial)]

---

# Helpful Documents

_API Documents & Blocks_
* [github.com/d3/d3-hierarchy](https://github.com/d3/d3-hierarchy) - D3 API, including d3.stratify()
* [github.com/d3/d3-shape](https://github.com/d3/d3-shape#links) - D3 API, especially link generators
* [blockbuilder.org/search](http://blockbuilder.org/search) - Search the Blocks

_Hierarchical Tutorials_
* [d3indepth.com/layouts/](http://d3indepth.com/layouts/) - Peter Cook's D3 in Depth Layouts
* [denjn5.github.io/unconf/hierarchy.html](https://denjn5.github.io/unconf/hierarchy.html) - This presentation

_David Richards_
* [denjn5.github.io/](https://denjn5.github.io) - Sunbursts, Line-by-line
* [bl.ocks.org/denjn5](https://bl.ocks.org/denjn5) - My Blocks, more sunbursts...
* [github.com/denjn5](https://github.com/denjn5) - GitHub

* [linkedin.com/in/david-richards-do-good/](https://www.linkedin.com/in/david-richards-do-good/) - LinkedIn
* [twitter.com/learn_do_repeat](https://twitter.com/learn_do_repeat) - Twitter

---

# Thanks!