Lecture 20

class: title-slide, left, bottom

# Lecture 20
----
## **DANL 100: Programming for Data Analytics**
### Byeong-Hak Choe
### November 10, 2022

---
class: inverse, center, middle

# Getting started with `numpy`
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=796px></html>

---
# Introduction to `numpy`
### Basic Indexing and Slicing

- NumPy array indexing is a deep topic, as there are many ways you may want to select a subset of your data or individual elements.

- One-dimensional arrays are simple; on the surface they act similarly to Python lists

.pull-left[

```python
arr = np.arange(10)
arr[5]
arr[5:8]
```

]
.pull-right[

```python
arr[5:8] = 12
arr

arr_slice = arr[5:8]
arr_slice[:] = 64
```

]

---
# Introduction to `numpy`
### Basic Indexing and Slicing

- In a two-dimensional array, the elements at each index are no longer scalars but rather one-dimensional arrays:

```python
arr2d = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr2d[2]
arr2d[0][2]
arr2d[0, 2]
```

---
# Introduction to `numpy`
### Basic Indexing and Slicing

---
class: inverse, center, middle

# Getting started with `pandas`

---
#  `pandas`

- `pandas` is a Python library including the following features:
  - Data manipulation and analysis,
  - DataFrame objects and Series,
  - Export and import data from files and web,
  - Handling of missing data.

- `pandas` provides high-performance data structures and data analysis tools.

```python
import pandas as pd
```

---
# Getting started with `pandas`
### Create `Series`

- `pd.Series()` creates one-dimensional array-like object including values
and an index.

```python
obj = pd.Series([4, 7, -5, 3])
obj
```

- Simple `Series` formed only from a list.
- An index is added automatically.

---
# Getting started with `pandas`
### Create `Series`

- NumPy `arrays` can only be indexed by integers, while `Series` can be indexed by the manually set index.

```python
obj2 = pd.Series([2, -5, 9, 4], index=["a", "b", "c", "d"])
npobj = np.array([2, -5, 9, 4])

obj2
obj2["b"]
npobj[1]
```

---
# Getting started with `pandas`
### Create `Series`

- `Series.values` returns the values of a `Series`.
- `Series.index` returns the index of a `Series`.

```python
obj.values
obj.index
obj2.index
```

- The values and the index of a Series can be printed separately.
- The default index, if none was explicitly specified, is a `RangeIndex`.

---
# Getting started with `pandas`
### Create `Series`

- pandas `Series` can be created from NumPy arrays.

.panelset[

.panel[.panel-name[example]

.pull-left[

```python
npobj = np.array([2, -5, 9, 4])
obj2 = pd.Series(npobj, 
                 index=["a", "b", 
                        "c", "d"])
obj2
```
]
.pull-right[

```python
obj2.index
obj2["a"]
obj2["d"] = 6
obj2[ ["c", "a", "d"] ]
```
- Here `["c", "a", "d"]` is interpreted as a list of indices.
]

]

.panel[.panel-name[Series Operations]

- Using NumPy functions or NumPy-like operations will preserve the index-value link.

- Another way to think about a `Series` is as a fixed-length, ordered dictionary, as it is a mapping of index values to data values.

.pull-left[

```python
obj2[obj2 > 0]
obj2 * 2
np.exp(obj2)
```

]

.pull-right[

```python
"b" in obj2
"e" in obj2
```
]

]

---
# Getting started with `pandas`
### Create `Series`

- pandas `Series` can be created from dictionaries as well.
  - The index of the resulting `Series` consists of the dict’s keys.
  - The index can be set manually when passing a dictionary to a `Series`

.pull-left[

```python
dictdata = {"Rochester": 210_606, 
            "Buffalo": 276_807,
            "Syracuse": 146_103}
obj3 = pd.Series(dictdata)
obj3
```
]
.pull-right[

```python
cities = ["Niagara", "Buffalo", 
          "Syracuse"]
obj4 = pd.Series(dictdata, 
                index=cities)
obj4
```
]

- `NaN` (not a number) marks missing values where the index and the dict do not match.

---
# Getting started with `pandas`
### `Series` properties
- `Series.name` returns name of the `Series`.
- `Series.index.name` returns name of the `Series`'s index.

```python
obj4.name = "population"
obj4.index.name = "cities"
obj4
```

- The attribute `name` will change the name of the existing `Series`.
- There is no default name of the `Series` or the index.

---
# Getting started with `pandas`
### `pd.Series` vs. `np.array`
- NumPy arrays are accessed by their integer positions.

- Series can be accessed by a user defined index, including letters and numbers.

- Different Series can be aligned efficiently by the index.

- Series can work with missing values, so operations do not automatically fail.
  - The `isna` and `notna` functions are used to detect missing data:
.pull-left[

```python
pd.isna(obj4)
pd.notna(obj4)
```
]
.pull-right[

```python
obj4.isna()
obj4.notna()
```
]

---
# Getting started with `pandas`
### `pd.DataFrame`

- `DataFrame` is the primary structure of pandas.

- `DataFrame` represents a table of data with an ordered collection of columns.
  
- Each column can have a different data type.

- `DataFrame` can be thought of as a dictionary of `Series` sharing the same index.

---
# Getting started with `pandas`
### What is *tidy* `DataFrame`?

- There are three rules which make a dataset tidy:

1. Each **variable** has its own *column*.
  2. Each **observation** has its own *row*.
  3. Each **value** has its own *cell*.

---
# Getting started with `pandas`
### Create `DataFrame`

.panelset[

.panel[.panel-name[DataFrame]

- `pd.DataFrame()` creates a `DataFrame` which is a two-dimensional tabular-like structure with labeled axis (rows and columns).

```python
data = {"state": ["Ohio", "Ohio", "Ohio", "Nevada", "Nevada", "Nevada"],
        "year": [2000, 2001, 2002, 2001, 2002, 2003],
        "population": [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
frame = pd.DataFrame(data)
```

- In this example the construction of the `DataFrame` is done by passing a dictionary of equal-length lists.

- It is also possible to pass a dictionary of NumPy arrays.

]

.panel[.panel-name[NaN]

- Passing a column that is not contained in the dict, it will be
marked with `NaN`:

```python
frame2 = pd.DataFrame(data, columns=["state", "year",
"population", "income"])
frame2
```

- The default index will be assigned automatically as with `Series`.

]
.panel[.panel-name[NaN]

- If we specify a sequence of columns, the DataFrame's columns will be arranged in that order:

```python
frame2 = pd.DataFrame(data, columns=["year", "state",
                                     "population"])
frame2
```

]

.panel[.panel-name[inputs]

- We can pass the following types of objects to `pd.DataFrame()`:
  
  - 2D NumPy arrays

- Dict of lists, tuples, dicts, arrays, or Series

- List of lists, tuples, dicts, or Series
  
  - Another DataFrame

]

---
# Getting started with `pandas`
### Indexing `DataFrame`

.panelset[

.panel[.panel-name[Adding a column]

- We can add a new column to `DataFrame` as follows:

```python
frame2["change"] = [1.2, -3.2, 0.4, -0.12, 2.4, 0.3]
frame2["change"]
```

- Selecting the column of `DataFrame`, a `Series` is returned,
- A attribute-like access, e.g., `frame2.change`, is also possible.
- The returned `Series` has the same index as the initial `DataFrame`.

]

.panel[.panel-name[Selecting columns]

- The result of using a list of multiple columns is a DataFrame:

```python
frame2[ ["state", "population"] ]
```

]

.panel[.panel-name[Naming properties]

- We can name what the index and the columns are representing by using `index.name` and `columns.name` respectively:

```python
frame2.index.name = "number:"
frame2.columns.name = "variable:"
frame2
```

- In `DataFrames`, there is no default name for the index or the columns.
]

.panel[.panel-name[Reindexing]

- `DataFrame.reindex()` creates new `DataFrame` with data conformed to a new index, while the initial `DataFrame` will not be changed:

```python
frame3 = frame.reindex([0, 2, 3, 4])
frame3
```

]

.panel[.panel-name[Missing values]

- Index values that are not already present will be filled with `NaN` by
default.

- The `pd.isna()` and `pd.notna()` functions detect missing data:

```python
companies3 = companies.reindex(index=[0, 2, 3, 4, 5], 
                               columns=["company", "price", "market cap"])
companies3
pd.isna(companies3)
pd.notna(companies3)
```

]

.panel[.panel-name[Dropping rows]

- Calling `drop` with a sequence of labels will drop values from the row labels (axis 0):

```python
obj = pd.Series(np.arange(5.), 
                index=["a", "b", "c", "d", "e"])
obj
new_obj = obj.drop("c")
new_obj
obj.drop(["d", "c"])
```

]

---
# Getting started with `pandas`
### Dropping columns

.panelset[

.panel[.panel-name[(1)]

- With `DataFrame`, index values can be deleted from either axis. To illustrate this, we first create an example `DataFrame`:

```python
data = pd.DataFrame(np.arange(16).reshape((4, 4)),
                    index=["Ohio", "Colorado", "Utah", "New York"],
                    columns=["one", "two", "three", "four"])

data
data.drop(index=["Colorado", "Ohio"])
```

]

.panel[.panel-name[(2)]

- To drop labels from the columns, we can use the `columns` keyword:

```python
data.drop(column=["two"])
```
]

.panel[.panel-name[(3)]
- We can also drop values from the columns by passing `axis=1` or `axis="columns"`:

```python
data.drop("two", axis=1)
data.drop(["two", "four"], axis="columns")
```
]

.panel[.panel-name[del]
- `del DataFrame[column]` deletes column from `DataFrame`.

```python
del data["two"]
data
```

]

---
# Getting started with `pandas`
### Indexing, selecting and filtering

.panelset[

.panel[.panel-name[example]
- Indexing of DataFrames works like indexing an `np.array`.
  - We can use the default index values:

```python
data = {"company": ["Daimler", "E.ON", "Siemens", "BASF", "BMW"],
"price": [69.2, 8.11, 110.92, 87.28, 87.81],
"volume": [4456290, 3667975, 3669487, 1778058, 1824582]}

companies = pd.DataFrame(data)
companies
companies[2:]
```

]

.panel[.panel-name[setting index]

- We can also use a manually set index.

```python
companies2 = pd.DataFrame(data, index=["a", "b", "c", "d", "e"])
companies2
companies2["b":"d"]
```

- When slicing with labels, the *end element is inclusive*.
]

.panel[.panel-name[.loc() and .iloc()]

- `DataFrame.loc()` selects a subset of rows and columns from a DataFrame using **axis labels**.

- `DataFrame.iloc()` selects a subset of rows and columns from a
DataFrame using **integers**.

```python
companies2.loc[ "c", ["company", "price"] ]
companies2.iloc[ 2, [0, 1] ]

companies2.loc[ ["c", "d", "e"], ["volume", "price", "company"] ]
companies2.iloc[ 2:, : :-1 ]
```

]

.panel[.panel-name[summary]

- `df[val]` selects single column or set of columns;
  
  
  - `df.loc[val]` selects single row or set of rows;
  - `df.loc[:, val]` selects single column or set of columns;
  - `df.loc[val1, val2]` selects row and column by label;
  
  
  - `df.iloc[where]` selects row or set of rows by integer position;
  - `df.iloc[:, where]` selects column or set of columns by integer position;
  - `df.iloc[w1, w2]` Select row and column by integer position.

]

---
# Getting started with `pandas`
### Operations between `DataFrame` and `Series`

.panelset[

.panel[.panel-name[example]

- Here the `series` is generated from the first row of the `DataFrame`:

```python
companies3 = companies[["price", "volume"]]
companies3.index = ["Daimler", "E.ON", "Siemens", "BASF", "BMW"]
series = companies3.iloc[2]

companies3
series
```

]

.panel[.panel-name[+]

- By default, arithmetic operations between `DataFrames` and `Series` match the index of the `Series` on the `DataFrame`'s columns:

```python
companies3 + series
```

]

.panel[.panel-name[.add()]

- `DataFrame.add()` does addition along a column matching the `DataFrame`'s row index (axis=0).

```python
series2 = companies3["price"]
companies3.add(series2, axis=0)
```

]

.panel[.panel-name[Transpose]

- `DataFrame.T` transposes DataFrame.

```python
companies3.T
```

]

---
# Getting started with `pandas`
### NumPy functions on `DataFrame`

- `DataFrame.apply(np.function, axis)` applies a NumPy function
on the `DataFrame` axis.

```python
companies3.apply(np.mean)
companies3.apply(np.sqrt)
companies3.apply(np.sqrt)[ :2]
```

---
# Getting started with `pandas`
### Import/Export data
`pd.read_csv("PATH_NAME_OF_*.csv")` reads the csv file into `DataFrame`.
 - `header=None` does not read the top row of the csv file as column names.
 - We can set column names with `names`, for example, `names=["a", "b", "c", "d", "e"]`.
 
- `DataFrame.head()` and `DataFrame.tail()` prints the first and last five rows on the Console, respectively.

```python
nbc_show = pd.read_csv("https://bcdanl.github.io/data/nbc_show_na.csv")
# `GRP`: audience size; `PE`: audience engagement.
nbc_show.head()   # showing the first five rows
nbc_show.tail()   # showing the last five rows
```

---
# Getting started with `pandas`
### Export data
`DataFrame.to_csv("filename")` writes `DataFrame` to the csv file.
 - `index=False` and `header=False` do not write row index and column names in the csv file.
 - We can set column names with `header`, for example, `header=["a", "b", "c", "d", "e"]`.

```python
nbc_show.to_csv("PATH_NAME_OF_THE_csv_FILE")
```

---
# Getting started with `pandas`
### Summarizing `DataFrame`

- `DataFrame.count()` returns a Series containing the number of non-missing values for each column.
- `DataFrame.sum()` returns a Series containing the sum of values for each column.
- `DataFrame.mean()` returns a Series containing the mean of values for each column.
  - Passing `axis="columns"` or `axis=1` sums across the columns instead:

```python
nbc_count = nbc_show.sum()
nbc_sum = nbc_show.sum()
nbc_sum_c = nbc_show.sum( axis="columns" )
nbc_mean = nbc_show.mean()
```

---
# Getting started with `pandas`
### Grouping `DataFrame`

- `DataFrame.groupby(col1, col2)` groups `DataFrame` by columns (grouping by one or more than two columns is also possible!).

- Adding the functions `count()`, `sum()`, `mean()` to `groupby()` returns the sum or the mean of the grouped columns.

```python
nbc_genre_count = nbc_show.groupby(["Genre"]).count()
nbc_genre_sum = nbc_show.groupby(["Genre"]).sum()
nbc_network_genre_mean = nbc_show.groupby(["Network", "Genre"]).mean()
```

---
# Getting started with `pandas`
### Class Exercise

Use the `nbc_show_na.csv` file to answer the following questions:

1. Find the top show in terms of the value of `PE` for each Genre.

2. Find the top show in terms of the value of `GRP` for each Network.

3. Which genre does have the largest `GRP` on average?