I tracked every book I read for five years.

Five Years of Reading


## Another one

It’s your boy, back for another year in review. If you’re new here and wondering what this is, it boils down to me half-heartedly throwing some words together in an attempt to fight off the feeling that I didn’t accomplish anything last year. 

Before I sat down to write this, I had a quick read through [last year's review](https://dyl.dev/posts/year-in-review-2021), and it seems like many of the same themes were present in the last two years of my life. I’m going to take that as a sign that I need to make some bigger changes in 2024. For now though, let’s recap my life in 2023...


## What gave me energy?

**Permanent Residency** •
The year got off to a great start, which is when I was granted permanent residency in Australia. I was delighted to have all of the anxiety, worries and scheming that comes from being on short-term work visas behind me. Over the past four years of living in Sydney it has started to feel more and more like home to me.

Australia is by no means a perfect country, but I do find that life is just easier here. It's unfortunate for many immigrants (my friends included) there can be no clear path to permanent residency, even after living here for 4 years. Living with the uncertainty of this whilst trying to make this place your home is super stressful. I'm grateful to be past the visa stage of my Australian life and now looking forward to applying for Citizenship in the coming year. 

**Relationship** •
I’m happy to report that I have made it through another year of a fulfilling and loving relationship, where I have somehow convinced an amazing woman to spend her time with me. She truly makes my life so much easier, more exciting and full. My girlfriend (Gabby) makes me want to be a better person, cares for me when I’m sick, lifts me up when I’m feeling down and even calls me out on my bullshit, from time to time.

**Extended Leave** • 
I took a 6-month break from work to recharge, do lots of traveling and spend quality time with Gabby. Taking a long break like this has been something I’ve wanted to do for years now, and I finally got the opportunity to do it last year. 

Over the course of 2023, we travelled to: Hong Kong, Taiwan, Vietnam, Thailand, Indonesia, Perth, Ireland, London and Japan twice! Snowboarding is an expensive hobby to pick up but I might have caught the bug. Gabby is still working in Singapore which is the perfect base for traveling around Asia. I’ve spent a lot of time in Singapore over the past two years but my favorite part about it is probably the airport, because it means we’re leaving behind the oppresive Singaporean humidity. 

Being in a long-distance relationship (for now) can be challenging at times and so I am really grateful for these 6 months together, and all the memories that were created. It's great to have an employer like Canva that was so supportive in my decision to take this extended period of time off. I’ve just finished my first week back at Canva and I’m feeling refreshed and ready to dive back into some work.

I did my best to get away from my laptop for the majority of the break, but I did do some work on personal projects and reading/writing in the first two months: I picked up [Golang](https://go.dev/), played around with recent ML models, went through MIT's [6.5840: Distributed Systems](https://pdos.csail.mit.edu/6.824/) and completed the fantastic [Nand to Tetris](https://www.nand2tetris.org/) project.

**Comedy** • 
I feel like political correctness and cancel culture has gone too far in recent years. We're on a path to an environment of fear, self-censorship, suppression of personality and excessive virtue signaling. I find that it’s more and more difficult to have nuanced conversations with people, especially in the workplace. People are so deathly afraid of how their words might be perceived by others that they prefer not to speak about anything real at all.  

Listening to more comedians and comedy podcasts has helped me to take the world less seriously again. Some of my favorites are: [Stavros](https://www.youtube.com/@StavvyBaby), [Mark Normand](https://www.youtube.com/@marknormand) and [Matt & Shane's Secret Podcast](https://www.youtube.com/@mattandshanessecretpodcast9138/videos). Matt and Shane have such great chemistry and I really appreciate how down to earth and self deprecating they are.  Usually they are just making random observations about real life and don’t feel the need to censor themselves out of fear of being cancelled, if they think something is funny they will say it. Their humour really reminds me a lot of a TV show that I love, [Peep Show](https://www.imdb.com/title/tt0387764/).

**Miscellaneous** • 
The first few times I tried bubble tea I was fairly unimpressed, but then I discovered Fruit tea in Taiwan this year and got a little addicted (with Passion Fruit tea specifically). Some other snack obsessions I picked up in Singapore last year were: Coconut Water, Pineapple and Iced Coffees.

In november, after almost four years of being a strict vegetarian, I started eating fish and meat again. I'm stilling eating mostly vegetarian food, but I'm now more flexible and less restrictive. As a vegetarian I found it very difficult to get enough protein in my diet and at least for now I’ve opted for the ease and freedom of a more flexible diet. 


## What took my energy?

**Neck Injury** • 
Since February I have been dealing with chronic neck stiffness and pain, which I can’t seem to get rid of. We’re still not exactly sure what’s causing it or the best way to treat it, which really sucks. I’ve tried so many different physiotherapists, doctors, specialists, medical imaging and medications but am still fairly in the dark about how to get better.

Luckily, the pain levels are manageable and it’s more uncomfortable than anything, but when this goes on indefinitely with no definitive diagnosis and no path to improvement other than a “wait and see” kind of approach it’s extremely frustrating. It’s likely that I got some kind of whiplash injury when I was snowboarding, but the waters are muddied by some nerve issues I have from Brazilian Jiu-Jitsu injuries. 

At times this has been a real drainer on me physically and mentally, but I'm living with it as best as I can. I've had to give up Brazilian Jiu-Jitsu and Football, and I can’t really do any resistance training or any other strength based exercise. I’ve had to fall back on walking/running as my main form of exercise (which is fairly boring tbh) with some Pickleball thrown into the mix.

This injury has really made me conscious of how intricate and fragile human bodies are. If/when I recover, I will definitely be more careful about taking care of my body (especially as I age) and I will take immense pleasure in small things like being able to sit back and read a book without my nervous system going nuts. It’s also been a good reminder of the importance of having a good health insurance policy. Some other things that really flare up my neck include: carrying a backpack, sitting on a soft chair/sofa or looking down at a phone/book for more than a couple of minutes. I mainly read lying flat in bed now, holding a kindle in the air above my head. I much prefer physical books but I’ve had to give them up for the time being.

**Anxiety/Worry** • 
Despite the extended break from work, last year was definitely a pretty stressful year for me. The obvious culprit being the ongoing neck issues and the chronic discomfort, stress and anxiety that goes along with a long term injury like this. 

It was great to spend Christmas with my family in Ireland (my third time being back since we moved to Australia). This was the first where I really noticed that my parents are getting old and that some day in the not too distant future I will likely have to care for them, in the same way that I see them caring for my now elderly grandparents. Ireland is quite literally on the opposite side of the world to Australia and I’m worried about how I’m going to be able to be there for my parents without making big compromises of my own independence and happiness. While not yet a pressing concern, I know it’s coming at some point and it’s a bit unsettling if I’m being honest. Being an adult sucks. And to add salt in the wound, my hairline is starting to recede :)

The Russia/Ukraine and Palestine/Gaza conflicts are evidence that humanity is not yet as enlightened as the west has liked to pretend in recent years. I’ve read enough history to know that people can and will do horrific things out of fear, hatred and necessity. We don’t know what kind of societal changes climate change and AGI will bring, but I’m willing to bet that there’ll be an awful lot of conflict and suffering along the way.

One of the main downsides of Australia is the unaffordability of housing and the inequality is growing because of this. It often feels like the clock is ticking and that I need to take out a horrifyingly large mortgage now or I’ll never be able to afford a house in the future. At the same time, the housing market seems like it’s in a bubble that has to burst. When you consider how few large cities there are and how many immigrants will be moving here over the next 30 years it does help to rationalize the “property go up” narrative that is dogma here in Sydney. The Australian economy and housing market is completely dependent on immigration to keep it growing and no government is going to stop the music any time soon. People are property-obsessed here, and it’s starting to rub off on me in ways that I don’t like.

**Smartphone/Media** • 
I did cover this at length in last year’s post, but I wanted to give a quick update here. The war for my attention is fully raging still. In my goal to use my smartphone less, I definitely made some progress last year. I’m hoping to write about this and share what I’ve learned and what worked for me, but I want to give it more time before I start preaching about how great and virtuous I am.

When it comes to reducing my media consumption, I still have a lot of work to do. While I rarely check social media and don’t really watch much TV, I 100% have a youtube and podcast addiction. I spend way too much time listening to other people talk and not nearly enough time thinking for myself. 

I've also caught myself in cycles of refreshing [BBC World News](https://www.bbc.com/news/world) every couple of hours. Consuming global news like this on the internet only serves to reinforce anxiety about the economy, natural disasters, climate change, etc. The old saying “if it bleeds, it leads”' is certainly true and as somebody who is inclined towards cynicism, it doesn't do good things for my worldview and makes me less optimistic overall. History and geopolitics are my biggest interests though so it’s really hard for me to stay away. I’m really interested in exploring the concept of an  “Information Diet” some more and how this could be beneficial to me.

**Job Dissatisfaction** • 
There’s a lot of positive aspects to being a software engineer (especially the pay). If your main goal is to have a secure job, build some wealth and get through life comfortably it’s a great career choice. Despite this, I’ve often struggled to find a lot of satisfaction and meaning in my work. I love being able to roll out of bed at 9.am, obviously. Or the freedom to work from anywhere with an internet connection.

But I think I need more than this. I need to feel challenged. I need to learn new skills. I need to spend less time talking and more time doing. I need to feel like I’m working towards something. I need to know that if I work 50% harder it’s going to make a difference. I need to feel like the work I complete was meaningful, that it moved a needle somewhere. I find myself less and less interested in computers and more interested in learning about the real world: history, physics, evolution, geopolitics and economics. The grass is always greener on the other side, of course. 

## Random favorites 

* Album: [Sampha - Lahai](https://open.spotify.com/album/5GuWww4OaildzkmTTlfMN3?si=Au7TpllpTwy8hGKoQ5fLzQ) (2023 was a meh year for music)
* Movie: [Oppenheimer](https://www.imdb.com/title/tt15398776/)
* Tv Show: [The Bear](https://www.imdb.com/title/tt14452776/)
* Book: [The Count of Monte Cristo](https://www.goodreads.com/book/show/7126.The_Count_of_Monte_Cristo)
* Audiobook: [The Hobbit](https://www.goodreads.com/book/show/5907.The_Hobbit) (Andy Serkis' narration is phenomenal)
* Podcast: [Dwarkesh Podcast](https://open.spotify.com/show/4JH4tybY1zX6e5hjCwU6gF) (great guests and a likeable host)
* Hot Take: The hot takes from my last two year in reviews were lame and obvious

## Looking ahead

I’m definitely looking forward to a more relaxed (and stationary) 2024 in Sydney. In the prior 12 months I have moved apartments three times and been on more than 35 flights. I'm hoping to cap next year's travel at 2-3 trips. 

As a closing note, I’m publicly committing to diving back into Machine Learning this year. I may end up switching specialities at Canva, joining a early-stage startup or even something more drastic like starting a Phd. program. Let's see what 2024 holds :) 
  
  
  

Random thoughts and reflections on last year, again.

Year in Review (2023)

Reminiscing on my path through university and into Software Engineering.

My Path to Software Engineering


## Introduction

This post accompanies some Typescript/Puppeteer code I wrote to download the entire essay catalog of Paul Graham in PDF form. If you don’t know who Paul Graham (PG) is, he’s the somewhat legendary co-founder of [Y Combinator](https://www.ycombinator.com/) and [Hacker News](https://news.ycombinator.com/news). Having been so heavily involved in the tech startup arena for the last two decades, Paul is somebody who’s worth listening to if you are interested in technology entrepreneurship.

PG also happens to be a prolific essayist, which is the reason why I created this post. Dating all the way back to 2001, PG has shared over 200 essays on his personal website (a no-frills static blog that presumably hasn’t changed in appearance since 2001). The [Founders](https://open.spotify.com/show/7txiovdzPARhjm18NwMUYj?si=cf288354665f494c) podcast did a couple of great [episodes](https://open.spotify.com/episode/5cjCCOsWLRRe5RayXcfH48?si=4eebf9a197ab4931) which are a good introduction to the writing of PG, if you want to see what the hype is about that’s a good place to start. 

As somebody who takes a lot of flights (more than I would like), I always have reading material downloaded on my Kindle for offline use. Originally, I manually downloaded 10 of the essays. After reading maybe 3 or 4 of these, I decided to write some code to download the full collection of essays from his website - code which I’ve shared below.  

## Overview

The task I set out to complete was to create an automated script for downloading all 200+ PG essays as PDF files. The full essay catalog is listed on the [articles](http://www.paulgraham.com/articles.html) page of PG's website: 

![essays_list](/posts/paul-graham-essays-download/pg-articles-page.png)
 
Before we jump into the code, let's look at the high level steps that our automated script will need to follow:

1. Programatically open a web browser (using [Puppeteer](https://developer.chrome.com/docs/puppeteer/overview/))
2. Visit the [articles](http://www.paulgraham.com/articles.html) page to get a list of urls for all PG essays
3. For each essay url:
    1. Visit the web page
    2. Save the page contents as a PDF file 

If the script is re-run, it first determines which essays have previously been downloaded and skips downloading these. So only newly added essays are downloaded. When writing any kind of web-scraper/crawler, it's good practice to limit your impact on the websites you interact with.

When all essays are downloaded in PDF format, they can be quickly merged into a single file with a free online [PDF-merge tool](https://www.ilovepdf.com/merge_pdf), before I [email the file to my kindle's email address](https://www.amazon.com/sendtokindle/email).

## Puppeteer 

[Puppeteer](https://developer.chrome.com/docs/puppeteer/overview/) is a NodeJS library that allows you to programatically control a web browser: 
* Puppeteer is commonly used for automating all sorts of browser automation and testing tasks. 
* The Puppeteer API let's you do most things a real user could do a browser, including visiting and interacting with pages (clicking buttons, filling in forms etc).
* By default Puppeteer uses the Chromium browser, which is a lightweight version of Chrome. 
* The browser can be launched in [headless](https://developer.chrome.com/blog/headless-chrome/) mode, which doesn't require a full graphical UI to be spun up. This feature comes in useful when running Puppeteer scripts in low-resource environments, like CI/CD pipelines or Lambda functions.

For now, all you need to know is that we'll be using the Puppeteer library to interact with PG's website in our Typescript code. 
We'll need to navigate to pages (by URL), use CSS selectors to parse some metadata from the HTML (like essay title, publish date) and save the page contents as a PDF.

Alternatively, I could have used traditional web scraping libraries (i.e. with a http request library and something like [Cheerio](https://www.npmjs.com/package/cheerio) to parse the returned HTML). However, I'm experienced working with Puppeteer and the save as PDF feature is perfect for my use-case. 

Without further ado, let's jump into the code.

## Code
I won't go into the full implementation of every function here, but the full code (Typescript) is available on [github](https://github.com/DigUpTheHatchet/paul-graham-essays-public).
Also, this isn't a tutorial for the Puppeteer library; the documentation is excellent and there are already plenty of great tutorials for using Puppeteer available on the web.

Below is the entrypoint for the script. We have broken down the logic into five logical steps:
```typescript
(async () => {
  // 1. Initialize a Puppeteer "Browser" object
  const browser: Browser = await getBrowser();
  // 2. Get the urls of all PG essays
  const allEssayUrls: string[] = await getAllEssayUrls(browser);
  // 3. Filter out essay urls that we don't need/want to download
  const essayUrlsToDownload: string[] = getEssayUrlsToDownload(allEssayUrls);
  // 4. Visit and download all essay urls
  await downloadEssays(browser, essayUrlsToDownload);
  // 5. Clean up the Puppeteer/Chromium resourses created
  await browser.close();
})();
```

**Step 1: Initialize the Browser**
```typescript
  // 1. Initialize a Puppeteer "Browser" object
  const browser: Browser = await getBrowser();
```
We first need to create a Puppeteer *Browser* object, which we will use to programatically control our Chromium browser instance. 
We will be using a single *Browser* instance for the entire lifetime of our script:

```typescript
// Instantiates a Puppeteer Browser, with Chromium browser as default
async function getBrowser(): Promise<Browser> {
  const browser: Browser = await puppeteer.launch({
    headless: true, // Change to false if you'd like to see the Browser window when the script is running   
    args: [`--window-size=1080,1024 --Mozilla/5.0 (iPhone; CPU iPhone OS 16_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Mobile/15E148 Safari/604.1`],
    defaultViewport: {
      width:1080,
      height:1024
    },
  });
  return browser;
}
```

**Step 2: Retrieve the Essay Urls**
```typescript
  // 2. Get the urls of all PG essays
  const allEssayUrls: string[] = await getAllEssayUrls(browser);
```

In this step we visit the [/articles](http://www.paulgraham.com/articles.html) page and extract the list of urls for all the essays.
First, we  create a *Page* on the Browser object (like a tab) and use *page.goto()* to navigate to the articles page.
Next, we use Puppeteer's [Query Selectors](https://pptr.dev/guides/query-selectors) to interact with the DOM and extract the list of all urls or links on this page:

```typescript
  const page: Page = await browser.newPage();
  // Visits the essays index page and determines the list of all essay urls 
  await page.goto('http://www.paulgraham.com/articles.html', { waitUntil: 'networkidle2' });

  const hrefElements = await page.$$("font > a");
  const allUrls: string[] = await Promise.all(
    hrefElements.map(async (he) => he?.evaluate(e => e.href))
  );     
```

Because we've extracted the links for every url on this page, we need to do some filtering to only get the essay urls that we want. 
In this filtering process I remove external links, duplicate essay urls and exclude some essays that were not interestin:

```typescript
  // Filter out any urls that link to a different website
  const pgUrls = allUrls.filter(url => url.startsWith("http://www.paulgraham.com/"))

  // This is a list of some urls/essays that I noticed were not interesting
  // identified by their url suffixes, these essays will not be downloaded
  const unwantedEssaySuffixes = [
    "rss.html",
    "index.html",
    "fix.html",
    "noop.html",
    "rootsoflisp.html",
    "langdes.html",
    "lwba.html",
    "progbot.html"
  ]

  // Filter out essay urls if they have one the unwanted url suffixes
  const keepUrls: string[] = pgUrls.filter(url => {
    return unwantedEssaySuffixes.every((suffix) => !url.endsWith(suffix))
  });

  // Remove duplicate essay urls
  const uniqueUrls = Array.from(new Set(keepUrls));
  
  return uniqueUrls;
}
```

If you log the *uniqueUrls* return value, you'll see the extracted list of essay urls: 
 
![essays_urls](/posts/paul-graham-essays-download/pg-essays-url-list.png)

**Step 3: Filter Essay URLs**
```typescript
  // 3. Filter out essay urls that we don't need/want to download
  const essayUrlsToDownload: string[] = getEssayUrlsToDownload(allEssayUrls);
```

This step is straightforward. We simply scan our */essays* folder to determine which essays to downloaded (i.e. new essays).
When the script is first run, the */essays* folder will be empty and all essays will be downloaded. 
If the script does not find any new/missing essays it will not download any essays:

```typescript
// Filters essay urls to exclude any essays that have already been downloaded 
function getEssayUrlsToDownload(essayUrls: string[]): string[] {
  const existingEssayIds: string[] = getExistingEssayIds();
  const essayUrlsToDownload: string[] = essayUrls.filter(url => {
    return !(_.includes(existingEssayIds, getIdFromEssayUrl(url)));
  });

  return essayUrlsToDownload;
}
```

**Step 4: Download Essays**
```typescript 
  // 4. Visit and download all essay urls
  await downloadEssays(browser, essayUrlsToDownload);
```

Now it's time for our script to actually visit all of the essay urls and download the contents:

```typescript
async function downloadEssays(browser: Browser, essayUrls: string[]) {
  const page: Page = await browser.newPage();

  // One-by-one calls 'downloadEssay' for each url provided
  // I have chosen not to use parallelization and a 4 second delay 
  // is added between each download (to be nice to PG's website)
  await Bluebird.mapSeries(essayUrls, url => {
    return Bluebird.delay(4000)
      .then(() => downloadEssay(page, url))
  });
}
```

 
As mentioned in the overview, it's good practice to write your webscraping scripts in a way that reduces the impact on the website you are interacting with.
In this case, I'm sure PG's website can handle the traffic, but in general services like cloudflare are often used by websites to detect and block bot traffic from scraping their content.

I can run this locally on my Macbook, so I don't need to parallelize/optimize for run-time.
I've purposely chosen to visit and download each essay sequentially, with a four second delay between each navigation.


Let's look at the implementation of the *downloadEssay()* function:
```typescript 
// Visits and downloads a single essay page (by url)
async function downloadEssay(page: Page, essayUrl: string) {
  await page.goto(essayUrl, { waitUntil: 'networkidle0' });
  // Use the url slug as essayId, e.g. `http://www.paulgraham.com/getideas.html` -> "getideas"
  const essayId: string = getIdFromEssayUrl(essayUrl);

  // Parse the essay title and publish year/month values 
  const titleElement = await page.$('body > table > tbody > tr > td:nth-child(3) > table:nth-child(4) > tbody > tr > td > img');
  const title = await titleElement?.evaluate(el => el.getAttribute('alt'))
  const [month, year] = await getMonthAndDateFromPage(page);

  ...
}
```

*Page.goto()* is used to navigate to the essay url passed. 
Once the page contents are completely loaded, we once again use the Query/CSS selectors to extract information from the page. 
This time, we are parsing metadata about the essay from the HTML contents. 
The metadata we want is the title of the essay and the publish year/month. These values are at the top of each essay page: 

![essay_metadata](/posts/paul-graham-essays-download/pg-metadata.png)


If you're not familiar with CSS selectors and DOM traversal, this line:
```typescript
  const titleElement = await page.$('body > table > tbody > tr > td:nth-child(3) > table:nth-child(4) > tbody > tr > td > img');
```
might look like gibberish to you. If you want to learn more about how this works I would recommend this [Youtube Video](https://www.youtube.com/watch?v=WDFsKeLdgZ4) and the excellent cheatsheet available in the video description.

Finally we save the page contents as a PDF, using the *Page.pdf()* method: 
```typescript
  // e.g 'essays/2023/greatwork-2023-07.pdf'
  const outputFilename = `${downloadDir}/${essayId}-${year}-${MONTH_MAP.get(month)}.pdf`; 
  
  // Create the download directory (if it doesn't already exist)
  if (!fs.existsSync(downloadDir)){
    fs.mkdirSync(downloadDir, { recursive: true })
  }

  // Download/save the page contents as a PDF
  await page.pdf({
    path: outputFilename,
    margin: { top: '50px', right: '25px', bottom: '50px', left: '25px' },
    printBackground: true,
    format: 'A4',
  });
```

Essays are saved in the */essays* directory, with subdirectories for each year:

![essays_dir](/posts/paul-graham-essays-download/pg-essays-directory.png)

Parsing HTML page contents with CSS selectors is notoriously brittle and error prone.
If the HTML structure of the page changes ever so slightly, it can cause the code that parses value (like our metadata) to throw an error.
To account for this, I've created default values for the parsed year/month values. If the script encounters any problems parsing these values on an essay
the essay will still be downloaded, but into the *2xxx* directory:

![essays_default_dir](/posts/paul-graham-essays-download/pg-essays-default-dir.png)

**Step 5: Cleanup**

Finally, we need to cleanup the Chromium browser instance that we created:
```typescript
  await browser.close(); 
```

## Running the Script

I've chosen not include the essay PDFs in this repository, as the essays are PG's content (not my own). 
Here are the simple steps if you want to get the essays for yourself:

1. **Clone** the code from my [github repo](https://github.com/DigUpTheHatchet/paul-graham-essays-public):
```bash
git clone git@github.com:DigUpTheHatchet/paul-graham-essays-public.git
```
2. **Install** the node dependencies:
```bash
npm install
```
3. **Run** the script:
```bash
npm run start 
```
As is, the script will take a couple of minutes to run. This duration can be greatly decreased by parallelizing the *downloadEssays()* function.

You now have your own personal archive of PG's essays available for offline consumption!

## Unfinished Work

I was going to title this section as **Future work**, but I don’t have any further plans to continue working on this.
Originally, I intended to have the script running on an automated schedule, without a human in the loop. 

However I've decided to use it as, manually running the script once or twice a year to pick up the new essays, for two reasons: 

1. PG publishes pretty infrequently now, so this infrequent cadence should work just fine.
2. His essays are usually timeless, rather than timely, so there’s no need to have new essays delivered to my kindle as soon as they are published. 

**Improvements**

If you wanted to polish this up a little further, some ideas are:
1. **Progamatically merge the downloaded PDF files**:
* Rather than manually merging PDFs, utilize something like the [pdf-merger-js](https://www.npmjs.com/package/pdf-merger-js) library.
2. **Deploy the script as an automated/scheduled workflow**:
* Instead of running this script locally, we could easily deploy it to run as an AWS Lambda Function. 
* Because of [headless](https://developer.chrome.com/blog/headless-chrome/) mode, Puppeteer works well in a Lambda runtime enviroment (without GUI resources). 
* I extensively used Puppeteer in automated workflows on Lambda/GCF in my previous role at [Rezdy](https://rezdy.com/). There are some considerations that do need to be accounted for however (e.g. the dependency bundle size) when running Puppeteer in Lambda/GCF.
* Instead of saving files to the git repo, they would instead be persisted in an S3 bucket. 
* The Lambda Function could then be scheduled to run weekly, saving any newly published essays to the s3 bucket.
3. **Automatically send any new essays to kindle:**
* Once the script is deployed as a (scheduled) Lambda Function, it could programatically send any new essays to your Kindle (e.g. using [SES](https://iamkanikamodi.medium.com/write-a-sample-lambda-to-send-emails-using-ses-in-aws-a2e903d9129e)).
  
  
  

A Puppeteer script to download all of Paul Graham's essays as PDF files.

Downloading Paul Graham's Essays


## Getting Called Out

Gab:
> You used way more emojis when I first met you

Dyl:
> I think it was a mating ritual haha

I recently got called out by my girlfriend for apparently sending her less emojis than I did when we first started dating..

I didn't like the implication that I had become a lazy texter over time; it is plausible though, as it's very likely that I was [mirroring](https://en.wikipedia.org/wiki/Mirroring) a higher level of emoji usage as a mating ritual. 
Naturally, I wanted to clear my name, so I downloaded our WhatsApp chat history and opened up a Jupyter notebook.

This post includes code snippets, and the full notebook code can also be found [here](https://github.com/DigUpTheHatchet/whatsapp-data-analysis/blob/main/Emoji%20Analysis.ipynb).

## Data Preparation

Firstly, I had to export the message history from our WhatsApp chat.
Instructions for doing so can be found [here](https://faq.whatsapp.com/1180414079177245/). The exported text file will contain a line for each message of your chat history.
e.g. 
> [18/5/2022, 6:05:12 pm] Dyl: I might need to get bus

> [18/5/2022, 6:07:25 pm] Gab: Okay

Then in a Jupyter notebook, it was simply a case of reading this text file line by line
```python
# Use readlines() to read the raw WhatsApp messages into a list of strings
file1 = open('gab_chat.txt', 'r', encoding = 'utf-8')
Lines = file1.readlines()
raw_lines = [line for line in Lines]
```

After inspecting the messages, I noticed that there were some gifs/images, forwarded messages and other system messages from WhatsApp that needed to be removed. Replies to gifs/images and reactions to messages will also be excluded
```python
# This function is applied to the raw input lines to drop unwanted and/or system messages 
def filter_raw_lines(raw_lines):
    lines_to_keep = []
    # First line is a system message from WhatsApp, drop it with "[1:]" splice
    for line in raw_lines[1:]:
        # Remove system messages, gifs/images and forwarded messages with "str.startswith(line, '[')"
        if (str.startswith(line, '[') and "Missed video call" not in line and "Missed voice call" not in line and "GIF omitted" not in line and "image omitted" not in line):
            lines_to_keep.append(line)

    return lines_to_keep

filtered_lines = filter_raw_lines(raw_lines)
```

With unwanted messages filtered out, I then proceeded to begin parsing the lines, e.g. to extract metadata 
```python 
# This function parses the message content and metadata out into separate components
def parse_line_contents(line):
    split_line = line.split('] ', 1)
    date_str = split_line[0][1:].split(',')[0]
    # Note: dropping hour/minute/sec information from this point on
    date = datetime.datetime.strptime(date_str, '%d/%m/%Y')

    author = split_line[1].split(': ', 1)[0]
    message = split_line[1].split(': ', 1)[1].strip()
    
    return [date, author, message]

parsed_lines = [parse_line_contents(line) for line in filtered_lines]
```

At this point the data was still a list of lists, so I converted it to a [polars](https://www.pola.rs/) dataframe. I actually tried to use pandas at first, but I didn't have the attention span to figure out aggregations in pandas and the polars interface was much easier to pick up. Note: I also filtered the dataframe to contain only the first 12 complete months of our chat history, which which was December '21 to November '22
```python 
messages = pl.DataFrame(data = parsed_lines, schema = {'datetime': 'datetime', 'author': 'str', 'message': 'str'})
messages = messages.with_columns(pl.col("datetime").cast(pl.Date).alias("date"))

# Filter the messages dataframe to only have the first full 12 months of data (Dec 2021 to Nov 2022)
filtered_messages = messages.filter(
    pl.col("date").is_between(
        datetime.datetime.strptime('2021-12-01', '%Y-%m-%d'),
        datetime.datetime.strptime('2022-11-30', '%Y-%m-%d'))
)
filtered_messages.head(5)
```
![filtered_messages](/posts/whatsapp/filtered-messages.png)

Finally, I extracted a bunch of information/metadata from the messages out into new columns on the dataframe. I'm using the [emoji](https://github.com/carpedm20/emoji/) package to extract the emoji information out of the messages
```python
prepped = filtered_messages.with_columns(
    pl.col('message').apply(lambda msg: emoji.emoji_count(msg)).alias('emoji_count'),
    pl.col('message').apply(lambda msg: len(re.findall(r'\w+', msg))).alias('word_count'),
    pl.col('message').apply(lambda msg: [match['emoji'] for match in emoji.emoji_list(msg)]).alias('emoji_list'),
    pl.col('date').dt.strftime("%b").alias("month"),
    pl.col('date').dt.strftime("%a").alias("day")
)

# Because I want a full 12 months of data only, but we are not in a complete calendar year
# we need to do some hacky stuff to get the December month as first in the ordering
# 0 = Dec 21, 1 = Jan 22, 11 = Nov 22
months_order = ['Dec', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov']
months_order_dict = {val: idx for idx, val in enumerate(months_order)}
prepped = prepped.with_columns(
    pl.col('month').apply(lambda mon: months_order_dict[mon]).alias('month_index')
)

prepped.tail(5)

```
![prepped_messages](/posts/whatsapp/prepped-messages.png)

We now have our prepared dataset and we're ready to begin our rigourous and highly professional data analysis, for which I'll be using the [seaborn](https://seaborn.pydata.org/) data visualization package.

## Data Analysis

Despite the title image I am definitely not a Data Scientist and infact have no idea what I'm doing here :)

Before getting into the trends over time, I wanted to first start with some visualizations of the total number of messages and emojis sent in 12 months

![total_messages](/posts/whatsapp/1-total-messages-sent.png)

![total_emojis](/posts/whatsapp/2-total-emojis-sent.png)

Despite sending a similar amount of messages, my total number of emojis sent (671)
is less than half the number sent by Gab (1516). In fact, Gab sent ~70% of all the emojis in our chat history. So, it's already not looking good for me. 


As a quick reminder, the original claim that I wanted to prove/disprove with this analysis was the following: "Do I send less emojis now than I did before?". To answer this, we're going to look at our data on a month-by-month basis, running from December to November (the first 12 complete months I had data for)

![monthly_messages](/posts/whatsapp/3-messages-sent-per-month.png)

If we look at the numbers for monthly sent messages above, we can see a very wide range: as little as ~400 in December peaking at ~1400 in July. I think the low total in month one is probably a result of us still getting to know each other and playing it relatively cool at this point. The wide variance for other months might correlate with how much time we spent physically together in those months, and the trend downwards in later months could be explained by our adoption of a long nightly phone call - meaning we would text less throughout the day

![monthly_emojis](/posts/whatsapp/4-emojis-sent-per-month.png)

Coming to emojis sent per month, we can see that I sent less emojis than Gab in every single month. I can already tell from this that month one and two do seem to have been some of my bigger months for emoji usage (relative to Gab's). Now let's graph a trend line

![trendline_emojis_sent](/posts/whatsapp/5-trendline-emojis-sent-per-month.png)

The above chart shows a trendline of emojis sent (over time). At first glance the slope of both lines seems pretty similiar, and I was actually happy with this (as it would imply we're both using less emojis, at a relatively similar decreased rate). This seemed a bit off to me: if you look at the start and end points of each trendline, you can see that the blue trendline (mine) drops from ~80 to ~40 (a ~50% decrease) whereas the yellow trendline (Gab's) drops from ~140 to ~110 (a ~23% decrease).

So what does this mean? Don't ask me! I told you I have no idea what I'm doing. If I had to guess though, I would say that the high variance of message counts from one month to the next is making it difficult to detect and pull out trends from the above charts. For example, it's hard to see how much emoji count decreases are a result of us sending less emojis or how much of the decrease is coming from us sending less messages overall. I now decided that I probably need to calculate and graph an emoji-per-message rate over time

![emoji_rate_per_month](/posts/whatsapp/6-emoji-rate-per-month.png)

The above chart graphs an "emoji rate" for each of us over time. Here we're defining the emoji rate as the number of emojis sent per message. For example, if we look at the month of December my emoji rate was ~0.1 which means that I sent approximately 0.1 emojis per message, or in other words I sent (on average) one emoji for every 10 messages.

![emoji_rate_per_month_table](/posts/whatsapp/8-emoji-rate-per-month-table.png)

This emoji rate graph gives us a better view of the relative emoji frequencies and how they changed over time, independent of how many messages happened to be sent in each month. We can observe that my (blue) emoji rate was highest in month one and two and has indeed decreased over time. Gab's (yellow) emoji rate fluctuates more widely month to month, but has a much higher baseline and is actually trending upwards in the later months. Speaking of trends, I wanted to see how the trendlines would look on this emoji rate data

![trendline_emoji_rate_per_month](/posts/whatsapp/7-trendline-emoji-rate-per-month.png)

Right away I can say that the slopes of these two trendlines appear more distinct from each other (compared to the previous trendline graph above). The trendline of Gab's emoji rate (yellow) is relatively level, but the trendline for my emoji rate (blue) is clearly declining over time. 

At this point, I'm happy to concede defeat and say that I have indeed become lazier with my emojis over time. Hopefully this blog post will be a small consolation Gab :)


## Future Work

In the spirit of done is better than perfect, I'm posting this now with some future items left todo:
* Complete the analysis of the actual emojis used and trends (see second half of the [notebook](https://github.com/DigUpTheHatchet/whatsapp-data-analysis/blob/main/Emoji%20Analysis.ipynb)).
* Re-do some analysis in which emoticons like ":)" are counted as emojis (as I use them a lot).
* Regenerate the graphs after redefining the "emoji rate" as the number of emojis per 10 messages. This would make the emoji rate graphs more interpretable (as decimal numbers can be trickier to compare to each other).
 
 

Analysis of my emoji usage over time, using Polars and Seaborn.

Have I become a lazy texter?

Year in Review (2022)


## So.. what even is this?

Earlier this year I discovered James Clear's (author of [Atomic Habits](https://www.goodreads.com/book/show/40121378-atomic-habits)) blog, and came across his [Annual Review](https://jamesclear.com/annual-review) posts which I found quite to be quite a compelling idea.
This is my first attempt at completing my own "Annual Review" exercise, and actually the first post I'll be writing for this site.

The idea is quite simple: I'll take a look back on the year of 2021 (as lived by me) and reflect on: 
  - what I achieved (and failed to)
  - what I wish I did more of (and less of)
  - what I learned

I think most of us don't take enough time to reflect on where we are (and where we're heading) in life. 
I'm hoping that this exercise will make me be more mindful of my habits and how I've been spending my time lately. 
This will also be an exercise in gratitude and expressing my appreciation for the people and things in my life that I am thankful for.
Considering how poor my memory is — I struggle to remember the plot of the movie I watched last night — these year in review posts might act as a fun little time capsule for me in the future.


## What went well?

**Reading** • 
Over the last few years reading has definitely become my favorite past-time and preferred way to un-wind. 
This was my fourth year completing the [52 Book Challenge](https://www.reddit.com/r/52book/) (the aim being to read at least one book for each week in the year).
In previous years I've had some mixed feelings about the challenge; how it might be negatively influencing my reading experience. 
I found myself avoiding very long books because they might slow down my progress on the challenge and forcing myself to read more than I would have naturally just to reach the basically arbitrary goal I'd set. I was worried that this was taking the fun out of reading for me.

Thankfully I didn't run into either of these feelings when completing the challenge this year (I finished with a total of 57 books).
Allowing myself to count an audiobook every month has given me the leeway I need to read some massive books that I might have otherwise avoided (looking at you *Infinite Jest*).
When I've noticed lately that I'm forcing myself to read, it's usually because I have another five books on my bookshelf that I can't wait to get started on.

For the last three years I've tracked my reading history in a Google Sheet (I know I'm a dork; please don't @ me). I used the data to create the [/reading](https://dyl.dev/reading) page on this site. 
I'm aiming to write up a blog post soon (lol) outlining how I track my reading and how I built the [/reading](https://dyl.dev/reading) page. 

**Wellbeing** • 
Having dabbled with various meditation apps (quite momentarily) over the last few years, I was finally able to build a lasting daily meditation habit this year. 
We all exist (literally) inside our own heads, and this can be a lonely and anxious place to be. 
I'm definitely liable to ruminate/over-think/over-analyse in times of stress or worry, and admittedly I don't believe that sitting cross legged on the floor with my
airpods in for 20 minutes a day is not going to magically rid me of these tendencies, but I do feel that the practice has definitely had a positive impact on my well-being and my ability to 
resist getting caught up in unneccessary worry or imaginary future conversations in my head. 

Progressing from five, ten, and up to twenty minutes a day has been quite rewarding; considering how difficult or long some of those earlier five minute sessions felt when starting out.
If you're wondering, the app I use for guided meditations is: "Waking Up". For directed meditation I would definitely recommend the Daily Stoic, or any of Seneca's work.

I made the transition to the famed "morning person" this year; usually being in bed by 10.30 pm and up by 6.30 am. 
Since I've been working from home more or less full-time this year, I really enjoyed waking up early and having two full hours to relax before the workday began.
I read a lot of heavy non-fiction stuff this year (which is certainly not good bed-time reading material), so these morning relaxations hours were where I did most of my daily reading.

I've done fairly well at limiting my phone usage this year. Instagram is the last hold out of social media apps that I seem to periodically delete but later re-install (usually when drunk or hungover FWIW).
I've managed to avoid TikTok, but have seen first-hand on (the similar) Instagram Reels how much of a deep void of attention that you can get sucked into. I wish I could say that I didn't now know the names of all four members of BlackPink.

**Fitness** • 
My biggest highlight of the year was definitely receiving my blue belt in Brazilian Jiu-Jitsu (BJJ), having re-picked it up in November 2020 after a 5 year hiatus. 
I can honestly say that getting back into BJJ was one of the best decisions I've ever made, having found a really great gym and community here in Sydney at [GFTeam Bondi](https://gfteambondi.com.au/). 

BJJ is more than just a sport: it can teach you so much about yourself (and about your training partners). As a grown man getting regularly beaten up by teenage girls you quickly learn humility and lose whatever ego you brought into the sport with you.
The BJJ community is full of friendly and open-minded people, no matter where you go in the world. In addition to the obvious physical benefits that come from training, being on the mats is a fantastic way to relieve stress, forget about life's problems for two hours, and get into that [Flow State](https://en.wikipedia.org/wiki/Flow_(psychology)).

Another sport I got back into this year was Sunday League Football, playing for Coogee United (8th Division lol).
Despite the great community and camaraderie in BJJ, I was really excited to be playing a "proper" team sport again.
Unfortunately the July/August COVID-19 lockdown ended our season (and our ten game win streak!) prematurely. Having won every game played, we were already planning our victory parade. Lesson learned: don't count your Sunday League trophies before they hatch.
In the light of our team being predominately made up of Irish and Brits, we've done quite well to patch over 800 years of animosity during post-game pints in the Donny each Sunday.

**Work** • 
I'm now coming up on my second year of working as a Data Engineer at Rezdy. Rezdy is a SAAS based in Sydney that creates booking software for tours & activities suppliers.
It's been a bit of a roller-coaster working at Rezdy during this global pandemic (as our industry is very much travel adjacent). 
Considering all of the uncertainty in the market and life in general, I really can't say enough good things about the people and management team at Rezdy and the support/flexibility they've provided to me over the last two years.

I was promoted to Senior Data Engineer back in April and since then it's been amazing to see the Insights team grow from it's initial three members to it's current total of six.
For most of my time at Rezdy I was the sole Data Engineer; now that we have three Data Engineers I've had a great opportunity to develop my technical leadership and mentoring/coaching skills.
In January I will be doubling down on this skill development and moving into a Tech Lead role in a new — Data Engineering specific — squad at Rezdy.

Outside of the 9-to-5, I finally got around to building this personal website. 
I created it using Next.js, React & ChakraUI and it's deployed on Vercel. This was my first real taste of front-end development and the frameworks have definitely become a lot easier to 
use than the last time I tried (and gave up on) building a website.
I also made my first (very small) open source contribution to [Data Build Tool (DBT)](https://www.getdbt.com/); I'm hoping to do more of this next year as I'm a huge fan of the the tool and it's community in general.

**Personal Relationships** • 
I'm still not really sure where on the spectrum of Person-Professional I'm going to target this blog, so I won't go into too much detail here for now.
Two years ago, I moved across the world (from Ireland to Australia) with a group of ~20 close friends. Due to the Australian Border Restrictions during COVID-19, we haven't been able to visit home in over two years. In light of this I feel especially lucky and grateful to have this amazing group of friends with me, like a family away from home.
I'm also thankful to have made a lot of new friendships this year as well: through BJJ, Football and work colleagues that have become close friends.

## What could have gone better?

If you know me at all you'll know that I have deeply rooted tendency to be cynical (is "cynical" just sugar coating on the word "negative"?) 
which is something that I've been working on this year. I think I've definitely made good progress on my goals of complaining less and being more positive in general. Preface aside, in light of my move
to being less cynical, I'm going to keep this section a bit lighter in detail than the previous one.

The socially distant elephant in the room, COVID-19, has obviously been pretty shitty for all of us. I'm actually writing this whilst home alone, covid positive and isolating on New Years Eve.
The main impact on me has been the fact that I haven't been able to visit my family in Ireland in over 2 years. 
During this time my Grandfather passed away, and my two surviving grandparents are getting older each year. Thankfully the border restrictions were finally lifted in December this year,
which means I'll be getting to visit home in the next couple of months (aiming for March).
In the grand scheme of things, COVID-19's impact on my life has been pretty minimal, and for this I'm extremely grateful.

I spent a lot of money at the Physio; so-much-so that I'm expecting an invitiation to the college graduation ceremony of his first born child.
BJJ is notorious for injuries, and I've certainly had my fair share this year. My turf-toe injury is now a year old and still has barely healed. At this point it's become second nature to tape my foot everytime I train BJJ, play Football or go for a run. Having a dodgy foot has become part of my character backstory.
I'm also dealing with a double-crush nerve issue on my right side (forearm/shoulder/neck) that consitently impacts my ability to train... even in my day-to-day work I have to wear an elbow brace.

Every concert/festival we booked this year got postponed or cancelled. 
Hopefully now that the tantilizing Covid-Zero policy will never be an option here again, we won't have as many cancellations next year.
I also didn't make enough of an effort to go to live music/gigs in the periods when we didn't have lockdowns this year.

I was hoping to have written at least a handful of blog posts on the site by now — and this is the first — so yeah..

In February, I decided to get a Buzzcut.. thinking I would look like Jake Gylenhal in Southpaw. Spoiler Alert: I didn't look like like Jake Gylenhal in Southpaw.


I had a four-month legal battle with a rental property agency. We moved into a new apartment, only to discover a massive cockroach infestation on day one. 
How bad could it have been you might ask? Let's just say that Sony Pictures have made inquiries about buying the rights for the next Jumanji sequel.
Naturally, we noped out of there, but had to go through the court system to get my $3,600 back from the recalcitrant landlord. 
The stress of these months has definitely fast-tracked my greying hair.

## What did I learn? 

* The more material possesions you accumulate the more annoying it is to move apartment.
* Control is an illusion... your wellbeing and routines are fragile.
* As we get older our bodies will start to fail us. Injuries are inevitable. Hangovers are devastating.
* I'm in a privileged position and have a lot of be grateful for in life.
* Don't fall into the great attention black-hole that is TikTok/Instagram.
* Don't impulse buy clothes without properly considering how they will fit in with the rest of your wardrobe.
* Working from home has it's pros and cons, but ultimately the flexibiltity it provides is awesome.
* It feels good to be charitable. Accumulating wealth should not be your north star.


## Random Favorites 

* Album: [Tyron - Slowthai](https://open.spotify.com/album/7qE3WaOVAAIxgH8WtjbBBj?si=NyIQDkA8RWyHqd-hdzemSw)
* Movie: [Licorice Pizza](https://www.youtube.com/watch?v=AMcwN94D_mY)
* Tv Show: [Sex Education S3](https://www.imdb.com/title/tt7767422/)
* Book: [Guns, Germs and Steel](https://www.goodreads.com/book/show/1842.Guns_Germs_and_Steel)
* Audiobook: [Empire of Pain: The Secret History of the Sackler Dynasty](https://www.goodreads.com/book/show/43868109-empire-of-pain?from_search=true&from_srp=true&qid=fdbC98OtRh&rank=1)
* Podcast: [The Lex Fridman Podcast](https://open.spotify.com/show/2MAi0BvDc6GTFvKFPXnkCL)
* Hot Take: The NFT craze is pure nonsense.


## 2022 Goals
* Priority numero-uno is to get home to Ireland to see my family.
* Spend less time inside. Go for a swim every day.
* Travel (rolls eyes) - I would love to get to visit New Zealand and Japan in 2022.
* Limit unneccessary/impulse purchases. Give money to people who need it more than I do.
* Play more Tennis and get properly into Bouldering/Indoor-Climbing.
* Get back to the gym and put on some weight (I'm currently 72 kg and 78 kg is my target).
* Write at least six blog posts for this site.
  
  

A look back on the trials and tribulations of the 27th year of my existence.