Introduction

This post is a personal story of my path to becoming a Software Engineer.

Lately, I’ve been having serious doubts about whether I want to continue in the Data Engineering/Analytics space. I'm considering alternative routes: should I switch back to Backend? Perhaps I'll rekindle my interest in Machine Learning? Or, do something even more drastic like going back to graduate studies?

To help me evaluate these options, I’ve spent some time reflecting on what got me interested in Software/Data Engineering in the first place - all the way back to starting my Computer Science degree in Ireland.

I’ve wanted to do more writing in public, and so I’m going to share those reflections below :)

Why Computer Science?

The year was 2012 and I was dropping out of college in the first semester. Two months into a Bachelow of Laws degree, I realized I had made a huge mistake. Although I enjoyed Law lectures, I was soon overcome by the feeling deep down that I would absolutely hate climbing the career ladder. Slowly, and then all at once, I was hit with the overwhelming desire to drop out and change course. I knew I wanted to study something more exciting, and that something else ended up being Computer Science (CS).

In school, my favorite subject had always been Mathematics and I was relatively interested in computers. I don’t want to give the wrong impression here however - I was not some technology wunderkind, building PCs/electronics or anything like that. To this day, I still have only a rudimentary understanding of how computers work. I had tried to learn programming once or twice, but this never amounted to anything worth talking about. My interest in computers mainly manifested itself in time wasted browsing online forums and a video game addiction.

So, when my nascent law career came to an abrupt termination, CS seemed like a decent choice for my redo. Growing up in semi-rural Ireland, I didn’t have any role models or adults around me that were working in Tech or Finance, so I only had a vague idea of what CS was and what kind of jobs there were. I couldn't start CS until the following academic year, so I spent the rest of the year working as a cashier in a Betting shop.

First Year

My CS degree did NOT get off to a good start. Although I had never been an overachiever in highschool, I was kind of a straight-B student. I developed a habit of doing the smallest amount of work that would keep me out of trouble with teachers and get a passable grade, but no more than that. I was generally lazy, loved taking shortcuts and was always dodging homework. After all, how could homework compete with video games?

I carried this approach with me into my CS studies, only now, I didn’t have teachers policing my attendance or homework. I was living away from home, with zero parental supervision, surrounded by friends who were even less interested in academics than me. I was a rare sight on the college campus: I never went to lectures or never interacted with any other CS students in my class.

To complicate things, I was still working 20 hour weekends in the Betting shop. I spent most of the year either hungover in bed, watching tv or working. CS has a very high dropout rate in year one: the course material is challenging and it's easy to quickly fall behind as each week progessively builds on the previous weeks lessons.

In my university, your final degree grade comes from 30% of your 3rd year results and 70% of your 4th year results. So if my grades for first year didn’t count for anything, I was not very motivated to make it to lectures or expend effort on assignments. I barely made it through the year with an average grade of 53%, and multiple subjects (e.g. C Programming) dangerously close to the 40% failure cliff.

Second Year

Having scraped my way through 1st year, 2nd year was progressing without much change. Then, in the middle of the year, an internship program with SAP was advertised to our class. The SAP program offered two internships (one 3-month, one 8-month). Additionally, there was a kicker: the interns would get a 500 euro (per month) stipend during the college term. I didn't know anything about SAP, but I knew that I wanted this monthly stipend. Despite my terrible 1st year results, I somehow convinced the hiring manager that I had potential and would reform.

I began to put some effort into my studies now. My attendance was still poor, but I started to make friends with some of my classmates, which helped me to be more engaged with lectures and collaborate on assignments. I was not always committed throughout the school year, I was always able to grind through an intense two week cramming session before each exam period. I went from a being straight-D student to a straight-C student, eventually finishing 2nd year exams with a 68% average.

Looking back, exam season was always an intense time in my circle of friends. The threat of failing a course and having to repeat a whole year was always hanging over our heads. For serial absentees like us, it came down to doing whatever you had to do to get the 40% mark required to pass each course.

When exam season rolled around, each man was on his own, battling their personal procrastination demons. You had to scour the depths of your soul for any pockets of willpower available to put to the task of reaching that 40% threshold. On any given study day, your demons might get the better of you. Often, the only recourse was to transform into a beastly nocturnal creature, fueled by rolled cigarettes and energy drinks, haunting the university library, deep into the twilight hours.

SAP Internship 1

When summer rolled around, it was time for the first of my two SAP internships. I spent 3 months living and working with some of my classmates, and interns from other universities, with SAP in Dublin.

For this first internship, the work itself was not super interesting: we were mostly onboarding and then learning about SAP's HANA database. When this training was completed, we spent the rest of the summer working as support engineers for the HANA product.

Before we wrapped up this internship, we had to apply/interview for our desired return positions for the longer 8 month internship, the following year. At this point I had limited programming skills and what knowledge I did have was more theoretical than practical. Working as a support engineer wasn’t super beneficial in this regard, and I knew I needed to secure a more interesting role for my return. I really wanted the intern position that was open in the ML/Analytics team.

Spoiler Alert: Luckily, I was able to get this! But for the time being, we were heading back to college for 3rd year.

Third Year

This was somewhat of a turning point for me - it was the first time I took a college year seriously from start to finish. I was still on a high from having secured the most interesting role (according to me) for the upcoming internship.

Importantly, because of the monthly stipend from SAP, I didn't have to work part time on the weekends, which meant I had much more time to focus on university while still having a social life. Having spent months living and working closely with some of my classmates, my social/college boundary became more blurry. Going to lectures no longer felt like a chore, it was closer to hanging out with friends, incidentally on campus.

To allow for the 8-month internship, our 3rd year courses were condensed into a single jam-packed semester, with a grueling end of semester exam session. I was up to the challenge however: having been through the wars during year one and two exams, I was by now an experienced crammer. I was accustomed to these short but intense bursts of study (previously I was usually compensating for almost complete absence during the school term.

For the first time I was going into an exam session having gone to the lectures and submitted all of the assignments. When our results came back months later, I would find out that I finished the year with the highest overall grade of 89% :)

SAP Internship 2

With 3rd year classes finished, I started my second internship at SAP in January 2016. I had to move back to Dublin, this time for 8 months, living with the same friends from the previous summer.

For the first few months of this internship, I was working on SAP’s Predictive Analytics offering. This product was a desktop application with a drag-and-drop UI for chaining together Machine Learning (ML) workflows. Working on this product, I finally got to see what software development in the real world was like. I learned about the software development lifecycle: debugging, writing/reviewing code, unit tests, build processes and agile/scrum practices etc.

This was SAP in 2016, however. The application was a beastly Java monolith, on a 6-month release cadence. At the time, SAP was almost entirely on-prem (and likely still is lol). There wasn’t a huge amount of actual ML work involved, as the product was mainly a UI wrapper around existing implementations of various ML algorithms. I did get to spend a lot of time learning about different ML algorithms and typical Data Science workflows. Kaggle was just getting big around this time, and (with my manager's blessing) I got to spend downtime working on Kaggle challenges.

After a few months, I rotated onto the “Big Data” team. In 2016, there was a big buzz around Data Science and Big Data. The term “Data Engineer” hadn't been coined yet.. everything was referred to as “Big Data”. Working on this team, I got my first exposure to Distributed Systems and the Hadoop ecosystem. Learning about Distributed Systems and Big Data processing tools was probably the first time I ever associated programming with something that was fun/exciting.

This was an R&D team, experimenting with Big Data tools and trying to integrate them into SAP's suite of analytics products (e.g. the ML workflow UI I had worked on). I really dived into learning functional programming, specifically Scala (which would be my go to programming language for the subsequent years).

I also learned a lot about distributed tools and infrastructure, e.g. Apache Spark. At the time, we were building ML pipelines using early versions of Spark MLlib, which was still using plain ol’ RDDs. After getting up to speed on Scala and Spark, I spent a couple of months implementing custom MLlib transformers. For example, we needed a bunch of pre-processing algorithms for Sampling and Normalization that were not yet part of Spark's MLlib.

There was something about this new world of Big Data and ML that really excited and inspired my younger self. I was able to read and understand the foundational distributed systems whitepapers, which made me feel like I was somehow smart. In university we learned how to use tools and studied concepts, but it always felt very theoretical. Now, I was working on these distributed systems day-in day-out and I could bridge the gaps between concepts and practice.

I have very fond memories from my time as an intern here, not only because the technology was exciting, but because the people were great too. I was probably more of a net-drain and than net-positive to the team’s output, as I rubbed shoulders with some really talented engineers and team leads, who were always extremely patient and generous with their time. I will always remember how I was treated as an intern here, and I try to emulate this experience for the interns and juniors that I get to work today. I love to explain concepts and ideas without assuming any prior knowledge and to encourage the ethos of “there are no stupid questions”.

As that summer came to a close, I was looking forward to heading back to Galway to start the final year of my degree. I was feeling confident, after getting the results of my 3rd year exams, that I had done well in (for the first time). This confidence took a bit of a tumble when my girlfriend ended our relationship; my first ever heartbreak. After some miserable weeks of being down in the dumps, I found a distraction by rekindling my childhood love of reading. Since this time (2016), I’ve completed an annual 52-book challenge 6 times. Reading has changed my self and who I am, but I will have to leave that story for another day.

Fourth Year

I really threw myself into college in fourth year. I came back from the internship with a determination to get good grades, and a sharpened interest in AI/Machine Learning. There were a lot of interesting electives in this year, e.g A.I, Cryptography and Machine Learning. My internship experience working on ML, along with decent math skills (I had taken a lot of math electives) gave me a head start on these subjects.

In our ML course, we focused on training, evaluating and interpreting models. We focussed on traditional ML algorithms (i.e. pre-deep learning), including Reinforcement Learning and probabilistic approaches. This was 2017, so Neural Networks’s had been back on the scene for a few years by then. I’m not sure if our instructor intentionally omitted Deep Learning because he wanted us to learn the fundamentals first, or if the syllabus had just not caught up with recent developments. There had of course been some impressive NN results by this time, but the hype was nowhere near as strong as it seems to be now.

During the year I supplemented my monthly stipend from SAP with a couple with tutoring work. Because of my internship experience, I was offered a place as the teaching assistant for the Large Scale Data Analytics (graduate level) course, where I ran the labs and graded assignments (e.g. on MapReduce and Spark programming).

I also did a lot of 1:1 private tutoring in math and programming (which was generally less enjoyable than being a teaching assistant). My clients were generally clueless. They presumed that I could teach them enough at the last minute to get them to the 40% required to pass their exams. There was also the admin overhead coordinating and traveling to sessions. As a result it was often painful, but I did get satisfaction from helping to get these stragglers over the line. After all, that had been me two years earlier, scrambling at the last minute to avoid failing my exams.

By this time I was constantly reading. At one point I was lugging The Rise and Fall of The Third Reich around between classes, squeezing in some reading time in the breaks between lectures. I read a bunch of books about Artificial Intelligence and the Philosophy of mind, and decided I wanted to wanted to do academic research in this field. I considered starting a PhD program, but without any means to support myself financially, I would have had to live with my parents and stay on at the University of Galway. Overall UoG is a decent university, but it’s a bit of a backwater when it comes to Computer Science research.

I decided that a masters in AI would be a decent compromise. I would still need to take out student loans to support myself, but this was much much less burdensome for a one-year program and as a result I would be able to study overseas. I finished the second semester strong and ended up with a final overall grade of 87%, which placed me first in my class. I was going to apologize for this shameless brag, but this is my website after all.. so I wont.

Before the semester ended I applied to three of the best AI schools in the United Kingdom: the University of Cambridge, the University of Edinburgh and Imperial College London. I was rejected by Cambridge but got accepted by Edinburgh and Imperial. In the end I chose the Edinburgh (mainly because living in London would have been more expensive).

As the program didn’t start until September, I was able to spend a great summer traveling around Europe. Most of my CS friends had started working as software engineers by this point and were suddenly flush with cash. It was an exciting time. We were all living in Galway: socializing after work, going to music gigs every week and planning lots of group holidays. Before I knew it, September had rolled around.

Masters in AI

When I applied to the masters, I didn’t know that I would be giving up all of this fun to move overseas alone, weighed down by high interest student loans and tackling an intense academic workload. From the get go, the program was challenging. For someone who was a lifelong underachiever, finishing top of my university class had given me an inflated self-confidence and illusions of grandeur, which were quickly dashed when I got to the University of Edinburgh (UoE).

Unlike my undergraduate university, UoE had a serious research oriented informatics department. Most of my classmates had been CS undergrads at UoE, and they were ahead of me on much of the course material. Those who were not UoE graduates were typically older and returning to graduate studies with fresh resolve after working in industry. These people seemed to know why they were here, and for some reason it felt like I did not.

The week before the term started, I used facebook to find a room in a flatshare with a group of guys from Northern Ireland. My new flatmates were in the peak of their undergraduate lives: waking up at midday, eating frozen food and watching Peep Show on repeat. I felt like I was transported back to my own earlier college years, which was strangely comforting. Unfortunately, this meant I was spending more time at home watching netflix and playing Fifa, and less time going to lectures.

Out of the courses I took, Iain Murray’s MLPR stands out. It was exceptionally challenging but as a result, rewarding. This course focused on mathematical foundations and statistical learning process, with an emphasis on probabilistic algorithms in the second half of the semester. I also really enjoyed my elective courses on Distributed Systems and Blockchains in the first Semester.

I didn't, however, enjoy the course dedicated to Deep Learning. I found it difficult to intuit the mechanisms of how the various Neural Network architectures worked, and it seemed like all we were really doing was tweaking minor parameters, and blindly re-training over and over, until the cost function moved in the right direction. I didn't like the blackbox nature and arbitrary seeming architectures of Neural Networks, which meant I found it difficult to truly get excited about Deep Learning.

I had had grand ideas that I would be learning about the mechanisms of intelligence and conciousness, but I was starting to feel more like a monkey toying with something I couldn't understand. As it turned out AI was not special, it is just statistics. When it came to forming groups for our capstone research project, pretty much everybody wanted to work exclusively on Deep Learning applications. When so many other people were interested in this stuff, why wasn't I?

Whilst I was struggling with Gaussian process kernels, my friends were all back in Ireland enjoying life. Relegated to being a distant snapchat spectator, I would watch in envy as they enjoyed newly found freedoms, having finished university for good. I found myself flying back to Ireland frequently; jumping at any opportunity to escape from what was starting to feel my own pit of academic despair.

When I wasn’t back in Ireland, I was battling procrastination demons daily. Through sheer resolve (and thanks to my well honed cramming skills) I was able to make it through the first semester exams with an A in every semester 1 exam. But this effort took an huge toll on me.

Shortly into the second semester, I went on a Ski trip to Bulgaria with a big group of friends. It was up there with one of my favorite holidays ever; we really had a blast. At the end of the trip, when it was time to fly home, my flight (back to Edinburgh) was 10 hours after everybody else's flight to Ireland. I have vivid memories of loitering in the airport, dreading my return to the cold, dark and wet conditions that constitute winter in Edinburgh.

Emboldened by the unforgiving schedule and dissillusions of why I was studying AI, my procrastination demons began to gain ground. My daily reading and exercise habits evaporated, and soon I had reverted to my old ways of doing the bare minimum: low attendance, eating poorly, drinking too much and sleeping til midday.

I decided it was time for the second course correction on my eventual path to Software Engineering: I dropped out of the AI masters. It was not an easy decision and I felt like a complete failure for a couple of weeks, but looking back I don’t think I would change my decision if I could (or at least that’s how I’ve rationalized it lol). I like being an engineer rather than a data scientist. But, I do wonder if I would still be in research/academia today if I had completed the AI masters.

Irish Goodbye

With the decision to drop out made, I said an Irish goodbye to Edinburgh and eloped back to Ireland. It felt like a huge weight had been lifted off my shoulders. I eventually got back on my feet and started to feel better mentally and physically. Moving in with friends in Galway really helped. A couple of weeks I started working as a Software Engineer at Galway’s hottest “AI” startup, Altocloud. Which was acquired by Genesys as soon as I joined lol.

Experiencing this sense of failure, but coming out of okay the other side, was an important life lesson for me. I know now that what might seem like a disastrous setback in the moment can work out better in the long run. Internalising this has helped to give me the confidence to believe that I can always land on my feet, no matter what short-term obstacles fall into my path.

This would be a good place to add some words of wisdom or cliche sayings like: “when one door closes another one opens”.. but I do have some self-respect :)