out to somebody on LinkedIn I’d by no means met, asking what he thought was the very best path into information engineering, and whether or not certifications had been even value pursuing. Seems he’d already learn considered one of my articles. So as an alternative of giving me a contemporary reply, he handed me again my very own. “You’re on the precise path,” he stated. “You’re following the 12 month roadmap, keep on with it. Don’t take recommendations from anybody, it would confuse you.”
I learn that message twice. As a result of two weeks earlier than that, I had almost talked myself out of the very roadmap I’d publicly dedicated to.
Let me again up.
It’s been a month since I printed my first article on this journey, “From Knowledge Analyst to Knowledge Engineer: My 12-Month Self-Research Roadmap.” Since then I’ve written three extra, each strolling by one thing I constructed. An ETL pipeline from scratch. That very same pipeline made manufacturing prepared with SQLite and idempotency. Then scheduled with GitHub Actions, which become a lesson on portability earlier than it turned a lesson on scheduling.
These 4 articles are a good report of what I did this month. They aren’t a good report of what the month truly felt like. So this one is completely different. No code, no walkthroughs. Simply the elements that didn’t match anyplace else.
The plan stated one factor, the month did one other
The roadmap laid issues out so as. SQL first, then Python, then Git, then Spark, then Airflow, then Databricks. Tidy. Sequential. The type of plan that appears nice on a Notion board.
I didn’t observe it. Probably not. As a substitute of shifting by the stack so as, I constructed one small pipeline and saved pushing on it till it broke in new methods. SQL confirmed up. Python confirmed up. Git confirmed up. However not within the order I deliberate, and never as a result of I scheduled it that manner. They confirmed up as a result of the pipeline demanded them.
I used to assume that meant I used to be off monitor. Now I believe it simply means the plan was a place to begin, not a contract. The roadmap obtained me shifting. The pipeline determined what I truly wanted to be taught subsequent.
The partitions had been by no means actually concerning the instruments
When you learn the construct articles, there’s a sample I didn’t name out on the time. Each wall I hit had a technical repair and a non-technical lesson sitting beneath it.
- Idempotency wasn’t actually about SQLite. It was about studying to mistrust my very own assumption that “it labored as soon as” meant “it would maintain working.”
- Persistence wasn’t actually about Google Drive. It was about realizing my work might vanish the second I closed a tab, and that I’d already misplaced a file as soon as earlier than with out noticing how shut I’d come to shedding extra.
- Portability wasn’t actually about GitHub Actions. It was about discovering that one hardcoded path had quietly made my entire pipeline depending on a single setting, with out me ever deciding that on objective.
None of these are coding classes. They’re considering classes that occurred to floor by code. That seems like the true form of knowledge engineering to date, no less than from the place I’m standing. The instruments are how the teachings present up. They’re not the teachings themselves.
The half the construct articles didn’t present
Right here’s what they didn’t present. Someplace in the course of this month, the vitality that got here from individuals watching began to fade. Not dramatically. It simply obtained quieter. I finished feeling the stress I wrote about in article two, the one the place strangers reached out excited to observe alongside. That stress didn’t disappear as a result of I finished caring. It light as a result of the objective on the finish of it, touchdown a excessive paying information engineering function, began to really feel far-off. Far sufficient that it stopped being motivating and began being exhausting to even take into consideration.
It didn’t assist that my precise job obtained busier, and that I needed to be taught Laravel for work on high of the whole lot else. Three to 4 hours a day sounds cheap till you’re drained out of your day job and the factor ready for you at house is one other unfamiliar stack.
There have been stretches the place I critically thought of slowing down. What saved me on the desk wasn’t the imaginative and prescient of the job anymore. It was smaller than that. I saved desirous about the one individual on the market who’s precisely the place I used to be in Might, caught someplace between analytics and engineering with no clear map, who is perhaps ready on the following article. Serving to that one individual turned extra motivating than the summary concept of my very own future function. I additionally made a degree of in search of one thing genuinely difficult to fill my free time with, as an alternative of the simple, senseless stuff that fills time with out leaving something behind. Even on the times the job felt far-off, no less than I wasn’t losing the hours.
Nearly getting pulled off target
I additionally should be sincere about one thing I known as out earlier than I even began this journey. In my first article I admitted to having shiny object syndrome, leaping between design, animation, advertising, IT, and now information, earlier than any of them obtained the possibility to stay. I stated I’d must be intentional to keep away from that occuring right here too.
It occurred anyway. Partway by the month I fell right into a rabbit gap about certifications, whether or not I wanted one, which one, whether or not I used to be even on the precise path in any respect. That spiraled into critically contemplating a pivot towards AI engineering as an alternative, as a result of within the second it appeared shinier and extra in demand. I needed to consciously cease and remind myself of the trail I’d already chosen, and already advised an viewers I used to be dedicated to.
Which brings me again to that message.
The stranger who handed me my very own recommendation
That LinkedIn message I discussed in the beginning got here proper after that wobble. I’d reached out to somebody for recommendation on the very best path into information engineering, half in search of permission to second guess myself, half questioning out loud whether or not a certification would assist. As a substitute he advised me he’d already learn my roadmap article, and that the reply to my query was sitting in my very own writing. Persist with the plan. Cease amassing opinions from strangers, as a result of too many opinions simply confuse you.
There’s one thing virtually humorous about being talked again into your individual plan by somebody who solely knew about that plan as a result of I’d printed it. However that’s the entire level of doing this in public. The accountability doesn’t solely come from individuals watching you construct. Typically it comes again round and palms you your individual phrases at precisely the second you want them.
What I truly discovered about how I be taught
The opposite realization this month had nothing to do with motivation and the whole lot to do with how I work. I used to assume I wanted to construct one thing large to really be taught something. An actual undertaking, a number of weeks, one thing portfolio worthy from the beginning. That intuition almost killed my momentum greater than as soon as, as a result of large initiatives are straightforward to begin and simple to desert midway by.
What’s labored as an alternative is small. A pipeline I can construct and break in a weekend teaches me simply as a lot as a sprawling multi week undertaking, and I truly end it. Going into month two, I’m buying and selling the concept of 1 large construct for a string of small ones. Mini initiatives, sized to suit round a 9 to five and a Laravel deadline, each geared toward a selected idea as an alternative of an entire resume line.
The place that leaves me
When you’d requested me on day one what success appeared like, I might have stated the excessive paying function, full cease. A month in, that’s nonetheless someplace within the image, however it’s not what’s truly getting me to open my laptop computer after a protracted day at work anymore.
What’s getting me there now could be smaller, and truthfully extra sustainable. I like constructing issues. I like writing about what breaks. And each time somebody tells me an article truly helped them determine their very own subsequent step, that does extra for me than the considered some future job provide ever has.
The roadmap remains to be the roadmap. I’m nonetheless following it, cron job and all. However I’m beginning month two with a clearer sense of what’s truly going to hold me by the following eleven, and it’s not the factor I assumed it might be after I wrote that first article.
That is a part of my ongoing sequence documenting my transition from programs analyst to information engineer. When you’ve been following alongside, thanks. If that is your first article within the sequence, the sooner ones are linked above.
