During my time in college, I always wanted to become an engineer. I was hardworking and a fast learner but I lacked financial resources as I didn’t belong to a well-off family. Both my parents worked hard to put me into a well-reputed university to pursue my dreams. However, I wasn’t sure about the type of engineer I wanted to become. It was tough to choose between a software engineer and an electrical engineer. After much consultation with my elders, I opted to start my higher education as a software engineer.
A year ago, I was enjoying my time in becoming a software engineer but at the same time, my problems were growing at an alarming rate. Due to their age, my parents were unable to work hard enough to meet my education expenses. This was when I decided not to become a burden on them anymore and started finding an income.
I wasn’t happy working as a waiter so I hopped on the internet to search for other options. That was when I came across an article that cleared my questions about headless browsers, web scraping, and how big data is important for big companies.
Getting to work
At that time, I had no money to get a pre-built web scraper that could help me scrape data from websites. All I had was a normal laptop, a slow internet, elementary knowledge about programming (related to building a personal web scraper), and the will to make things comfortable for my family.
I got to work the next day after classes and it took me 2 months to get a properly functioning web scraper that was able to crawl through the internet and store data related to Amazon and Google reviews. Deciding the perfect headless browser was the most difficult task so I decided to try all of them and see how my web scraper worked. To my surprise, the scraper worked flawlessly on headless chrome. Even though it was late, I still decided to give the web scraper a proper try and start scraping.
To my surprise, my laptop hanged as soon as I launched the self-built web scraper but luckily, a quick restart fixed it. I waited for some time before crossing my fingers and giving the web scraper a retry. My happiness was unmeasurable as I saw my hand-built web scraper crawl using a headless browser to flawlessly crawl through the internet to scrap reviews.
It only gets better from here
On my first try, I was able to save credible reviews on multiple products, services, and locations. After that, I made a habit of running the web scraper for 2 hours every night so that I could get my hands on vital information and within a month I was able to scrape almost 2 GB of reviews from numerous sources over the internet. It was later that night that I read another article where the author had talked about people hiring freelancers to scrape data for them.
Immediately I made my account on Upwork and started my freelancing career. For a month, there was no response from any of the proposals I was sending. It even started to get to me as things started to go very bad at home as my mother got sick and wasn’t able to work. Putting the burden on my father. Despite everything, I tried my best to remain calm and stay focused on my work.
2 months later, I finally got my first job as a freelancer. My employer wanted me to scrape data related to beverages. I quickly configured my web scraper and got to work. He gave me 3 days to return 100 credible results, it took me a day and I returned 115 credible results. My employer was impressed and gave me a five-star review and I made more than what my parents were able to earn in a month.
After that, the work kept on coming and it wasn’t soon when I felt the need to upgrade my web-scraper. I was able to buy a dedicated powerful machine from the freelancing revenues and built an even more capable and powerful web scraper that was able to crawl through multiple headless browsers simultaneously. At this time, I fell deep into the world of programming to build the best web scraper in the world.
Big achievements
I was able to achieve my goal very quickly as I worked hard, by myself for myself. My Upwork profile had 100+ five-star reviews and I was ranking near the best freelancers of my time. Our financial position had greatly improved as I was earning in USD. My parents no longer needed to work as I was able to manage everything. After I graduated, I got an offer regarding my web scraper. A big data company wanted to license it for more than a year and was paying handsomely for the service.
I built my own company to have my web scraper copyrighted and started giving other companies a license to use my web scraper. After this, I never wanted to stop building my web scraper. I decided to give regular updates to it and further improve its capabilities as more companies decided to license my web scraper.
Conclusion
Money became the motivator for me to build my career in coding web scrapers, as I needed money to support my family and complete my education to become the best web scraper in the world.