Hi everyone
Introduction: I wanted to share my experience and what motivated me to build my own web scraping service. A few years ago, I had an idea to create an interactive, unbiased news application for both mobile and web. The goal was to aggregate news articles and topics from multiple sources and present them in a neutral, structured way. Since I already had experience in IT and software development, I decided to build it myself.
So I began working on the app seriously designing the UI, logo, and overall user experience with care. When I moved on to the backend, I quickly ran into a major challenge, sourcing the news content. I initially planned to use Google News RSS feeds and its related APIs, only to discover that Google had retired the Google News API. This was disappointing and forced me to look for alternatives. I explored several options, including newsapi.org, which worked technically but cost $449/monthly, not at all affordable for an small or independent project. I continued searching for a long time, but most available solutions were bundled web scraping APIs with pricing that was far beyond my budget. With no viable alternative, I decided to build my own news scraping API. This meant pausing development of the news app itself, but there was no other practical option. I spent considerable time reading blogs and Reddit posts, trying to understand the most efficient and reliable way to scrape news content and structure it properly. Eventually, I found an approach that worked and started implementing it. (The blog I learned from is no longer available otherwise, I would have shared it here.) Once the news scraping service was complete, I built a backend around it and hosted the API on RapidAPI. I decided to make the service publicly accessible because I did not want others to face the same difficulties I experienced. Over time, and mostly during my spare time, I also built additional APIs, including Google Images API, Google Trends API, Stock Market Data API, YouTube Data API, and a few others. All of these APIs are hosted securely on RapidAPI. Pricing starts at $6.99, which is mainly to cover server and infrastructure costs. If you are interested, you can find all the available APIs curated on my website: https://cognifyapi.com/ I appreciate you taking the time to read this, and I am open to feedback or questions.