Blog

Web Scraper and Patterns
Extending Web Scraper Cloud with Patterns, a data app platform

October 25, 2022

Web Scraper Cloud, Patterns, data visualisation

Below, we’ll describe what Patterns is and walk through the set up for an app in Patterns that uses Web Scraper Cloud to acquire data and scan for product price differences on e-commerce websites. This app is a simple boilerplate template that can be extended to provide a more advances functionality. 

Continue Reading

Web Scraper 0.6.5 release

September 09, 2022

web scraping, release

For the last months, we have been working towards the Web Scraper 0.6.5 release and finally, it is here!

Continue Reading

A quick guide to CSS and jQuery selectors image
A Quick Guide to CSS and jQuery Selectors for Web Scraper

June 28, 2022

JQuery selectors, web scraper, Scraping public data, scraper, CSS selectors

Selectors when it comes to CSS and jQuery refer to code you can use to interact with specific elements in the DOM. They are particularly useful when you want to style or act upon certain elements of the document without altering other elements. Through a careful use of selectors, you can manipulate a page exactly as intended or isolate the specific data you’re looking to scrape.

Continue Reading

data analysis using amazon athena
Data Analysis using Amazon Athena

May 02, 2022

data analysis, web scraper, amazon athena

Amazon Athena lets you use SQL to interactively query any data you have within an Amazon S3 Bucket. With Web Scraper Cloud, you can automatically sync web scraping job results to an Amazon S3 Bucket where you can then use Athena to quickly query this data with little additional overhead. In this article, we’re going to guide you through syncing Web Scraper Cloud with an S3 bucket and using Athena to generate and perform calculations on table data. While we’re covering a specific use case, you can adapt this same process to any other data set to gather insights.

Continue Reading

Headless Scraping with Cheerio

March 21, 2022

web scraper, cheerio, web scraping, headless

Cheerio is a node package that allows you to easily parse and extract elements from markup. Unlike a browser, Cheerio doesn’t produce a visual rendering, load external resources, execute Javascript code, or apply CSS. As a result, it’s much faster than other solutions when it comes to scraping web pages for valuable data.

In this article, we’re going to specifically cover how to set up a headless scraper with Cheerio and spoof browser headers to access a wider range of web pages.

Continue Reading

Scraping public data
Scraping public data. Is it legal?

March 03, 2022

web scraper, GDPR, data extraction, linkedin, Scraping public data, web scraping, facebook

We’ve all been hearing about the ongoing cases of companies failing lawsuits against web scraping activities. Worth mentioning are the cases of LinkedIn vs. HiQ on public data scraping and the case of Facebook Inc. vs. BrandTotal Ltd. on scraping data from Facebook. What is more interesting is that in one case it was ruled in favour of the data scraper, but in the other - of the company whose platform is being scraped. Seeing such cases with different outcomes raises questions, especially due to the recent General Data Protection Regulation (GDPR), on how lawful web scraping of publicly available data actually is? 

Continue Reading

Web-Scraper-0.6.0-release-blog-image
Web Scraper 0.6.0 - Pagination Selector and More!

September 01, 2021

Feature, Web Scraper Cloud, Release, Tutorial, Pagination

The moment has come that we are ready to release the most requested feature of all time! Creating pagination correctly has been one of the most frequently reported problems; however, that is about to change. The pagination selector is here and so are other additional updates that will make scraping with Web Scraper easier.

Continue Reading

web-scraping-data-science-questions-answered-blog-post-image
Most Frequently Asked Questions Answered.

August 02, 2021

Guide

For discovery and greater understanding, questions are a must. Therefore, we have collected the most frequently asked questions by the public about different web scraping topics.

Continue Reading

Google-Append-Cryptocurrency-Crypto-Visualization-Web-Scraper-Cloud-Feature
Automatized Crypto Visualization with the New Web Scraper Cloud Data Export Feature

June 14, 2021

Google, Google Sheets, Feature, Web Scraper Cloud, Visualization, Release

Most people could agree that observing a picture, a simple graph, or a pie chart is easier and more efficient than scrolling through blocks of text or trying to parse out useful insights from looking at numerous columns in a spreadsheet. This common knowledge is the basis of why visualization is so important.

Also, in a rapidly changing world, people seek ways of becoming more efficient, ways of increasing the time spent focusing on the insights, rather than pouring endless hours and effort into collecting the necessary assets to gain those end results that determine so much. This is why automatization nowadays is a key topic.

Continue Reading

Scraping-Customer-Reviews-Blog-Image
Why Should You Scrape Customer Reviews?

June 01, 2021

Analysis, Data

Just like a vehicle tire leaves an imprint in the mud which can be washed away only by a great storm or rainfall, an impression leaves an impact on a product which then becomes the formatting foundation of an opinion that can not be altered easily. For this simple reason, creating an interesting, positive, and captivating impression on your customers will go a long way for your business's success and overall reputation.

Continue Reading

Brief-History-Of-Web-Scraping-Web-Scraper-Blog-Image
Brief History of Web Scraping

May 14, 2021

Data, web scraping

Web scraping is becoming a more widely known term. Most associate it with web data extraction, the most efficient and the simplest way of copying large chunks of information online; however, did you know that web scraping was born for a completely different purpose and it took almost two decades for it to transform into web scraping we are familiar with now?

Continue Reading

Data-Quality-Control-And-Notification-Feature-Release-Web-Scraper-Blog-Image
Web Scraper Cloud Data Quality and Notification Feature

April 30, 2021

Feature, Web Scraper Cloud, Release, Update

With constant development process and continuous growth, we are happy to have released a new notification system as well as a data quality feature to allow further automatization optionality.

Continue Reading

Ultimate-Web-Scraping-Guide-Web-Scraper-Blog-Image
The Ultimate Web Scraping Guide.

March 15, 2021

web scraping

It is no lie that data is power in many ways. For different reasons and applications, different information available online can be used for gaining an advantage in various spheres of life, especially in business.

Continue Reading

Shopify-Webscraper-fast-tracking-your-online-business
Fast-track your online Shopify business

February 17, 2021

Data, E-commerce, web scraping

Nowadays, with almost everything being available online - many are looking for ways to start a new eCommerce business or automate an existing one. 

Imagine a fast-track to improve the efficiency of your new, already existing online store. CSV export from Web Scraper and import to Shopify can get you updating your product lists in only a few minutes. 

Continue Reading

Scrape-at-scale-Web-Scraper-Cloud
Extracting Data at Scale Using Web Scraper Cloud.

November 24, 2020

Big Data, Data, Web Scraper Cloud

Data nowadays can be the driving fuel of a company. With the huge increase in technology and data, it has become more important than ever to retrieve, transform, and store that data correctly and effectively.

Continue Reading

CSS-Selectors-That-You-Need-To-Know-Web-Scraper-Blog
Top 5 CSS Selectors You Need to Know.

November 06, 2020

Data, web scraping, Tutorial

Scraping might get hard at times when you're dealing with website structures that frequently change or in general, are hard to scrape with just the point-and-click interface. No need to stress, with the knowledge of the CSS selector - any website structure can be overcome and any website - scraped. Here are the TOP 5 CSS selectors that we frequently use and might be of great use to you.

Continue Reading

Scraping-Ecommerce-Fastest-Way-Web-Scraper-Blog
Scraping E-commerce the Fastest Way

September 29, 2020

Data, E-commerce, Tutorial

In the previous blogs, we have gone through the classical way of scraping and exposed a tip of how to potentially prevent sitemaps from breaking when an e-commerce site changes product placements in designated categories. Now we are going to take a look at another scraping method which is - scraping with the “Sitemap.xml Links” selector.

Continue Reading

Scraping-Through-Brands-E-Commerce-Web-Scraper-Blog
Scraping E-commerce through Brands

September 24, 2020

Data, E-commerce, Tutorial

In the previous blog, we explained how to retrieve the necessary data the classical way by going through the categories, sub-categories, then the products, etc. It is the most primitive and intuitive way of gathering data with Web Scraper extension; however, sometimes a problem arises when the website layout is changed or the placement of various products is altered. For this reason. previously created sitemaps for that specific website may break, or stop working properly. 

Continue Reading

Scraping-Ecommerce-Classical-Way-Blog-Image-Web-Scraper-Blog
Scraping E-commerce the Classical Way

September 16, 2020

Data, E-commerce, Tutorial

With the enormous growth and development in technology, data being the main driver of modern and fast-growing companies, online business has evolved over the recent years. However, it comes as no surprise since ordering, reserving goods, and services online while not leaving the house is a huge time saver and accessible for everyone with a stable internet connection.

Continue Reading

Data-Transformation-Open-Refine-Web-Scraper-Blog-Image
Data Transformation with OpenRefine

August 27, 2020

Data, Data transformation, Tutorial

With cleaner data, we can begin to transform it. Data transformation can manifest in different forms. It can be clustering, merging, adding information, replacing strings, and so on. OpenRefine covers them all. 

Continue Reading

Web-Scraper-0.5.0-Release-Blog-Image
Web Scraper 0.5.0 Release

August 21, 2020

Release, Update

We are happy to announce that Web Scraper 0.5.0 has been released! This release contains new features such as a new data selection UI engine, a new page load detection system, a welcome page, and a whole lot more!

Continue Reading

Data-Transformation-With-Open-Refine-Web-Scraper-Blog
Data Cleaning with OpenRefine

August 14, 2020

OpenRefine, Data, Data transformation, Tutorial

Data transformation is a step for preparing data sets for AI training and analysis. Clean and transformed data is a vital part of precise and correct data reports and analysis. In this blog series part, we will look at what importance it is to detect and delete blank, inconsistent and duplicate data entries.

Continue Reading

Data-Transformation-Open-Refine-Web-Scraper-Blog-Image
Introduction to OpenRefine

July 28, 2020

Data, Data transformation

As the saying goes “garbage in, garbage out”. This can be associated with the idea of data analysis without data transformation beforehand. With bad, messy data, only lousy, chaotic analysis can be done. However, with data transformation, not only the process of analysis becomes easier but also the precision increases.

Continue Reading

Data-Transformation-Web-Scraper-Blog
Data Transformation with Web Scraper.

July 06, 2020

Data, Data post processing, Parser

Have you ever been in a situation when you have two or more data sets of completely different structures? So different that it is impossible to analyze, manage, or integrate. It sounds like every professional’s worst nightmare.

Continue Reading

Web-Scraper-Collaboration-Blog
In Need of Data?

June 05, 2020

Data, collaboration

If you are a journalist or a data enthusiast and you are in need of data, look no further. We, Web Scraper, offer a collaboration.

Continue Reading

Data-Extraction-For-Business-Web-Scraper-Blog
3 Ways In Which Web Scraping Can Improve Your Business And Save You Time.

May 29, 2020

Data, web scraping

Nowadays, with increasing digitalization, where new technologies are constantly arising and computational power is available almost to everyone, the necessity for data has escalated drastically. 

Continue Reading

The Instrumental Role Of Big Data Web Scraper Blog
The Instrumental Role of Big Data in Our Lives

May 12, 2020

Big Data, Data, Global pandemic

Enormous amounts of data that traditional data-processing application software is not capable of dealing with, that is used for analyzing to reveal trends, patterns, associations of especially human behavior and interactions. The subject of Big Data and the usage has increased highly over recent years. With the immense development of technology and the large sets of data that are processed every day, the questions of how to take advantage of it is on everyone’s mind, regardless of the industry.

Continue Reading

Data-Visualization-With-Google-Data-Studio-Web-Scraper-Blog
The Importance of Data Visualization

April 29, 2020

As possibly one of the wisest investments in big data future, data visualization can be a crucial tool for any business. With the continuous advancements in technology and the huge increase in the necessity for big data, visual representation of it can display and showcase many factors of a business that would have been hard to do without data visualization. In this blog, we are gonna explore ways of data representation with the help of a highly easy and versatile tool.

Continue Reading

Google-Trend-Topic-Analysis-Blog-Image
Coronavirus Pandemic Google Trends Analysis

April 08, 2020

Analysis, Google trends

Of everything going on, we took interest in what people are most searching for on google search engine while the coronavirus outbreak.

Continue Reading

Sports-Arbitrage-Web-Scraper-IO-Blog-Post
Is Sports Arbitrage Betting Real?

March 09, 2020

Sports Arbitrage Betting

Imagine a scenario where you invest an amount of money and there is a guaranteed profit! Hard? Impossible? Not at all! Let us provide you with a mathematically proven strategy, which, when applied correctly, actually provides such possibilities.

Continue Reading

The Ultimate Tool for Big Data Access Using Google Sheets

February 03, 2020

Google Sheets, Query Function

Imagine learning a function, which can work as various other functions; therefore, replacing them with only one and making work with big data easier. With the query function of Google Sheets that scenario is not that absurd because it provides exactly what described. Accessing, summarizing, filtering data from various spreadsheets with only one function - thanks to Google Sheets, the opportunity to do exactly that has been around for quite some time already.

Continue Reading

Web Scraper Cloud Parser feature release

December 23, 2019

Web Scraper Cloud, Data post processing, Parser

We are happy to finally introduce a Parser feature for Web Scraper Cloud.

Usually, to post process data, a custom written script or extra time editing the data manually in a spreadsheet software would be the case; however, the Parser takes care and eases this process.

Continue Reading

How to find all the local restaurants in Yellow Pages using Web Scraper

September 02, 2019

Yellow Pages, Tutorial

Have you ever planned a trip to a different state, city or even a country but did not know any places to have a delicious dinner at? We all know that Yellow Pages is the go-to site if we want to find restaurants in specific areas but it can often be frustrating to get boggled up with bottomless pages on it. Luckily, Web Scraper is here to solve this by allowing you to extract all the information you need from Yellow Pages. So you can enjoy great meals during your trip.

Continue Reading

Web Scraper 0.4.0 release

April 17, 2019

Sitemap.xml, Release

We are happy to announce that Web Scraper 0.4.0 has been released. This release contains a new selector, updates to other selectors and improved CSS selector generator. Starting from version 0.4.0 Web Scraper is also available in Firefox.

Continue Reading