Photo by Bruno Nascimento on Unsplash

In this tutorial, we will scrape product details by following links using the Scrapy-Splash plugin.

Create a virtual environment to avoid package conflicts. Install necessary packages and start scrapy project.

Install scrapy:

pip install Scrapy 

If you have trouble with installing Scrapy through pip, you can use conda. See docs here.

conda install -c conda-forge scrapy

Start the project with:

scrapy startproject productscraper
cd productscraper

Also, install scrapy-splash as we will use it further in the tutorial. I assume you already have Docker installed on your device. Otherwise, go ahead and install it first. …


Apache beam is the best way to automate the Reading-Transforming-Writing process to make a robust pipeline.

For my work project, I needed to do read data from a collection and write it to another collection with a transform. It is easy as there is an official MongoDB IO reader and writer module. You can check out the mongodbiomodule here.

This is an example usage of how you read data from MongoDB.

pipeline | ReadFromMongoDB(uri='mongodb://localhost:27017',
db='testdb',
coll='input')

This is an example usage of how you write data to MongoDB.

pipeline | WriteToMongoDB(uri='mongodb://localhost:27017',
db='testdb',
coll='output',
batch_size=10)

Note: When writing and reading data…


2019'dan beri öğrencisi olduğum 42 ile ilgili çok soru alıyorum. Yakin zamanda, Türkiye’de de 42 okulu açılacağını öğrendim. Merak edenler için biraz okulu anlatmak istiyorum.

Öncelikle 42 Paris, Fransada bir kaç milyarderin, kar amacı gütmeden tamamen ücretsiz olarak kurduğu bir okuldur. İlk başta Fransadaki iyi yazilimci açığını doldurmak için ortaya çıkmıştır. Daha sonra finansal desteklerle bir çok ülkede 42 Yazılım okulu açılmıştır ve aslında yeni bir okul tipine de öncülük etmektedir. Çünkü 42 tamamen ücretsizdir, Silicon Valley kampüsü gibi bir çok kampüsünde ayrıca yurtlar da ücretsizdir. (Amerika için devrimsel bir durum diyebilirim.) …


As a developer, you may find yourself wishing to gather, organize, and clean data. You need a scraper to extract data and a crawler to automatically search for pages to scrape.

Scrapy helps you complete both easy and complex data extractions. It has a built-in mechanism to create a robust crawler.

In this article, we’ll learn more about crawling and the Scrapy tool, then integrate Scrapy with Django to scrape and export product details from a retail website. To follow this tutorial, you should have basic Python and Django knowledge and have Django installed and operating.

Selectors

Scraping basically makes a…


If the only way you access the Internet is through your browser, you are missing out on a huge range of possibilities.

Photo by Nathan Dumlao on Unsplash

If you search “Cheapest Flight to Istanbul” on Google, you will get some popular flight search website results as well as some advertisements. Google will only report what these websites say in their content. …


Photo by Andrew Neel on Unsplash

A modal is a popup window that is displayed in front of the current page when pressed a button.

Here’s my example, I have recently created a Twitter-like website using Django.


Photo by Caspar Camille Rubin on Unsplash

First of all, while browsing, I couldn’t find the most effective and short solution to find the most frequent element in an array but I only found a lot of long solutions with for loop however the best way is not using nested for loop. Because in Big O Notation, it’s a O(n2) complexity which means it is not effective.

The best way to find the most frequent element in an array is by using reduce function and creating a hashmap. And that’s the whole code you need:

function getMostFrequent(arr) {
const hashmap = arr.reduce( (acc, val) =>…


To Theo van Gogh. Arles, Monday, 9 or Tuesday, 10 July 1888.

My dear Theo,

Is life visible to us in its entirety, or before we die do we know of only one hemisphere?

Painters — to speak only of them — being dead and buried, speak to a following generation or to several following generations through their works. Is that all, or is there more, even? In the life of the painter, death may perhaps not be the most difficult thing.

For myself, I declare I don’t know anything about it. But the sight of the stars always makes…


A quick overview of what URL is consist of, how HTTP requests works, some important request headers, Cross-Origin Resource Sharing (CORS).

Photo by Alina Grubnyak on Unsplash

First of all, let's start with how a URL (Universal Resource Locator) looks like.

A sample URL consists of 4 parts.


Photo by Nate Grant on Unsplash

JavaScript is a single-threaded synchronous language. So that means that a function that takes a long time to run will cause a page to become unresponsive.

Open your dev tools, follow the steps with me, and we will demonstrate this. Let’s write a function called ‘hang’.

function hang(secs){
const doneAt = Date.now() + (secs * 1000)
while( doneAt > Date.now ) {// just do nothing }}
hang(50)

Now when we invoke the hang function, we will be stuck for 50 seconds. While this 50 seconds; it’s just going through that while loop, and it locks up the entire page.

And…

Bilge Demirkaya

Software Engineer at 42 Silicon Valley | Technical Writer at ContentLab.io

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store