Oxylabs Playwright Web Scraping Guide

Use page.route(url_pattern, handler) to intercept network requests.

• To abort requests: Pass a lambda or function that calls route.abort(). This is useful for blocking heavy assets like images to speed up scraping.
• To modify responses: Use route.fetch() to get the original response, then use route.fulfill() to provide a modified version. You can change the response body, status, or headers (e.g., changing content-type).

Example patterns for url_pattern:

• "**/*.{png,jpg,jpeg,svg}": Matches all files with these extensions.
• "**/*": Matches all requests.

from playwright.async_api import async_playwright
import asyncio
import requests

async def handle_route(route) -> None:
    response = await route.fetch()
    body = await response.text()
    body = body.replace("<title>", "<title>Modified Response")
    await route.fulfill(
        response=response,
        body=body,
        headers={**response.headers, "content-type": "text/html"},
    )
 
async def main():
   async with async_playwright() as pw:
       browser = await pw.chromium.launch(
          proxy={
              'server': "http://pr.oxylabs.io:7777",
              "username": "USERNAME",
              "password": "PASSWORD"
              },
          headless=False
      )
 
       page = await browser.new_page()
       # abort image loading
       await page.route("**/*.{png,jpg,jpeg,svg}", lambda route: route.abort())
       await page.route("**/*", handle_route)
       await page.goto('https://www.oxylabs.io')
       await page.wait_for_timeout(5000)
       await browser.close()
 
if __name__ == '__main__':
   asyncio.run(main())

Use page.route(url_pattern, handler) to intercept network requests in Node.js.

• To block assets: Use a regex like /(png|jpeg|jpg|svg)$/ to match specific file extensions and call route.abort() within the handler.
• To modify responses: Use an async handler that calls await route.fetch() to retrieve the response, modifies the body or headers, and then calls route.fulfill() with the updated data.

This technique can be used to improve scraping performance by preventing the loading of unnecessary resources.

const playwright = require("playwright")
(async() =>{
   const launchOptions = {
       headless: false,
       proxy: {
          server: "http://pr.oxylabs.io:7777",
          username: "USERNAME",
          password: "PASSWORD"
        }
     }
   const browser = await playwright["chromium"].launch(launchOptions)
   const context = await browser.newContext()
   const page = await context.newPage()
   await page.route(/(png|jpeg|jpg|svg)$/, route => route.abort())
   await page.route('**/*', async route => {
              const response = await route.fetch();
              let body = await response.text();
              body = body.replace('<title>', '<title>Modified Response: ');
              route.fulfill({
                      response,
                      body,
                      headers: {
                              ...response.headers(),
                              'content-type': 'text/html'
                      }
               })
   })
   await page.goto('https://oxylabs.io');
   await browser.close()
})

To use a proxy in Node.js, pass a launchOptions object to the launch() method. This object should contain a proxy property, which is itself an object containing server, username, and password keys.

Example configuration:

const launchOptions = {
    headless: false,
    proxy: {
       server: "http://pr.oxylabs.io:7777",
       username: "USERNAME",
       password: "PASSWORD"
     }
};
const browser = await playwright[browserType].launch(launchOptions);

const playwright = require("playwright")
 
(async() =>{ 
 for (const browserType of ['chromium', 'firefox',  'webkit']){ 
   const launchOptions = {
       headless: false,
       proxy: {
          server: "http://pr.oxylabs.io:7777",
          username: "USERNAME",
          password: "PASSWORD"
        }
     }
   const browser = await playwright[browserType].launch(launchOptions)
 }
})

To scrape text data in Node.js, use $$eval to iterate over a collection of elements and then use querySelector on each element to find specific data points (like titles or prices). Use the .innerText property to extract the text content.

const products = await page.$$eval('.a-spacing-base', all_products => {
    const data = []
    all_products.forEach(product => {
        const title = product.querySelector('span.a-size-base-plus').innerText
        const price = product.querySelector('span.a-price').innerText
        const rating = product.querySelector('span.a-icon-alt').innerText
        data.push({ title, price, rating})
    });
    return data
})

const playwright = require("playwright")
(async() =>{ 
for (const browserType of ['chromium', 'firefox',  'webkit']){ 
   const launchOptions = {
       headless: false,
       proxy: {
          server: "http://pr.oxylabs.io:7777",
          username: "USERNAME",
          password: "PASSWORD"
        }
     }
   const browser = await playwright[browserType].launch(launchOptions)
   const context = await browser.newContext()
   const page = await context.newPage()
   await page.goto('https://www.amazon.com/b?node=17938598011');
   const products = await page.$$eval('.a-spacing-base', all_products => {
       const data = []
       all_products.forEach(product => {
           const title = product.querySelector('span.a-size-base-plus').innerText
           const price = product.querySelector('span.a-price').innerText
           const rating = product.querySelector('span.a-icon-alt').innerText
           data.push({ title, price, rating})
       });
       return data
   })
   console.log(products)
   await browser.close()
   }
})

In Python, you can scrape images by using page.query_selector_all('img') to retrieve all image elements. Iterate through the elements, extract the src attribute using get_attribute("src"), and use the requests library to download the content and write it to a file using standard file I/O (open(..., "wb")).

Configure the Oxylabs proxy within the pw.chromium.launch method using a dictionary containing server, username, and password.

from playwright.async_api import async_playwright
import asyncio
import requests
 
 
async def main():
   async with async_playwright() as pw:
       browser = await pw.chromium.launch(
          proxy={
              'server': "http://pr.oxylabs.io:7777",
              "username": "USERNAME",
              "password": "PASSWORD"
              },
          headless=False
      )
 
       page = await browser.new_page()
       await page.goto('https://www.oxylabs.io')
       await page.wait_for_timeout(5000)
 
       all_images = await page.query_selector_all('img')
       images = []
       for i, img in enumerate(all_images):
           image_url = await img.get_attribute("src")
           content = requests.get(image_url).content
           with open("image_{}.svg".format(i), "wb") as f:
               f.write(content)
           images.append(image_url)
       print(images)
       await browser.close()
 
if __name__ == '__main__':
   asyncio.run(main())

To scrape images using Playwright in Node.js, use page.$$eval to select all img elements and extract their href or src attributes. You can then use the built-in https and fs modules to download the image data and write it to the local file system.

When using Oxylabs proxies, ensure you pass the proxy configuration (including server, username, and password) into the launchOptions of the browser instance.

const playwright = require("playwright")
const https = require('https')
const fs = require('fs')

(async() =>{
   const launchOptions = {
       headless: false,
       proxy: {
          server: "http://pr.oxylabs.io:7777",
          username: "USERNAME",
          password: "PASSWORD"
        }
     }
   const browser = await playwright["chromium"].launch(launchOptions)
   const context = await browser.newContext()
   const page = await context.newPage()
   await page.goto('https://oxylabs.io');
   const images = await page.$$eval('img', all_images => {
       const image_links = []
       all_images.forEach((image, index) => {
          const path = `image_${index}.svg`
          const file = fs.createWriteStream(path)
          https.get(image.href, function(response) {
               response.pipe(file);
          })
          image_links.push(image.href) 
       })
       return image_links
   })
   console.log(images)
   await browser.close()
})

In Python, it is recommended to use query_selector_all to retrieve a list of elements and then iterate through them using a loop. For each element, use query_selector to find child elements and inner_text() to extract the text.

all_products = await page.query_selector_all('.a-spacing-base')
data = []
for product in all_products:
    result = dict()
    title_el = await product.query_selector('span.a-size-base-plus')
    result['title'] = await title_el.inner_text()
    price_el = await product.query_selector('span.a-price')
    result['price'] = await price_el.inner_text()
    rating_el = await product.query_selector('span.a-icon-alt')
    result['rating'] = await rating_el.inner_text()
    data.append(result)

from playwright.async_api import async_playwright
import asyncio
 
 
async def main():
   async with async_playwright() as pw:
       browser = await pw.chromium.launch(
          proxy={
              'server': "http://pr.oxylabs.io:7777",
              "username": "USERNAME",
              "password": "PASSWORD"
              },
          headless=False
      )
 
       page = await browser.new_page()
       await page.goto('https://www.amazon.com/b?node=17938598011')
       await page.wait_for_timeout(5000)
 
       all_products = await page.query_selector_all('.a-spacing-base')
       data = []
       for product in all_products:
           result = dict()
           title_el = await product.query_selector('span.a-size-base-plus')
           result['title'] = await title_el.inner_text()
           price_el = await product.query_selector('span.a-price')
           result['price'] = await price_el.inner_text()
           rating_el = await product.query_selector('span.a-icon-alt')
           result['rating'] = await rating_el.inner_text()
           data.append(result)
       print(data)
       await browser.close()
 
if __name__ == '__main__':
   asyncio.run(main())