repository·master·Indexed 27 days ago
https://github.com/berstend/puppeteer-extraA modular plugin framework that extends Puppeteer and Playwright. It allows developers to add advanced features such as stealth evasions, ad-blocking, user-agent anonymization, and automatic captcha solving. The ecosystem includes playwright-extra as a drop-in replacement for Playwright, as well as specialized plugins like @extra/proxy-router for dynamic proxy routing and puppeteer-extra-plugin-adblocker.
puppeteer-extra is a modular plugin framework for puppeteer. It allows you to extend Puppeteer's functionality using various plugins.
playwright-extra is a similar modular plugin framework designed for [Playwright].
Note: For detailed usage, API references, and installation instructions for the main library, please refer to the specific package documentation in the packages/puppeteer-extra directory.
To use the plugin, import it, initialize it, and pass it to puppeteer.use(). You can then create a tunnel for a browser instance to make DevTools accessible via a public URL.
const puppeteer = require('puppeteer-extra')
const devtools = require('puppeteer-extra-plugin-devtools')()
puppeteer.use(devtools)
puppeteer
.launch({ headless: true, defaultViewport: null })
.then(async browser => {
console.log('Start')
const tunnel = await devtools.createTunnel(browser)
console.log(tunnel.url)
const page = await browser.newPage()
await page.goto('https://example.com')
console.log('All setup.')
})Use puppeteer-extra as a drop-in replacement for puppeteer. You can augment it with plugins using the puppeteer.use() method. The following example demonstrates using the stealth and adblocker plugins.
// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality.
// Any number of plugins can be added through `puppeteer.use()`
const puppeteer = require('puppeteer-extra')
// Add stealth plugin and use defaults (all tricks to hide puppeteer usage)
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
// Add adblocker plugin to block all ads and trackers (saves bandwidth)
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker')
puppeteer.use(AdblockerPlugin({ blockTrackers: true }))
// That's it, the rest is puppeteer usage as normal 😊
puppeteer.launch({ headless: true }).then(async browser => {
const page = await browser.newPage()
await page.setViewport({ width: 800, height: 600 })
console.log(`Testing adblocker plugin..`)
await page.goto('https://www.vanityfair.com')
await page.waitForTimeout(1000)
await page.screenshot({ path: 'adblocker.png', fullPage: true })
console.log(`Testing the stealth plugin..`)
await page.goto('httpshttps://bot.sannysoft.com')
await page.waitForTimeout(5000)
await page.screenshot({ path: 'stealth.png', fullPage: true })
console.log(`All done, check the screenshots. ✨`)
await browser.close()
})Use playwright-extra as a drop-in replacement for playwright. It augments the standard Playwright browser instances with plugin functionality via the .use() method. The following example demonstrates how to use the puppeteer-extra-plugin-stealth plugin to hide automation traces.
// playwright-extra is a drop-in replacement for playwright,
// it augments the installed playwright with plugin functionality
const { chromium } = require('playwright-extra')
// Load the stealth plugin and use defaults (all tricks to hide playwright usage)
// Note: playwright-extra is compatible with most puppeteer-extra plugins
const stealth = require('puppeteer-extra-plugin-stealth')()
// Add the plugin to playwright (any number of plugins can be added)
chromium.use(stealth)
// That's it, the rest is playwright usage as normal 😊
chromium.launch({ headless: true }).then(async browser => {
const page = await browser.newPage()
console.log('Testing the stealth plugin..')
await page.goto('https://bot.sannysoft.com', { waitUntil: 'networkidle' })
await page.screenshot({ path: 'stealth.png', fullPage: true })
console.log('All done, check the screenshot. ✨')
await browser.close()
})To use the REPL, register the plugin with puppeteer-extra and call the .repl() method on a Page or Browser instance. This will pause your code execution and open an interactive prompt in your terminal.
REPL Commands:
tab (twice): Show all available properties via auto-completion.inspect: Returns the current object.exit (or ctrl+c): Leaves the REPL session.const puppeteer = require('puppeteer-extra')
puppeteer.use(require('puppeteer-extra-plugin-repl')())
puppeteer.launch({ headless: true }).then(async browser => {
const page = await browser.newPage()
await page.goto('https://example.com')
// Start an interactive REPL here with the `page` instance.
await page.repl()
// Afterwards start REPL with the `browser` instance.
await browser.repl()
await browser.close()
})By default, the plugin only solves reCAPTCHAs on the immediate page. To solve captchas located within iframes, you must iterate through the page's child frames and call solveRecaptchas() on each frame.
Additionally, you may need to disable site isolation in your Puppeteer launch configuration to allow access to cross-origin iframes.
// Loop over all potential frames on that page
for (const frame of page.mainFrame().childFrames()) {
// Attempt to solve any potential captchas in those frames
await frame.solveRecaptchas()
}To handle cross-origin iframes, launch Puppeteer with these arguments:
puppeteer.launch({
args: [
'--disable-features=IsolateOrigins,site-per-process,SitePerProcess',
'--flag-switches-begin --disable-site-isolation-trials --flag-switches-end'
]
})Install the plugin using yarn:
yarn add puppeteer-extra-plugin-flashInstall the @extra/proxy-router plugin using npm or yarn.
Standard installation:
npm install @extra/proxy-router
# - or -
yarn add @extra/proxy-routerIf using Playwright (first-time setup):
npm install playwright playwright-extra @extra/proxy-router
# - or -
yarn add playwright playwright-extra @extra/proxy-routerIf using Puppeteer (first-time setup):
npm install puppeteer puppeteer-extra @extra/proxy-router
# - or -
yarn add puppeteer puppeteer-extra @extra/proxy-routernpm install @extra/proxy-routerTo create a plugin, extend the PuppeteerExtraPlugin class. The base class provides convenience methods to handle common Puppeteer browser events (like onPageCreated or beforeLaunch) without manual boilerplate. You must implement a name getter. You can also define defaults for your plugin options, requirements to signal needs to the base class, and dependencies to ensure other plugins are loaded.
// hello-world-plugin.js
const { PuppeteerExtraPlugin } = require('puppeteer-extra-plugin')
class Plugin extends PuppeteerExtraPlugin {
constructor(opts = {}) {
super(opts)
}
get name() {
return 'hello-world'
}
async onPageCreated(page) {
this.debug('page created', page.url())
const ua = await page.browser().userAgent()
this.debug('user agent', ua)
}
}
module.exports = function(pluginConfig) {
return new Plugin(pluginConfig)
}
// foo.js
const puppeteer = require('puppeteer-extra')
puppeteer.use(require('./hello-world-plugin')())
;(async () => {
const browser = await puppeteer.launch({ headless: false })
const page = await browser.newPage()
await page.goto('http://example.com', { waitUntil: 'domcontentloaded' })
await browser.close()
})()Install the stealth plugin using npm or yarn. If you are setting up a new project with puppeteer-extra, you should also install puppeteer and puppeteer-extra.
# Install only the plugin
yarn add puppeteer-extra-plugin-stealth
# - or -
npm install puppeteer-extra-plugin-stealth
# Install the full stack if needed
yarn add puppeteer puppeteer-extra puppeteer-extra-plugin-stealth
# - or -
npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealthInstall the REPL plugin using yarn to enable interactive debugging and exploration in your Puppeteer sessions.
yarn add puppeteer-extra-plugin-replYou can create new plugins for puppeteer-extra by extending the PuppeteerExtraPlugin base class.
For implementation guidance:
PuppeteerExtraPlugin base class documentation.