Extract All Links from All Open Tabs #Chrome

I created a Chrome extension that extracts (scrapes) all links (URLs) from all open tabs (pages/HTML) in Google Chrome.

Why I Wanted to Create This

I have several media sites, and there are many sites and pages where monetization isn’t optimized or well-maintained. I was thinking about how to monetize them - in other words, what kind of affiliate ads or programs would be optimal.

Let’s say there’s an article (e.g., “How to Use Linux Commands [For Beginners]”). It’s a simple idea, but I thought that if I could search with a certain query (e.g., “Linux commands”), crawl all the sites and pages that appear in the search results, get the links, and filter/extract URLs that look like affiliate ASP (e.g., filter by “a8.net” or filter by “aff”), I might get hints about what programs other people are using.

How the Tool Works and Usage

First, run the script.

When the script runs, it automatically retrieves information from all tabs in the active window (the frontmost window), processes link extraction, logs it, and finishes.

(Image left source: “Webkaru”)

When you copy the object from the log, it’s copied in JSON format.

Open the copied JSON in a text editor and search with Ctrl-F.

I was able to find several a8.net links. There were also sites using moshimo (Moshimo Affiliate) and others.

(Image right source: “Kitsune no Wakusei”)

This time I extracted links, but any information on HTML can be extracted without restrictions. For example, you can also extract all image URLs on HTML.

Alternatively, if the same link is duplicated, you can remove duplicates, or count and add statistical information.

Tool to Extract All Links from All Open Tabs

The complete code is as follows.

(async () => {
    const sleep = m => new Promise(r => setTimeout(r, m));
    const moment = (await import('https://cdn.jsdelivr.net/npm/[email protected]/+esm')).default;
    const axios = (await import('https://cdn.skypack.dev/[email protected]')).default;
    const URLParse = (await import('https://cdn.skypack.dev/[email protected]')).default;
    const uuid = (await import('https://cdn.skypack.dev/@lukeed/[email protected]')).v4;
    const jsyaml = (await import('https://cdn.skypack.dev/[email protected]')).default;
    const script = (await axios.get(`http://localhost:8080/web/lib/utils.js?ts=${(+new Date())}`)).data;
    eval(script);

    app.console.setLogPrefix("[Link Scraper]");

    app.console.warn("waiting... (a window)");
    let activeWindowTabs;
    for (;;) {
        activeWindowTabs = await ChromeUtils.getActiveWindowTabs();
        await sleep(200);
        if (activeWindowTabs.length >= 1) break;
    }
    
    app.console.info({activeWindowTabs, });

    let detailedTabs = [];
    for (let tab of activeWindowTabs) {
        let outerHTML = await ChromeUtils.exec(tab.id, `document.documentElement.outerHTML`);
        outerHTML = outerHTML?.[0];
        const parser = new DOMParser();
        const doc = parser.parseFromString(outerHTML, 'text/html');
        let links = Array.from(doc.getElementsByTagName('a')).map(link => link.href);
        let linksWithTexts = Array.from(doc.getElementsByTagName('a')).map(link => { return { text: link.innerText, link: link.href }})
        detailedTabs.push({
            ...tab,
            linksWithTexts,
        });
    }

    app.console.info(detailedTabs)
    app.console.info("y.");
})();

(The product includes my custom library that is not included in the above code but has dependencies with it.)

Summary

Using this tool, you can extract all links from all open tabs. For example, you can use it to investigate what affiliate programs other people are using, what ads they’re posting, etc. There are probably many other uses as well.