Skip to content
All work

AI agents & automation2026

Research Crawler

A research agent that runs on your own computer, reads LinkedIn, X and the open web in your own signed-in Chrome, and turns a goal written in plain words into a ranked report with links. The reading costs zero AI tokens.

Select any part of the research run to see what it does.

A research run, animated in your browser: posts, comments, likes and pages are read, ranked and written into a report. Not a screenshot.

Role
Design and engineering
Timeline
2026
Team
Solo or team size
Stack
  • JavaScript (Node.js 22),
  • Chrome DevTools Protocol,
  • playwright-core,
  • MCP server,
  • Claude CLI or Ollama (optional)
Links

    Overview

    Research Crawler (its codebase is called Social Scout) reads LinkedIn, X and the open web: searches, posts with their comments and replies, profiles, companies, LinkedIn groups, X communities and web pages. Give it a goal such as “what developers think of AI coding agents this week” and it runs the searches, reads the comments under the most-discussed posts and ranks it all into a report. It can also watch topics, hashtags, accounts and communities all day and show what is rising on a local dashboard. It runs from the command line, and as an MCP server for AI apps such as Claude Code, Claude Desktop, Cursor and VS Code.

    Problem

    Following a conversation on social media means scrolling feeds and comment threads by hand. Paid scraping APIs and browser agents that send screenshots to a model cost hundreds of times more than a model reading a compact digest. And automated reading is against LinkedIn's and X's terms, and LinkedIn can restrict accounts it thinks are automated, so a tool that reads them has to keep that risk as low as it can and say so plainly.

    Approach

    Everything is read in the user's own signed-in Chrome, with a profile of its own, and extracted with code from two places that are merged: the JSON the page itself downloads (X's GraphQL, LinkedIn's Voyager data) and the rendered page. When a site changes one, the other usually still works, and reading costs zero AI tokens. A model is optional. It can plan the searches and write the report, and then it reads a compact digest of a few thousand tokens instead of whole pages. Around it all sit a read-only network guard, human pacing, daily caps, active hours and an automatic stop. There are no stealth tricks: it does not spoof fingerprints, hide automation or solve captchas.

    How a research run works

    Research a goal

    One command or one MCP call. Reading is done by code in your own browser. From the command line, a model, when one is set up, plans the searches and writes the report. From an AI app, the run uses no model of its own and returns a report built by code.

    • agent
    • check
    • tool
    • person

    1. 01 / 07person

      You write the goal

      What you want to know, in plain words, and which sites to read: LinkedIn, X, the open web, or all three. You can also give your own queries.

    2. 02 / 07agent

      Plan the searches

      A model plans up to six searches written the way people post, in the goal's language, plus the terms that mark a relevant result and the ones that mark an irrelevant one. Without a model, or if it fails, it searches your keywords instead.

    3. 03 / 07tool

      Search and read

      Each search runs in Chrome under its own profile, one page at a time with human pauses, and every item is extracted with code from the page's own JSON and the rendered page, likes, comments and reposts included.

    4. 04 / 07tool

      Read the comments

      On each site it takes the most-discussed matching posts, three by default from the command line (one from an AI app), and reads up to 30 comments or replies under each, because that is where the opinions are.

    5. 05 / 07tool

      Read the web

      When asked, it runs up to three web searches and, from the command line, can also read the top few results as clean text. LinkedIn and X links stay search results, so they never skip those sites' caps and checks.

    6. 06 / 07check

      Rank

      Duplicates go, and plain code ranks everything by match to the goal with a light lift for engagement and freshness. Items with an excluded term are dropped.

    7. 07 / 07agent

      Write the report

      A model reads a compact digest instead of whole pages and writes key findings, notable voices, disagreements and open questions. It is told to cite items by their links, say where the data is thin and invent nothing. Without a model, code builds the report.

    Watch all day for trends

    For following a topic over time rather than a single question. The scheduler spreads its reading through the day so the daily caps are never exceeded.

    • check
    • tool
    • person

    1. 01 / 06person

      You choose what to watch

      Searches, hashtags, profiles, companies and communities (LinkedIn groups or X communities), on either site.

    2. 02 / 06tool

      Scheduled checks

      Checks run through the day inside active hours (08:00 to 23:00 by default), spread to stay within 85% of each site's daily cap, with the rest kept for your own research.

    3. 03 / 06check

      Automatic stop

      On a security check, a restriction notice, a “too many requests” answer or a sign-out, it stops, pauses that site (24 hours for checks, 2 for rate limits, 6 for a sign-out) and shows a notification.

    4. 04 / 06tool

      History

      Every sighting is saved on your computer as one JSON line, and the history keeps 60 days by default.

    5. 05 / 06tool

      Trends

      Plain code finds rising words, phrases and hashtags against the past week, the fastest-growing posts by engagement per hour, what commenters say with a rough word-count tone, and top voices, in English and Arabic.

    6. 06 / 06person

      You read the brief

      As a terminal summary, a written brief twice a day when a model is set, or a local dashboard at 127.0.0.1:7777. A paused site stays paused for those hours; you can lift it sooner with resume after checking the account, and signing in again lifts a sign-out pause.

    Checks again later in the day

    Acting only when you confirm

    It can like, comment, reply, follow, connect, message or repost on your own account, but only one confirmed action at a time.

    • check
    • tool
    • person

    1. 01 / 03tool

      Preview

      Every action is previewed first. From an AI app, the preview returns a one-time token, and the action only happens when it is called again with that token and a confirmation.

    2. 02 / 03person

      You confirm

      You approve it in the terminal or in the app. In an app that cannot ask you, it only previews unless you turn actions on yourself.

    3. 03 / 03check

      One action through the guard

      The guard opens a short window for that one action only, within a small daily cap per action type, then blocks writes again.

    Architecture

    • Command line to Research
    • MCP server to Research
    • Command line to Watch
    • Research to Own Chrome, read
    • Watch to Own Chrome
    • Read-only guard to Own Chrome, no writes
    • Own Chrome to Extractors, JSON + page
    • Own Chrome to Web reader
    • Extractors to Report
    • Web reader to Report
    • Extractors to Trends, history
    The command line and the MCP server start research runs; watches run from the command line or a background service. Every read goes through Chrome under its own profile, LinkedIn and X behind a read-only guard, and code extracts the items that become reports and trends.

    Challenges and solutions

    Challenge: Reading sites that keep changing their pages.
    Solution: Each read takes items from two places and merges them: the JSON the page downloads and the rendered page, so when a site changes one the other usually still works. The extractors are tested against saved page fixtures, and a --debug run saves the last page's HTML, the JSON it loaded and any error, which shows which half missed.
    Challenge: Reading sites whose terms forbid automation, on a real account.
    Solution: Its README says so plainly, and its instructions for AI assistants tell them to get the user's consent before setting it up. Then it keeps the risk as low as it can: read-only by default, human pace, daily caps, active hours, its own Chrome profile with DevTools on 127.0.0.1 only, and an automatic stop that recognises security notices in six languages. It does not disguise itself, and it tells you that a visible window is less noticeable than hidden Chrome.
    Challenge: Spending almost no AI tokens.
    Solution: Extraction is plain code, so reading is free. A model only sees a digest of a few thousand tokens, and planning falls back to your keywords when no model is set or it fails.
    Challenge: Letting AI apps use it without handing over the account.
    Solution: The action tool previews first and returns a one-time token, acts only when called again with that token and a confirmation, and in an app that cannot ask you it only previews unless you turn actions on yourself.

    Outcomes

    • Searches and reads LinkedIn, X and the open web: posts with comments and replies, profiles, companies, LinkedIn groups, X communities and any web page as clean text.
    • Twelve MCP tools for AI apps, from search and research to trends, watches and confirmed actions.
    • Trends in English and Arabic, as a terminal summary, a written brief or a local dashboard.
    • Everything stays on the computer under one folder; only the model you choose sees the compact digest.