Open-source tools for market research and competitor monitoring

What this guide covers

• What each of the three tools does, according to its own README and documentation
• A first small project for each tool, sized for a small business
• What you need to run each one: Docker or Python, API keys, skills
• A comparison table: tool, purpose, requirements, licence
• What the LICENSE files say, including the attribution clause in Crawl4AI
• A short weekly monitoring routine as a step list
• How to verify an AI research report, and the limits of collecting data

Three tools, three different jobs

GPT Researcher is a research agent: you ask a question and get a written report with citations. Crawl4AI is a crawler: you give it addresses and get those pages as clean Markdown or structured data. changedetection.io is a monitor: it checks a page on a schedule and notifies you when it changes.

All three are published on GitHub and can run on your own machine. On 4 October 2026 GitHub showed about 30 thousand stars for GPT Researcher, about 85 thousand for Crawl4AI and about 35 thousand for changedetection.io. Stars measure attention, not quality.

GPT Researcher: a first research draft with its sources

The README describes GPT Researcher as an open deep research agent for web and local research. A planner turns your task into research questions, other agents gather information for each one, every source is summarised and tracked, and the findings are combined into one report. It works from web pages and from your own files, such as PDF, Excel and Word, and exports the report to PDF, Word and other formats.

The default setup needs Python 3.12 or later (or Docker), an OpenAI key for the language model and a Tavily key for web search. The documentation lists other model providers, including local models through Ollama, and other search engines such as DuckDuckGo and Searx, each with its own key requirements and limits. Providers bill API usage at their own rates.

A first small project: ask for an overview of a niche you know well, for example coffee subscriptions for offices in your city, so you can see where the report is thin or wrong. The README calls the project experimental, provides it as is, and says the aim is to reduce incorrect and biased facts, not to remove them.

Crawl4AI: competitor pages as one comparable table

According to its README, Crawl4AI is an open-source crawler that opens pages in a browser and returns clean Markdown, with filters that strip menus, footers and other boilerplate. It can run JavaScript and scroll to the end of a page, which matters for shops that load products as you scroll.

For structured data, CSS or XPath schemas and regular expressions need no language model: you describe where the name, price and delivery terms sit, and the schema works on every page with that layout. Extraction by a language model needs a model API key or a local model, and the documentation notes that model output can vary or hallucinate, while selectors do exactly what you specify.

You need Python 3.10 or later and a browser, which the setup command installs. This is the most technical of the three: someone must write a short Python script and find CSS selectors in the browser's developer tools. A first small project: build one table from ten competitor product pages with the same columns (product, price, pack size, delivery terms, date collected). Start with three sites, because every layout needs its own schema.

changedetection.io: knowing when a page changes

changedetection.io checks web pages on a schedule you set and, according to its README, notifies you by email, Slack, Telegram, Discord, webhook or other channels when the content changes. For product pages there is a restock and price detection mode with price limits and a percentage change. It also tracks text changes in PDF files, such as price lists.

Filters let you watch one part of a page through a CSS selector, XPath or the visual selector and ignore the rest. Trigger conditions fire only when a keyword appears or disappears, or when a price crosses an amount you set. Pages that build their content with JavaScript need the Chrome-based fetcher, as does the visual selector.

Installation is the simplest of the three: one Docker command, or pip with Python 3.11 or later, then you paste addresses into a web interface or import them from an Excel file. A first small project: watch five competitor pages and review the changes once a week. The optional AI summaries send detected changes to a model provider you connect, and the README warns never to rely on such output as complete or accurate.

What the licences allow, in plain words

Each repository has a LICENSE file at its root, and GitHub labels all three as Apache-2.0. The Apache License 2.0 text grants a perpetual, worldwide, no-charge, royalty-free right to reproduce the software, prepare derivative works and distribute them. It has no clause limiting this to non-commercial use.

Crawl4AI's LICENSE adds an Attribution Requirement section: distributions, publications and public uses of the software or of works based on it must carry a stated attribution line naming the Crawl4AI project. The changedetection.io file is the standard text with the authors' copyright line; the repository has no separate commercial licence file, and the README offers a paid hosted subscription and commercial support.

The licence covers the code, not the pages you collect, the APIs you connect or the hosted versions sold by the authors. Read each LICENSE file before commercial use; this summary is not legal advice. When you pass the software on, changed or unchanged, the Apache conditions are:

• Give recipients a copy of the licence.
• Mark the files you changed.
• Keep the existing copyright, patent, trademark and attribution notices.
• Do not use the project's names and trademarks, except to say where the software comes from.
• The software comes as is, without warranties, and contributors are not liable for damages.

A short weekly monitoring routine

changedetection.io does the watching, a plain spreadsheet keeps the record. The tool tells you that something changed; what it means is your call.

• List five competitor pages and the one thing you watch on each: a price, stock, delivery terms, a promotion, vacancies.
• Add each page as a watch, narrow it to the block that matters and set the interval. Once or twice a day is enough and keeps the load on the other site low.
• On a fixed day, open the list and the difference view for each page marked as changed.
• Confirm each change on the live page. A redesign, a cookie banner or a rotating block can look like one.
• Record each real change in one table: date, competitor, page, what it was, what it became, link.
• Finish with one line: what you will do, or that no action is needed.

How to verify an AI research report

Treat a generated report as a map of sources and a first outline, not as a finding. A model can cite a weak page, such as an unsigned blog, and misread a strong one: the wrong year, the wrong country, a forecast presented as a measured figure. The GPT Researcher README says the project prefers the most frequent information across many sites. Frequency is not proof: twenty sites can repeat one press release.

Check every figure you plan to use in a decision or a publication:

• Open the cited link and find the exact number on the page. If it is not there, remove it.
• Check the unit, currency, year and territory. A world figure is not a figure for your country.
• Follow the citation to the original: the statistics office, the company report, the price page.
• Keep a log: figure, source link, date opened, status (confirmed, differs, not found).
• For a second run, limit the research to addresses or domains you trust; the documentation describes both options.

Limits and caution when collecting data

None of these tools knows whether a published page is true or current. A crawler brings back what the site shows, including an outdated price. A monitor reports a change in text, not its cause.

The changedetection.io README states that you alone are responsible for complying with the terms of service, robots.txt and access policies of the sites you monitor, and with applicable law. In Crawl4AI the robots.txt check is off by default, so turn it on. What follows is practical caution, not legal advice:

• Read the site's terms and robots.txt first, skip sites that forbid automated access, and keep the frequency low.
• Do not go behind logins or paywalls, even though the tools have features for sessions and proxies.
• Collect business facts: prices, specifications, terms. Leave out names, contacts and reviews tied to people. Personal-data law (in the EU, the GDPR) can apply even to public data.
• Use the material for your own analysis. Republishing a competitor's texts or photos is a separate copyright question.