Deep Crawling – Multi-Page Logic Made Simple

Navigate into internal links and extract deeper data:

• Begin with a list agent (e.g., product cards).
• Add a Get Link action, then select Start Deep Crawl.
• Build a sub-agent for the linked page — e.g., extract description, followers, or metadata.
• Test with Run & Populate to preview multiple rows at once.
• Edit sub-agents independently, remove redundant fields, and save with confidence.

🕸️ How to Create Deep Crawling Agents in Bardeen

• Begin with a list agent (e.g., product cards).
• Add a Get Link action, then select Start Deep Crawl.
• Build a sub-agent for the linked page — e.g., extract description, followers, or metadata.
• Test with Run & Populate to preview multiple rows at once.
• Edit sub-agents independently, remove redundant fields, and save with confidence.

✅ Use Case Example

You have a list of products on Product Hunt, and you want to extract:

• Product name (on the list page)
• Then, for each product, go to the product detail page to get:
  • Description
  • Followers
  • Ranking

This is where Deep Crawling shines.

🧭 Step-by-Step: Build a Deep Crawling Agent

1. Define the List Structure

See List Extraction section on how to define list of items.

2. Extract Fields from Each Item

Inside the list operation:

• Click on any element you want to collect (e.g., title, votes, one-liner)
• Choose actions like Get Text or Get Link
• Rename the fields to match their content (e.g., Product Name, Vote Count)

These values will be collected for every item in the list.

3. Add the Destination Link

To go deeper:

• Click on the product title or external link inside the list
• Choose Get Link
• Name it something like Product URL

This will be the link the agent uses for deep crawling.

4. Start Deep Crawl

Next to the Product URL action:

• Click the three dots (⋮) menu
• Select Start Deep Crawl

This opens a new agent builder view for the destination page.

5. Configure the Sub-Agent

On the new page, extract additional fields that aren’t available in the list and hit Save Agent.

Suggestion: Use unique names.

7. Preview and Test with Run & Populate

After saving your deep crawling operation, you'll return to the main list view.

Bardeen will automatically populate the preview with data from the first linked page in the list.

To test the deep crawling logic on additional items, click Run & Populate.

This will:

• Trigger your agent for the next three rows in the list
• Execute the deep crawl on each linked page
• Display the updated results in the preview panel

Use this to verify that:

• Data is being collected consistently across different pages
• Your sub-agent works reliably beyond the first link

8. Edit the Deep Crawling Operation

If you need to update or fix the deep crawling logic after testing:

• Find the deep crawling operation in your list workflow
• Click the three dots (⋮) on the deep crawling card
• Select Edit

This will open the deep crawling operation in a separate builder window.

From there, you can:

• Add or remove fields
• Rename fields for clarity
• Adjust actions (e.g., add delays or clicks)
• Delete fields that may be duplicated or no longer needed

Once done, click Save Agent to return to the main workflow — your changes will be applied automatically.

9. Finalize and Use

Once satisfied:

• Click Save Template
• Now when you run the agent it will run for all the items unless a limit is specified.

You’ve now created a fully functioning Deep Crawling Agent 🚀