Skip to main content

Utilizing AI for Web Scraping in Salesforce

· 5 min read
GPTfy Team
GPTfy Team
GPTfy Product Team

Table of Contents​

TL;DR​

Learn how to create a powerful competitor analysis tool by combining Salesforce with web scraping (Firecrawl) and AI using GPTfy.

With a few hours of work and minimal code, you can automatically extract and analyze competitor website data directly within your Salesforce opportunities.

Not a fan of reading articles? Check out the video here:

What?​

A plain English guide on building a competitor analysis tool in Salesforce that automatically scrapes and analyzes competitor websites. We’ll cover integrating web scraping APIs, configuring GPTfy, and using AI to generate insights – all with minimal code.

Who?​

Salesforce admins, developers, solution architects, and business analysts who want to enhance their CRM with automated competitive intelligence capabilities.

Why?​

To help your sales team with instant competitive insights.

Improve win rates. Make data-driven decisions. Save hours of manual research.

What can you do with it?​

  • Automated Competitor Analysis: Extract and analyze competitor website data directly within Salesforce opportunities
  • Real-time Intelligence: Get instant comparisons and insights when opportunity competitors change
  • Customizable Insights: Configure AI prompts to focus on specific competitive areas important to your business

1. Setting Up the Foundation: Salesforce + Firecrawl Integration​

The integration starts with Firecrawl, a web scraping API service. Here’s the setup process:

  • Sign up for a Firecrawl trial account and get API keys
  • Create a Remote Site Setting in Salesforce to allow API calls
  • Use ChatGPT/Claude to generate basic Apex classes for API integration
  • Add custom fields to Opportunity for competitor websites

GPTfy’s API Data Source feature makes this integration seamless and mostly declarative.

Firecrawl API Data Source Code​

global class firecrawlMasterDataSource implements ccai.AIDataSourceInterface {
    global String getExternalData(ccai_AI_Data_Source_c dataSource, String extractedData) {
        try {
            System.debug('Extracted Data (Raw Input): ' + extractedData);
            Map<String, Object> inputData = (Map<String, Object>) JSON.deserializeUntyped(extractedData);
            String url1 = (String) inputData.get('CompetitorWebSite1__c');
            String url2 = (String) inputData.get('CompetitorWebSite2__c');

            if (String.isBlank(url1) || String.isBlank(url2)) {
                throw new IllegalArgumentException('One or both competitor websites are missing.');
            }

            Map<String, Object> combinedData = makeCallouts(url1, url2);
            return JSON.serializePretty(combinedData);
        } catch (Exception e) {
            System.debug('Error: ' + e.getMessage());
            throw new CustomException('Error processing external data: ' + e.getMessage());
        }
    }

    private Map<String, Object> makeCallouts(String url1, String url2) {
        FirecrawlService.FirecrawlResponse response1 = FirecrawlService.fetchWebsiteData(url1);
        FirecrawlService.FirecrawlResponse response2 = FirecrawlService.fetchWebsiteData(url2);

        String response1Data = response1 != null && response1.data != null
            ? String.valueOf(response1.data).substring(0, Math.min(String.valueOf(response1.data).length(), 5000))
            : null;
        String response2Data = response2 != null && response2.data != null
            ? String.valueOf(response2.data).substring(0, Math.min(String.valueOf(response2.data).length(), 5000))
            : null;

        Map<String, Object> combinedData = new Map<String, Object>();
        combinedData.put('Website1Data', response1Data);
        combinedData.put('Website2Data', response2Data);

        System.debug('Trimmed Combined Data: ' + JSON.serializePretty(combinedData));
        return combinedData;
    }

    global class CustomException extends Exception {}
}

2. Configuring GPTfy for Web Scraping​

alt text

GPTfy acts as the bridge between Salesforce, Firecrawl, and AI:

  • Create an API Data Source in GPTfy specifying your connector class
  • Configure data mapping to include relevant Opportunity fields
  • Set up field mappings for competitor website fields
  • Choose your preferred AI model and adjust settings like temperature

The beauty is that most of this is declarative – you only need Apex for the initial API connection.

3. Crafting AI Prompts for Competitive Analysis​

alt text

The final piece is creating effective prompts that turn raw website data into actionable insights:

  • Design prompts that extract key competitive information
  • Configure formatting for easy readability
  • Use AI to help refine and optimize your prompts
  • Test and iterate based on results

Prompt for Website Comparison in Salesforce​

Objective: Analyze and compare two websites based on provided JSON data, highlighting key differences and similarities.

Input: JSON containing scraped website content from two companies, including HTML structure and metadata.

Requirements:
1. Extract and identify:
- Company names from metadata, URLs, or titles
- Main navigation elements
- Content focus and purpose
- Design elements and layout
- Target audience indicators
- Special features or functionalities

2. Formatting:
- Use proper company names throughout
- Organize comparison by category
- Highlight direct contrasts
- Include specific examples
- Note unique features

3. Structure Analysis As:
- Primary Purpose/Focus
- Navigation Structure
- Content Organization
- User Experience Elements
- Distinguishing Features

4. Maintain Clarity:
- Use concise language
- Provide specific examples
- Avoid technical jargon
- Focus on meaningful differences

Output Format:
[Company1] vs [Company2]: A Feature Analysis

Primary Focus:
- [Company1]: [Main purpose/focus]
- [Company2]: [Main purpose/focus]

Key Differentiators:
1. Navigation & Structure
2. Content Approach
3. User Experience
4. Special Features

Unique Strengths:
- [Company1]: [List key advantages]
- [Company2]: [List key advantages]

Pro tip: Use AI to help write and refine your prompts by showing it sample data and desired outputs.

Conclusion​

Building a powerful competitor analysis tool doesn’t have to be complex or time-consuming. By combining Salesforce, Firecrawl, and GPTfy, you can create an automated solution in just a few hours – even with minimal coding experience.

The power lies in the combination of Salesforce’s CRM capabilities, Firecrawl’s web scraping capabilities, and AI’s analysis capabilities, all orchestrated seamlessly by GPTfy.

This approach demonstrates how quickly you can go from idea to working prototype using modern AI tools.