Social media and web content provide market intelligence, competitive insights, and trend analysis. Graphlit syncs Reddit, Twitter, YouTube, RSS feeds, and crawls websites—making everything searchable.
Covered Sources
- Reddit (subreddits, threads)
- Twitter/X (posts, threads)
- YouTube (videos, transcripts)
- RSS feeds (blogs, news)
- Web crawling (documentation sites)
- Tavily & Exa (web search APIs)
import { FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types';
const redditFeed = await graphlit.createFeed({
name: 'Tech Subreddit',
type: FeedServiceTypes.Reddit,
reddit: {
token: reddit_oauth_token,
subreddit: 'MachineLearning',
readLimit: 100,
sortBy: 'hot' // 'hot', 'new', 'top'
}
});
Use cases:
- Market research
- Customer sentiment
- Competitive intelligence
- Trend tracking
RSS Feeds
const rssFeed = await graphlit.createFeed({
name: 'Tech News',
type: FeedServiceTypes.Rss,
rss: {
uri: 'https://techcrunch.com/feed/',
readLimit: 50
}
});
No OAuth required - just provide RSS URL.
YouTube
const youtubeFeed = await graphlit.createFeed({
name: 'Channel Videos',
type: FeedServiceTypes.YouTube,
youtube: {
token: youtube_api_key,
channelId: 'channel-id',
readLimit: 50
}
});
What syncs:
- Video transcripts
- Titles, descriptions
- Comments (optional)
Web Crawling
const webCrawl = await graphlit.createFeed({
name: 'Documentation Crawler',
type: FeedServiceTypes.Web,
web: {
uri: 'https://docs.example.com',
readLimit: 500,
allowedDomains: ['docs.example.com'],
excludedPaths: ['/api/', '/archive/']
}
});
What gets crawled:
- Page HTML → markdown
- Links (follows to crawl more)
- Images (optional)
- Metadata
Production Patterns
Market Intelligence
// Monitor multiple sources
const sources = [
{ type: FeedServiceTypes.Reddit, config: { subreddit: 'startups' } },
{ type: FeedServiceTypes.Rss, config: { uri: 'https://news.ycombinator.com/rss' } },
{ type: FeedServiceTypes.Twitter, config: { hashtag: '#saas' } }
];
// Search across all sources
const insights = await graphlit.queryContents({
search: 'AI pricing strategies'
});
Competitive Analysis
// Crawl competitor docs
const competitorFeed = await graphlit.createFeed({
name: 'Competitor Docs',
type: FeedServiceTypes.Web,
web: {
uri: 'https://competitor.com/docs',
readLimit: 200
}
});
// Ask questions about competitors
const response = await graphlit.promptConversation(
'What features does the competitor offer?',
conversationId
);
Related Guides
- Data Connectors - All connector types
- Building AI Chat Applications - RAG for market intel