IGScraperKit 0.4.1

You want to add pod 'IGScraperKit', '~> 0.4' similar to the following to your Podfile:

target 'MyApp' do
  pod 'IGScraperKit', '~> 0.4'
end

Then run a pod install inside your terminal, or from CocoaPods.app.

Alternatively to give it a test run, run the command:

pod try IGScraperKit

IGScraperKit 0.4.1

By
Francis Chong

Create dynamic web scraper in Objective-C or Ruby.

Usage

Create a scraper:

#import "IGScraperKit.h"
IGScraper* scraper = [IGScraper scraperWithBlock:^id(IGXMLNode* node, NSString* url) {
                return [[[node queryWithXPath:@"//p"] firstObject] text];
            }];

Then scrape HTML with scraper:

[scraper scrape:@"<html><p>Hello World</p></html>" URL:nil];
// => @"Hello World"

#import "IGScraperKit.h"

IGScraperRecipe* recipe = [[IGScraperRecipe alloc] init];
[recipe addURLPattern:[NSRegularExpression regularExpressionWithPattern:@"https://www\.google\.com/search\?q=.+" options:0 error:nil]
     withScraperBlock:^id(IGXMLNode *node, NSString *url) {
  // handling for the page ...
  return data;
}];
[recipe addURLPattern:[NSRegularExpression regularExpressionWithPattern:@"https://www\.google\.com/" options:0 error:nil]
  withScraperBlock:^id(IGXMLNode *node, NSString *url) {
  // handling for the page ...
  return data;
}];
[recipe scrapeWithHTML:html URL:URL];

Write Scraper With Ruby

If you want something more dynamic, you can define a Recipe in Ruby:

# A recipe scrape page based on URL
class GoogleRecipe < ScraperKit::Recipe
  title "Google Search"

  # define a HTML scraper by `on url`, where url can be a string or a Regexp
  on %r{https://www\.google\.com/search\?q=.+} do
    # doc is a HTMLDoc object represent the document
    doc.xpath('//h3/a').collect {|node| node.text }
  end

  # if the page you need to parse is not HTML
  # use `on_text url`
  on_text %r{https://www\.google\.com/search\.json.+} do
    # doc is a string of the document
    JSON.parse(doc)
  end
end

Then load the recipe into IGRecipeRegistry and parse the page:

#import "IGScraperKit.h"

// load the recipe
NSString* path = [[NSBundle mainBundle] pathForResource:@"google" ofType:@"rb"];
NSString* recipe = [[NSString alloc] initWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil];
IGRecipeRegistry* registry = [[IGRecipeRegistry alloc] init];
[registry loadRecipe:Recipe(@"walmart")];

NSArray* result = [registry scrapeWithHTML:html url:@"https://www.google.com/search?q=doughnuts"];
// => <__NSArrayM 0xed85590>(
// Doughnut - Wikipedia, the free encyclopedia,
// Home - Krispy Kreme Doughnuts and Coffee,
// Voodoo Doughnut - The Magic is in the Hole!!!,
//【超人氣甜甜圈】Krispy Kreme Doughnuts 、台北店 ... - Yam天空部落,
// Doughnut Recipes - Allrecipes.com,
// Doughnut Plant,
// Revolution Doughnuts,
// Sidecar Doughnuts & Coffee,
// Top Pot Hand-Forged Doughnuts,
// Lucky's Doughnuts
// )

To use this, you will need to include JavaScriptCore framework (iOS 7, OS X 10.9) and define IGSCRAPERKIT_ENABLE_SCRIPTING before import IGScraperKit.h.

Installation

To install IGScraperKit throught CocoaPods, add following lines to your Podfile:

pod "IGScraperKit", '0.3.1'

Or with Ruby supports:

pod "IGScraperKit/Scripting", '0.3.1'

Dependencies

IGScraperKit use IGHTMLQuery for HTML processing. Check the ruby wrappers if you need to use the Ruby interface.
IGScraperKit use JavaScriptCore in iOS 7 and Opal for JavaScript support.
Use phantomjs to run Opal tests.

Development

Install gems: In the project folder, run the command: bundle install
Install cocoapods: Run the command: pod install

License

MIT License. Check LICENSE.txt.