org.apache.nutch.crawl
Class Generator.Selector

java.lang.Object
  org.apache.nutch.crawl.Generator.Selector

All Implemented Interfaces:

Closeable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,CrawlDatum,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>, org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.FloatWritable,org.apache.hadoop.io.Writable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Enclosing class:

Generator

public static class Generator.Selector
extends Object
implements org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,CrawlDatum,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>, org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.FloatWritable,org.apache.hadoop.io.Writable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Selects entries due for fetch.

Constructor Summary

Generator.Selector()

Method Summary

void	close()
void	configure(org.apache.hadoop.mapred.JobConf job)
int	getPartition(org.apache.hadoop.io.FloatWritable key, org.apache.hadoop.io.Writable value, int numReduceTasks) Partition by host.
void	map(org.apache.hadoop.io.Text key, CrawlDatum value, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output, org.apache.hadoop.mapred.Reporter reporter) Select & invert subset due for fetch.
void	reduce(org.apache.hadoop.io.FloatWritable key, Iterator<Generator.SelectorEntry> values, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output, org.apache.hadoop.mapred.Reporter reporter) Collect until limit is reached.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Generator.Selector

public Generator.Selector()

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf job)

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

close

public void close()

Specified by:

close in interface Closeable

map

public void map(org.apache.hadoop.io.Text key,
                CrawlDatum value,
                org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output,
                org.apache.hadoop.mapred.Reporter reporter)
         throws IOException

Select & invert subset due for fetch.

Specified by:

map in interface org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,CrawlDatum,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Throws:

IOException

getPartition

public int getPartition(org.apache.hadoop.io.FloatWritable key,
                        org.apache.hadoop.io.Writable value,
                        int numReduceTasks)

Partition by host.

Specified by:

getPartition in interface org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.FloatWritable,org.apache.hadoop.io.Writable>

reduce

public void reduce(org.apache.hadoop.io.FloatWritable key,
                   Iterator<Generator.SelectorEntry> values,
                   org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output,
                   org.apache.hadoop.mapred.Reporter reporter)
            throws IOException

Collect until limit is reached.

Specified by:

reduce in interface org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Throws:

IOException