Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewLocalExecutor ¶
func NewLocalExecutor( objectDownloader object.Downloader[object.GlobalReference], parsedObjectPool *model_parser.ParsedObjectPool, dagUploader dag.Uploader[object.InstanceName, object.GlobalReference], fetcher Fetcher, filePool pool.FilePool, ) remoteworker.Executor[*model_executewithstorage.Action[object.GlobalReference], model_core.Decodable[object.LocalReference], model_core.Decodable[object.LocalReference]]
NewLocalExecutor creates a remote worker protocol executor that is capable of processing "fetch" actions. URLs listed in the action are fetched using a HTTP client, and the resulting file is uploaded to storage.
Types ¶
type Fetcher ¶
type Fetcher interface {
Fetch(ctx context.Context, url string, headers []*model_fetch_pb.Target_Header) (io.ReadCloser, error)
}
Fetcher is called into by LocalExecutor to fetch files. Simple setups may only use a single instance of Fetcher. More complex ones can use multiple instances in case support for different network protocols is desired.
func NewHTTPFetcher ¶
NewHTTPFetcher creates a Fetcher that is capable of fetching files via HTTP and HTTPS.
func NewSchemeDemultiplexingFetcher ¶
NewSchemeDemultiplexingFetcher wraps a set of Fetchers, and forwards requests to them based on the URL scheme. This makes it possible to launch a single bonanza_fetcher process that supports various URL schemes.
Source Files