Class SitemapLoader

Interface representing the parameters for initializing a SitemapLoader. SitemapLoaderParams

Hierarchy (view full)

Implements

Constructors

constructor

Properties

allowUrlPatterns caller chunkSize timeout webPath selector? textDecoder?

Methods

load loadAndSplit parseSitemap scrape imports scrapeAll

Constructors

constructor

Properties

allowUrlPatterns

allowUrlPatterns: undefined | (string | RegExp)[]

caller

caller: AsyncCaller

chunkSize

chunkSize: number

The size to chunk the sitemap URLs into for scraping.

Default

{300}

timeout

timeout: number

The timeout in milliseconds for the fetch request. Defaults to 10s.

webPath

webPath: string

Optional selector

selector?: `.${number}${string}` | `.i${string}` | `.a${string}` | `.b${string}` | `.c${string}` | `.d${string}` | `.e${string}` | `.f${string}` | `.g${string}` | `.h${string}` | `.j${string}` | `.k${string}` | `.l${string}` | `.m${string}` | `.n${string}` | `.o${string}` | `.p${string}` | `.q${string}` | `.r${string}` | `.s${string}` | `.t${string}` | `.u${string}` | `.v${string}` | `.w${string}` | `.x${string}` | `.y${string}` | `.z${string}` | `.I${string}` | `.A${string}` | `.B${string}` | `.C${string}` | `.D${string}` | `.E${string}` | `.F${string}` | `.G${string}` | `.H${string}` | `.J${string}` | `.K${string}` | `.L${string}` | `.M${string}` | `.N${string}` | `.O${string}` | `.P${string}` | `.Q${string}` | `.R${string}` | `.S${string}` | `.T${string}` | `.U${string}` | `.V${string}` | `.W${string}` | `.X${string}` | `.Y${string}` | `.Z${string}` | `#${number}${string}` | `#i${string}` | `#a${string}` | `#b${string}` | `#c${string}` | `#d${string}` | `#e${string}` | `#f${string}` | `#g${string}` | `#h${string}` | `#j${string}` | `#k${string}` | `#l${string}` | `#m${string}` | `#n${string}` | `#o${string}` | `#p${string}` | `#q${string}` | `#r${string}` | `#s${string}` | `#t${string}` | `#u${string}` | `#v${string}` | `#w${string}` | `#x${string}` | `#y${string}` | `#z${string}` | `#I${string}` | `#A${string}` | `#B${string}` | `#C${string}` | `#D${string}` | `#E${string}` | `#F${string}` | `#G${string}` | `#H${string}` | `#J${string}` | `#K${string}` | `#L${string}` | `#M${string}` | `#N${string}` | `#O${string}` | `#P${string}` | `#Q${string}` | `#R${string}` | `#S${string}` | `#T${string}` | `#U${string}` | `#V${string}` | `#W${string}` | `#X${string}` | `#Y${string}` | `#Z${string}` | `:${number}${string}` | `:i${string}` | `:a${string}` | `:b${string}` | `:c${string}` | `:d${string}` | `:e${string}` | `:f${string}` | `:g${string}` | `:h${string}` | `:j${string}` | `:k${string}` | `:l${string}` | `:m${string}` | `:n${string}` | `:o${string}` | `:p${string}` | `:q${string}` | `:r${string}` | `:s${string}` | `:t${string}` | `:u${string}` | `:v${string}` | `:w${string}` | `:x${string}` | `:y${string}` | `:z${string}` | `:I${string}` | `:A${string}` | `:B${string}` | `:C${string}` | `:D${string}` | `:E${string}` | `:F${string}` | `:G${string}` | `:H${string}` | `:J${string}` | `:K${string}` | `:L${string}` | `:M${string}` | `:N${string}` | `:O${string}` | `:P${string}` | `:Q${string}` | `:R${string}` | `:S${string}` | `:T${string}` | `:U${string}` | `:V${string}` | `:W${string}` | `:X${string}` | `:Y${string}` | `:Z${string}` | `|${number}${string}` | `|i${string}` | `|a${string}` | `|b${string}` | `|c${string}` | `|d${string}` | `|e${string}` | `|f${string}` | `|g${string}` | `|h${string}` | `|j${string}` | `|k${string}` | `|l${string}` | `|m${string}` | `|n${string}` | `|o${string}` | `|p${string}` | `|q${string}` | `|r${string}` | `|s${string}` | `|t${string}` | `|u${string}` | `|v${string}` | `|w${string}` | `|x${string}` | `|y${string}` | `|z${string}` | `|I${string}` | `|A${string}` | `|B${string}` | `|C${string}` | `|D${string}` | `|E${string}` | `|F${string}` | `|G${string}` | `|H${string}` | `|J${string}` | `|K${string}` | `|L${string}` | `|M${string}` | `|N${string}` | `|O${string}` | `|P${string}` | `|Q${string}` | `|R${string}` | `|S${string}` | `|T${string}` | `|U${string}` | `|V${string}` | `|W${string}` | `|X${string}` | `|Y${string}` | `|Z${string}` | `>${number}${string}` | `>i${string}` | `>a${string}` | `>b${string}` | `>c${string}` | `>d${string}` | `>e${string}` | `>f${string}` | `>g${string}` | `>h${string}` | `>j${string}` | `>k${string}` | `>l${string}` | `>m${string}` | `>n${string}` | `>o${string}` | `>p${string}` | `>q${string}` | `>r${string}` | `>s${string}` | `>t${string}` | `>u${string}` | `>v${string}` | `>w${string}` | `>x${string}` | `>y${string}` | `>z${string}` | `>I${string}` | `>A${string}` | `>B${string}` | `>C${string}` | `>D${string}` | `>E${string}` | `>F${string}` | `>G${string}` | `>H${string}` | `>J${string}` | `>K${string}` | `>L${string}` | `>M${string}` | `>N${string}` | `>O${string}` | `>P${string}` | `>Q${string}` | `>R${string}` | `>S${string}` | `>T${string}` | `>U${string}` | `>V${string}` | `>W${string}` | `>X${string}` | `>Y${string}` | `>Z${string}` | `+${number}${string}` | `+i${string}` | `+a${string}` | `+b${string}` | `+c${string}` | `+d${string}` | `+e${string}` | `+f${string}` | `+g${string}` | `+h${string}` | `+j${string}` | `+k${string}` | `+l${string}` | `+m${string}` | `+n${string}` | `+o${string}` | `+p${string}` | `+q${string}` | `+r${string}` | `+s${string}` | `+t${string}` | `+u${string}` | `+v${string}` | `+w${string}` | `+x${string}` | `+y${string}` | `+z${string}` | `+I${string}` | `+A${string}` | `+B${string}` | `+C${string}` | `+D${string}` | `+E${string}` | `+F${string}` | `+G${string}` | `+H${string}` | `+J${string}` | `+K${string}` | `+L${string}` | `+M${string}` | `+N${string}` | `+O${string}` | `+P${string}` | `+Q${string}` | `+R${string}` | `+S${string}` | `+T${string}` | `+U${string}` | `+V${string}` | `+W${string}` | `+X${string}` | `+Y${string}` | `+Z${string}` | `~${number}${string}` | `~i${string}` | `~a${string}` | `~b${string}` | `~c${string}` | `~d${string}` | `~e${string}` | `~f${string}` | `~g${string}` | `~h${string}` | `~j${string}` | `~k${string}` | `~l${string}` | `~m${string}` | `~n${string}` | `~o${string}` | `~p${string}` | `~q${string}` | `~r${string}` | `~s${string}` | `~t${string}` | `~u${string}` | `~v${string}` | `~w${string}` | `~x${string}` | `~y${string}` | `~z${string}` | `~I${string}` | `~A${string}` | `~B${string}` | `~C${string}` | `~D${string}` | `~E${string}` | `~F${string}` | `~G${string}` | `~H${string}` | `~J${string}` | `~K${string}` | `~L${string}` | `~M${string}` | `~N${string}` | `~O${string}` | `~P${string}` | `~Q${string}` | `~R${string}` | `~S${string}` | `~T${string}` | `~U${string}` | `~V${string}` | `~W${string}` | `~X${string}` | `~Y${string}` | `~Z${string}` | `[${number}${string}` | `[i${string}` | `[a${string}` | `[b${string}` | `[c${string}` | `[d${string}` | `[e${string}` | `[f${string}` | `[g${string}` | `[h${string}` | `[j${string}` | `[k${string}` | `[l${string}` | `[m${string}` | `[n${string}` | `[o${string}` | `[p${string}` | `[q${string}` | `[r${string}` | `[s${string}` | `[t${string}` | `[u${string}` | `[v${string}` | `[w${string}` | `[x${string}` | `[y${string}` | `[z${string}` | `[I${string}` | `[A${string}` | `[B${string}` | `[C${string}` | `[D${string}` | `[E${string}` | `[F${string}` | `[G${string}` | `[H${string}` | `[J${string}` | `[K${string}` | `[L${string}` | `[M${string}` | `[N${string}` | `[O${string}` | `[P${string}` | `[Q${string}` | `[R${string}` | `[S${string}` | `[T${string}` | `[U${string}` | `[V${string}` | `[W${string}` | `[X${string}` | `[Y${string}` | `[Z${string}` | `${number}${string}` | `i${string}` | `a${string}` | `b${string}` | `c${string}` | `d${string}` | `e${string}` | `f${string}` | `g${string}` | `h${string}` | `j${string}` | `k${string}` | `l${string}` | `m${string}` | `n${string}` | `o${string}` | `p${string}` | `q${string}` | `r${string}` | `s${string}` | `t${string}` | `u${string}` | `v${string}` | `w${string}` | `x${string}` | `y${string}` | `z${string}` | `I${string}` | `A${string}` | `B${string}` | `C${string}` | `D${string}` | `E${string}` | `F${string}` | `G${string}` | `H${string}` | `J${string}` | `K${string}` | `L${string}` | `M${string}` | `N${string}` | `O${string}` | `P${string}` | `Q${string}` | `R${string}` | `S${string}` | `T${string}` | `U${string}` | `V${string}` | `W${string}` | `X${string}` | `Y${string}` | `Z${string}`

The selector to use to extract the text from the document. Defaults to "body".

Optional textDecoder

textDecoder?: TextDecoder

The text decoder to use to decode the response. Defaults to UTF-8.

Methods

load

  • load(): Promise<Document[]>

  • Extracts the text content from the loaded document using the selector and creates a Document instance with the extracted text and metadata. It returns an array of Document instances.

    Returns Promise<Document[]>

    A Promise that resolves to an array of Document instances.

loadAndSplit

  • loadAndSplit(splitter?): Promise<Document[]>

  • Loads the documents and splits them using a specified text splitter.

    Parameters

    • splitter: TextSplitter = ...

    Returns Promise<Document[]>

    A Promise that resolves with an array of Document instances, each split according to the provided TextSplitter.

parseSitemap

scrape

Static imports

  • imports(): Promise<{
        load: ((content, options?, isDocument?) => CheerioAPI);
    }>

  • A static method that dynamically imports the Cheerio library and returns the load function. If the import fails, it throws an error.

    Returns Promise<{
        load: ((content, options?, isDocument?) => CheerioAPI);
    }>

    A Promise that resolves to an object containing the load function from the Cheerio library.

Static scrapeAll

  • scrapeAll(urls, caller, timeout, textDecoder?, options?): Promise<CheerioAPI[]>

  • Fetches web documents from the given array of URLs and loads them using Cheerio. It returns an array of CheerioAPI instances.

    Parameters

    • urls: string[]

      An array of URLs to fetch and load.

    • caller: AsyncCaller
    • timeout: undefined | number
    • Optional textDecoder: TextDecoder
    • Optional options: CheerioOptions

    Returns Promise<CheerioAPI[]>

    A Promise that resolves to an array of CheerioAPI instances.

Settings

Member Visibility

Theme

On This Page

Generated using TypeDoc