DOCSEARCHbyAlgolia
Submit Docs

Ksoup Documentation

repository·release·Indexed 20 days ago

https://github.com/fleeksoft/ksoup

A Kotlin Multiplatform library for parsing and manipulating HTML and XML, ported from the Java library jsoup. Ksoup supports Android, JVM, Native, and JS platforms. It provides core parsing functionality, I/O extensions for kotlinx-io and okio, and network extensions for Ktor 3 to fetch and parse content from URLs. Key features include CSS selectors, metadata extraction, and a StreamParser for progressive parsing of large documents.

Tokens
5.1K
Snippets
22
Records
26
Agent score
68%

What's inside Ksoup

  1. Install Ksoup Network Extensions

    release

    To fetch and parse HTML/XML directly from URLs, add a network extension.

    Note: ksoup-network-ktor2 is deprecated. Use the Ktor 3 based extension instead.

    Ktor 3 (Recommended)

    implementation("com.fleeksoft.ksoup:ksoup-network:<version>")

    This extension provides Ksoup.parseGetRequest, Ksoup.parseSubmitRequest, and Ksoup.parsePostRequest (both suspend and blocking versions).

    implementation("com.fleeksoft.ksoup:ksoup-network:<version>")

  2. Install Ksoup Core Library

    release

    To parse HTML or XML from strings, include the core library in your commonMain dependencies. This is the base requirement for all Ksoup functionality.

    // Required core library
implementation("com.fleeksoft.ksoup:ksoup:<version>")
    implementation("com.fleeksoft.ksoup:ksoup:<version>")

  3. Install Ksoup I/O Extensions

    release

    If you need to parse HTML/XML from files or other input sources, add one of the following I/O extensions. These provide APIs like Ksoup.parseFile and Ksoup.parseSource.

    Option 1: kotlinx-io (Recommended)

    implementation("com.fleeksoft.ksoup:ksoup-kotlinx:<version>")

    Option 2: okio

    implementation("com.fleeksoft.ksoup:ksoup-okio:<version>")
    implementation("com.fleeksoft.ksoup:ksoup-kotlinx:<version>")

  4. How StreamParser works for progressive parsing

    release

    A StreamParser provides a progressive parse of its input. As each Element is completed, it is emitted via a Sequence or Iterator. Elements are returned in document order, meaning child elements are returned prior to their parents. Each returned Element is complete with all its children and an (empty) next sibling, if applicable.

    Memory Management

    To conserve memory when parsing extremely large documents, you can call Node.remove() on elements (or their children) during the parse. This allows you to process a document that would otherwise exceed available memory by removing parts of the DOM once they are no longer needed.

    Suspended Selection

    The parser supports selectFirst(query) and selectNext(query), which run the parser until a match is found, at which point the parse is suspended. You can resume parsing via subsequent select() calls or by continuing to consume the stream() or iterator().

  5. Parse XML

    release

    To parse XML content, use Ksoup.parse and specify an xmlParser from the Parser class.

    val doc: Document = Ksoup.parse(xml, parser = Parser.xmlParser())
    val doc: Document = Ksoup.parse(xml, parser = Parser.xmlParser())

  6. Extract Metadata from a Website

    release

    Ksoup provides a high-level way to extract common metadata (like OpenGraph and Twitter tags) using Ksoup.parseMetaData. This requires the ksoup-network extension if fetching from a URL.

    // Fetching from URL
val doc: Document = Ksoup.parseGetRequest(url = "https://en.wikipedia.org/")
val metadata: Metadata = Ksoup.parseMetaData(element = doc)

// Or parsing from an existing HTML string
// val metadata: Metadata = Ksoup.parseMetaData(html = HTML)

println("title: ${metadata.title}")
println("description: ${metadata.description}")
println("ogTitle: ${metadata.ogTitle}")
println("ogDescription: ${metadata.ogDescription}")
println("twitterTitle: ${metadata.twitterTitle}")
println("twitterDescription: ${metadata.twitterDescription}")
    val doc: Document = Ksoup.parseGetRequest(url = "https://en.wikipedia.org/")
val metadata: Metadata = Ksoup.parseMetaData(element = doc)

println("title: ${metadata.title}")
println("description: ${metadata.description}")

  7. Parse HTML from a String

    release

    Use Ksoup.parse to convert an HTML string into a Document object. You can then use DOM traversal or CSS selectors to extract data.

    val html = "<html><head><title>One</title></head><body>Two</body></html>"
val doc: Document = Ksoup.parse(html = html)

println(doc.title()) // One
println(doc.body().text()) // Two
    val html = "<html><head><title>One</title></head><body>Two</body></html>"
val doc: Document = Ksoup.parse(html = html)

println(doc.title())
println(doc.body().text())

  8. Fetch and Parse HTML from a URL

    release

    To fetch content from a URL, use the ksoup-network extension. You can use either a suspend function for asynchronous execution or a Blocking version.

    Note: Ksoup.parseGetRequest is a suspend function.

    // Using suspend function
val doc: Document = Ksoup.parseGetRequest(url = "https://en.wikipedia.org/")

// Using blocking function
val doc: Document = Ksoup.parseGetRequestBlocking(url = "https://en.wikipedia.org/")

// Extracting data with CSS selectors
val headlines: Elements = doc.select("#mp-itn b a")
headlines.forEach { headline: Element ->
    val title = headline.attr("title")
    val link = headline.absUrl("href")
    println("$title => $link")
}
    val doc: Document = Ksoup.parseGetRequest(url = "https://en.wikipedia.org/")
val headlines: Elements = doc.select("#mp-itn b a")

headlines.forEach { headline: Element ->
    val headlineTitle = headline.attr("title")
    val headlineLink = headline.absUrl("href")
    println("$headlineTitle => $headlineLink")
}

  9. Ksoup Network API Reference

    release

    Provided by the ksoup-network extension.

    Suspend Functions:

    • Ksoup.parseGetRequest
    • Ksoup.parseSubmitRequest
    • Ksoup.parsePostRequest

    Blocking Functions:

    • Ksoup.parseGetRequestBlocking
    • Ksoup.parseSubmitRequestBlocking
    • Ksoup.parsePostRequestBlocking

  10. Ksoup I/O API Reference

    release

    Provided by ksoup-kotlinx or ksoup-okio extensions:

    • Ksoup.parseInput(input: InputStream, baseUri: String, charsetName: String? = null, parser: Parser = Parser.htmlParser())
    • Ksoup.parseFile
    • Ksoup.parseSource

  11. Ksoup Core API Reference

    release

    The core library provides the following primary functions for parsing and cleaning HTML/XML:

    • Ksoup.parse(html: String, baseUri: String = ""): Document
    • Ksoup.parse(html: String, parser: Parser, baseUri: String = ""): Document
    • Ksoup.parse(reader: Reader, parser: Parser, baseUri: String = ""): Document
    • Ksoup.clean(bodyHtml: String, safelist: Safelist = Safelist.relaxed(), baseUri: String = "", outputSettings: Document.OutputSettings? = null): String
    • Ksoup.isValid(bodyHtml: String, safelist: Safelist = Safelist.relaxed()): Boolean

  12. Manage StreamParser lifecycle with stop() and close()

    release

    Properly managing the lifecycle of a StreamParser is important to release I/O resources.

    stop()

    Flags that the parse should be stopped. The backing iterator will not return any more elements. This is useful if you have found everything you need and want to prevent further reading.

    close()

    Closes the input and releases resources, including the underlying parser and reader. This should be called to ensure the Reader is properly closed.

    use(block)

    An extension-style method that executes a block of code with the StreamParser and automatically calls close() when the block finishes.

    // Using the use() pattern for safe resource management
streamParser.use { parser ->
    val firstDiv = parser.selectFirst("div")
    // parser.close() is called automatically after this block
}

// Manual stop and close
streamParser.stop()
streamParser.close()