# sttp: the Scala HTTP client you always wanted! Welcome! sttp client is an open-source HTTP client for Scala, supporting various approaches to writing Scala code: synchronous (direct-style), `Future`-based, and using functional effect systems (cats-effect, ZIO, Monix, Kyo, scalaz). The library is available for Scala 2.12, 2.13 and 3. Supported platforms are the JVM (Java 11+), Scala.JS and Scala Native. Here's a quick example of sttp client in action, runnable using [scala-cli](https://scala-cli.virtuslab.org): ```scala //> using dep com.softwaremill.sttp.client4::core::4.0.25 import sttp.client4.quick.* @main def run(): Unit = println(quickRequest.get(uri"http://httpbin.org/ip").send()) ``` sttp client addresses common HTTP client use cases, such as interacting with JSON APIs (with automatic serialization of request bodies and deserialization of response bodies), uploading and downloading files, submitting form data, handling multipart requests, and working with WebSockets. The driving principle of sttp client's design is to provide a clean, programmer-friendly API to describe HTTP requests, along with response handling. This ensures that resources, such as HTTP connections, are used safely, also in the presence of errors. sttp client integrates with a number of lower-level Scala and Java HTTP client implementations through backends (using Java's `HttpClient`, Akka HTTP, Pekko HTTP, http4s, OkHttp, Armeria), offering a wide range of choices when it comes to protocol support, connectivity settings and programming stack compatibility. Additionally, sttp client seamlessly integrates with popular libraries for JSON handling (e.g., circe, uPickle, jsoniter, json4s, play-json, ZIO Json), logging, metrics, and tracing (e.g., slf4j, scribe, OpenTelemetry, Prometheus). It also supports streaming libraries (e.g., fs2, ZIO Streams, Akka Streams, Pekko Streams) and provides tools for testing HTTP interactions. Some more features: URI interpolation, a self-managed backend, and type-safe HTTP error/success representation, are demonstrated by the below example: ```scala //> using dep com.softwaremill.sttp.client4::core::4.0.25 import sttp.client4.* @main def sttpDemo(): Unit = val sort: Option[String] = None val query = "http language:scala" // the `query` parameter is automatically url-encoded // `sort` is removed, as the value is not defined val request = basicRequest.get( uri"https://api.github.com/search/repositories?q=$query&sort=$sort") val backend = DefaultSyncBackend() val response = request.send(backend) // response.header(...): Option[String] println(response.header("Content-Length")) // since we're using basicRequest, the response.body is read into an // Either[String, String] to indicate failure or success println(response.body) ``` But that's just a small glimpse of sttp client's features! For more examples, see the [usage examples](examples.md) section. To start using sttp client in your project, see the [quickstart](quickstart.md). Or, browse the documentation to find the topics that interest you the most! ScalaDoc is available at [https://www.javadoc.io](https://www.javadoc.io/doc/com.softwaremill.sttp.client4/core_2.12/4.0.25). sttp client is licensed under Apache2, the source code is [available on GitHub](https://github.com/softwaremill/sttp). ## Other sttp projects sttp is a family of Scala HTTP-related projects, and currently includes: * sttp client: this project * [sttp tapir](https://github.com/softwaremill/tapir): rapid development of self-documenting APIs * [sttp model](https://github.com/softwaremill/sttp-model): simple HTTP model classes (used by client & tapir) * [sttp shared](https://github.com/softwaremill/sttp-shared): shared web socket, FP abstractions, capabilities and streaming code. * [sttp apispec](https://github.com/softwaremill/sttp-apispec): OpenAPI, AsyncAPI and JSON Schema models. * [sttp openai](https://github.com/softwaremill/sttp-openai): Scala client wrapper for OpenAI and OpenAI-compatible APIs. Use the power of ChatGPT inside your code! Third party projects: * [sttp-oauth2](https://github.com/polyvariant/sttp-oauth2): OAuth2 client library for Scala ## Try sttp client in your browser! [Check out & play with a simple example on Scastie!](https://scastie.scala-lang.org/adamw/aOf32MZsTPesobwfWG5nDQ) # Table of contents ```{eval-rst} .. toctree:: :maxdepth: 2 :caption: Getting started quickstart how support goals community .. toctree:: :maxdepth: 2 :caption: How-to's examples migrate_v3_v4 .. toctree:: :maxdepth: 2 :caption: HTTP model model/model model/uri .. toctree:: :maxdepth: 2 :caption: Request definition requests/basics requests/headers requests/cookies requests/authentication requests/body requests/multipart requests/streaming requests/type .. toctree:: :maxdepth: 2 :caption: Responses responses/basics responses/body responses/exceptions .. toctree:: :maxdepth: 2 :caption: Other topics other/websockets other/json other/xml other/resilience other/openapi other/sse other/body_callbacks .. toctree:: :maxdepth: 2 :caption: Backends backends/summary backends/start_stop backends/synchronous backends/akka backends/pekko backends/future backends/monix backends/catseffect backends/fs2 backends/scalaz backends/zio backends/http4s backends/finagle backends/javascript/fetch backends/native/curl .. toctree:: :maxdepth: 2 :caption: Backend wrappers backends/wrappers/opentelemetry backends/wrappers/prometheus backends/wrappers/logging backends/wrappers/cache backends/wrappers/custom .. toctree:: :maxdepth: 2 :caption: Testing testing/stub testing/curl .. toctree:: :maxdepth: 2 :caption: Configuration conf/timeouts conf/ssl conf/proxy conf/redirects .. toctree:: :maxdepth: 2 :caption: More information other ``` # Quickstart The core sttp client API comes in a single jar, with a transitive dependency on [sttp model](https://github.com/softwaremill/sttp-model). This also includes [synchronous](backends/synchronous.md) and [`Future`-based](backends/future.md) backends, based on Java's `HttpClient`. To integrate with other parts of your application and various effect systems, you'll often need to use an alternate backend, or backend wrappers (but what's important is that the API remains the same!). See the section on [backends](backends/summary.md) for a short guide on which backend to choose, and a list of all implementations. sttp client is available for Scala 2.12, 2.13 and 3, on the JVM (Java 11+), Scala.JS and Scala Native platforms. Note that not all modules are compatible with these platforms, and that each has its own dedicated set of backends. ## Using sbt The basic dependency which provides the API, together with a synchronous and `Future`-based backends, is: ```scala "com.softwaremill.sttp.client4" %% "core" % "4.0.25" ``` ## Using scala-cli Add the following directive to the top of your scala file to add the core sttp dependency: ``` //> using dep com.softwaremill.sttp.client4::core:4.0.25 ``` ## Using Ammonite If you are an [Ammonite](https://ammonite.io) user, you can quickly start experimenting with sttp by copy-pasting the following: ```scala import $ivy.`com.softwaremill.sttp.client4::core:4.0.25` ``` ## Imports Working with sttp is most convenient if you import the `sttp.client4` package entirely: ```scala import sttp.client4.* ``` This brings into scope the starting point for defining requests (`basicRequest`) and some helper methods. All examples in this guide assume that this import is in place. ## Synchronous requests And that's all you need to start using sttp client! To create and send your first request, import the above, type `basicRequest.` and see where your IDE's auto-complete gets you! Here's a simple request, using the synchronous backend: ```scala import sttp.client4.* val backend = DefaultSyncBackend() val response = basicRequest .body("Hello, world!") .post(uri"https://httpbin.org/post?hello=world") .send(backend) println(response.body) ``` Creating a backend allocates resources (such as selector threads / connection pools), so when it's no longer needed, it should be closed using `.close()`. Typically, you should have one backend instance for your entire application. In the example above, the `response.body` is an `Either[String, String]`. A left value indicates HTTP error (4xx or 5xx) response, while a right value indicates HTTP success (2xx). In case of connection errors, an exception is thrown. You can customize how the response body is handled using [response body handling descriptions](responses/body.md). ## Serializing and parsing JSON To serialize a custom type to a JSON body, or to deserialize the response body that is in the JSON format, you'll need to add an integration with a JSON library. See [json](other/json.md) for a list of available libraries. As an example, to integrate with the [uPickle](https://github.com/com-lihaoyi/upickle) library, add the following dependency: ```scala "com.softwaremill.sttp.client4" %% "upickle" % "4.0.25" ``` Your code might then look as follows: ```scala //> using dep com.softwaremill.sttp.client4::core:4.0.25 //> using dep com.softwaremill.sttp.client4::upickle:4.0.25 import sttp.client4.* import sttp.client4.upicklejson.default.* import upickle.default.* @main def run(): Unit = val backend = DefaultSyncBackend() case class MyRequest(field1: String, field2: Int) // selected fields from the JSON that is being returned by httpbin case class HttpBinResponse(origin: String, headers: Map[String, String]) given ReadWriter[MyRequest] = macroRW[MyRequest] given ReadWriter[HttpBinResponse] = macroRW[HttpBinResponse] val request = basicRequest .post(uri"https://httpbin.org/post") .body(asJson(MyRequest("test", 42))) .response(asJson[HttpBinResponse]) val response = request.send(backend) response.body match { case Left(e) => println(s"Got response exception:\n$e") case Right(r) => println(s"Origin's ip: ${r.origin}, header count: ${r.headers.size}") } ``` Similarly as above, since we've used the `asJson[HttpBinResponse]` response description, `response.body` is an `Either`. However here the left-side can indicate both an HTTP error, or a deserialization error. This is reflected in the type of the `response.body` value. Alternatively, you can use the `asJsonOrFail` response description, so that in case of any error, an exception is thrown. ## Adding logging Logging can be added using the [logging backend wrapper](backends/wrappers/logging.md). For example, if you'd like to use slf4j, you'll need the following dependency: ``` "com.softwaremill.sttp.client4" %% "slf4j-backend" % "4.0.25" ``` Then, you'll need to configure your backend: ```scala import sttp.client4.* import sttp.client4.logging.slf4j.Slf4jLoggingBackend val backend = Slf4jLoggingBackend(DefaultSyncBackend()) ``` Any requests sent using the backend will now be logged using slf4j! ## Even quicker You can skip the step of creating a backend instance, by using `import sttp.client4.quick.*` instead of the usual `import sttp.client4.*`. This brings into scope the same sttp API, and additionally a pre-configured synchronous backend instance, which can be used to send requests. This backend instance is global (created on first access), can't be customized and shouldn't be closed. The `send()` extension method allows sending requests using that `backend` instance: ```scala import sttp.client4.quick.* quickRequest.get(uri"http://httpbin.org/ip").send() ``` Additionally, above we're using `quickRequest`, instead of `basicRequest`, to build the request description. `quickRequest` is pre-configured to always read HTTP responses as a `String`, regardless of the status code. You can read more about the initial request definitions [here](requests/basics.md). ## Next steps Next, read on [how sttp client works](how.md) or see some [examples](examples.md). # How sttp client works ## Describe the request This first step when using sttp client is describing the request that you'd like to send. A request is represented as an immutable data structure of type `Request`. The initial "empty" request is provided as the `basicRequest` value, in the `sttp.client4` package. It can be refined using one of the available methods, such as `.header`, `.body`, `.get(Uri)`, `.responseAs`, etc. A `Request[T]` value contains both information on what to include in the request, but also how to handle the response body. The `T` type parameter defines the type, to which the response will be read. To start describing a request, import the sttp client package and customize `basicRequest`: ```scala import sttp.client4.* val myRequest: Request[String] = ??? // basicRequest.(...) ``` An alternative to importing the `sttp.client4.*` package, is to extend the `sttp.client4.SttpApi` trait. That way, multiple integrations can be grouped in one object, thus reducing the number of necessary imports. ## Send the request Once the request is described as a value, it can be sent. To send a request, you'll need a `Backend`. The backend is where most of the work happens: the request is translated to a backend-specific form; a connection is opened, data sent and received; finally, the backend-specific response is translated to sttp's `Response[T]`, as described in the request. A backend can be synchronous, that is, sending a request can be a blocking operation. When invoking `myRequest.send(backend)`, you'll get a value of type `Response[T]`. Backends can also be asynchronous, and evaluate the send operation eagerly or lazily. For example, when using the [Akka backend](backends/akka.md), `myRequest.send(backend)` will return a `Future[Response[T]]`: an eagerly-evaluated, asynchronous result. When using a [cats-effect backend](backends/catseffect.md), you'll get back a `F[Response[T]]`: a lazily-evaluated, but also non-blocking and asynchronous result. Backends manage the connection pool, thread pools for handling responses, depending on the implementation provide various configuration options, and optionally support [streaming](requests/streaming.md) and [websockets](other/websockets.md). They typically need to be created upon application startup, and closed when the application terminates. For example, the following sends a synchronous request, using the default JVM backend: ```scala import sttp.client4.* val myRequest: Request[String] = ??? val backend = DefaultSyncBackend() val response = myRequest.send(backend) ``` ## Next steps Read more about: * [describing the request](requests/basics.md) * the [`Request` type](requests/type.md) * specifying how to handle the [response body](responses/body.md) * [available backends](backends/summary.md) # Support & sponsorship ## Sponsors Development and maintenance of sttp client is sponsored by [SoftwareMill](https://softwaremill.com), a software development and consulting company. We help clients scale their business through software. We offer services around migrating and maintaining Java and Scala projects (e.g. to Java 21, or across Scala versions), ML/AI discovery workshops, introducing developer platforms (based on Kubernetes and observability technologies), and others. Our areas of expertise include performant backends, distributed systems, machine learning and data analytics, with a focus on Java, Scala, Kafka, TypeScript and Rust. [![](https://files.softwaremill.com/logo/logo.png "SoftwareMill")](https://softwaremill.com) ## Commercial Support We offer commercial support for sttp and related technologies, as well as development services. [Contact us](https://softwaremill.com/contact/) to learn more about our offer! # Goals of the project * provide a simple, discoverable, no-surprises, reasonably type-safe API for making HTTP requests and reading responses * separate definition of a request from request execution * provide immutable, easily modifiable data structures for requests and responses * support multiple execution backends, both synchronous and asynchronous * provide support for backend-specific request/response streaming * minimum dependencies See also the blog posts: * [Introduction to sttp](https://softwaremill.com/introducing-sttp-the-scala-http-client) * [sttp streaming & URI interpolators](https://softwaremill.com/sttp-streaming-uri-interpolator) * [sttp2: an overview of proposed changes](https://blog.softwaremill.com/sttp2-an-overview-of-proposed-changes-8de23c94684f) * [Migrating to sttp client 2.x and tapir 0.12.x](https://blog.softwaremill.com/migrating-to-sttp-client-2-x-and-tapir-0-12-x-7956e6c79c52) * [What’s coming up in sttp client 3?](https://blog.softwaremill.com/whats-coming-up-in-sttp-client-3-30d01ab42d1b) * [sttp client 3 is here!](https://blog.softwaremill.com/sttp-client-3-is-here-bb9ebe925931) ## Non-goals of the project * implement a full HTTP client. Instead, sttp client wraps existing HTTP clients, providing a consistent, programmer-friendly API. All network-related concerns such as sending the requests, connection pooling, receiving responses are delegated to the chosen backend * provide ultimate flexibility in defining the request. While it's possible to define *most* valid HTTP requests, e.g. some of the less common body chunking approaches aren't available ## How is sttp different from other libraries? * immutable request builder which doesn't impose any order in which request parameters need to be specified. Such an approach allows defining partial requests with common cookies/headers/options, which can later be specialized using a specific URI and HTTP method. * integration with various Scala programming stacks and libraries * support for multiple backends, both synchronous and asynchronous, with backend-specific streaming support * URI interpolator with context-aware escaping, optional parameters support and parameter collections * description of how to handle the response is combined with the description of the request to send # Community If you have a question, suggestion, or hit a problem, feel free to ask on our [discourse forum](https://softwaremill.community/c/sttp-client)! Or, if you encounter a bug, something is unclear in the code or documentation, don't hesitate and [open an issue](https://github.com/softwaremill/sttp/issues) on GitHub. We are also always looking for contributions and new ideas, so if you'd like to get into the project, check out the open issues, or post your own suggestions! # Examples by category The sttp client repository contains a number of how-to guides. If you're missing an example for your use-case, please let us know by [reporting an issue](https://github.com/softwaremill/sttp)! Each example is fully self-contained and can be run using [scala-cli](https://scala-cli.virtuslab.org) (you just need to copy the content of the file, apart from scala-cli, no additional setup is required!). Hopefully this will make experimenting with sttp client as frictionless as possible! Examples are tagged with the stack being used (direct-style, cats-effect, ZIO, Future) and backend implementation ```{eval-rst} ## Hello, World! * [Dynamic URI components](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/dynamicUriSynchronous.scala) HttpClient Direct * [POST form data](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/PostFormSynchronous.scala) HttpClient Direct * [POST multipart form](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/postMultipartFormSynchronous.scala) HttpClient Direct * [Post JSON data](https://github.com/softwaremill/sttp/tree/master/examples-ce2/src/main/scala/sttp/client4/examples/PostSerializeJsonMonixHttpClientCirce.scala) HttpClient Monix * [Upload file](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/fileUploadSynchronous.scala) HttpClient Direct ## Backend wrapper * [A backend which adds a header to all outgoing requests](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/wrapper/addHeaderBackend.scala) HttpClient Synchronous * [Integrate with resilience4j to implement circuit-breaking](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/wrapper/CircuitBreakerCatsEffect.scala) HttpClient cats-effect * [Integrate with resilience4j to implement rate-limiting](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/wrapper/rateLimiterFuture.scala) HttpClient Future * [Simple retrying backend wrapper](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/wrapper/retryingBackend.scala) HttpClient Synchronous * [Use the caching backend wrapper with Redis](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/wrapper/redisCachingBackend.scala) HttpClient Synchronous ## Error handling * [HTTP error handling using basicRequest](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/errors/httpErrorHandlingUsingBasicRequest.scala) HttpClient Direct * [HTTP error handling, adjusting the response description](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/errors/httpErrorHandlingAdjustResponse.scala) HttpClient Direct * [Parsing the response as JSON, with parsing failures and HTTP errors](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/errors/httpErrorHandlingJson.scala) HttpClient Direct ## JSON * [Receive & parse JSON using ZIO Json](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/json/GetAndParseJsonZioJson.scala) HttpClient ZIO * [Receive & parse JSON using circe](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/json/GetAndParseJsonCatsEffectCirce.scala) HttpClient cats-effect * [Receive & parse JSON using circe](https://github.com/softwaremill/sttp/tree/master/examples-ce2/src/main/scala/sttp/client4/examples/GetAndParseJsonOrFailMonixCirce.scala) HttpClient Monix * [Receive & parse JSON using json4s](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/json/getAndParseJsonPekkoHttpJson4s.scala) Pekko Future * [Receive & parse JSON using jsoniter](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/json/getAndParseJsonSynchronousJsoniter.scala) HttpClient Direct ## Logging * [A backend wrapper which logs the response body as a string](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/logging/logAsStringBackend.scala) HttpClient Synchronous * [Add a logging backend wrapper, which uses slf4j](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/logging/logRequestsSlf4j.scala) HttpClient Direct ## Observability * [Report metrics to a cloud service](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/observability/metricsWrapperPekkoHttp.scala) Pekko Future * [Use the OpenTelemetry tracing & metrics wrappers](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/observability/openTelemetryTracingAndMetrics.scala) HttpClient Synchronous ## Other * [Command output streaming with os-lib support](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/other/cmdOutputStreamingWithOsLib.scala) HttpClient Direct * [Download file with os-lib support](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/other/uploadFileWithOsLib.scala) HttpClient Direct * [Download file with os-lib support](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/other/downloadFileWitOsLib.scala) HttpClient Direct * [Handle the body by both parsing it to JSON and returning the raw string](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/other/GetRawResponseBodySynchronous.scala) HttpClient Direct ## Resilience * [Rate limit sending requests using Ox](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/resilience/RateLimitOx.scala) HttpClient Direct * [Retry sending a request using Ox](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/resilience/RetryOx.scala) HttpClient Direct * [Retry sending a request using ZIO's retries](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/resilience/RetryZio.scala) HttpClient ZIO ## Streaming * [Stream request & response bodies using Ox's Flow (synchronous, blocking streams)](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/streaming/streamOx.scala) HttpClient Direct * [Stream request & response bodies using ZIO-Streams](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/streaming/StreamZio.scala) HttpClient ZIO * [Stream request & response bodies using fs2](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/streaming/StreamFs2.scala) HttpClient cats-effect ## Testing * [Create a backend stub which simulates interactions using multiple query parameters](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/testing/TestEndpointMultipleQueryParameters.scala) * [Create a backend stub which simulates interactions with a WebSocket](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/testing/WebSocketTesting.scala) HttpClient cats-effect ## WebSocket * [Connect to & interact with a WebSocket](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/ws/WebSocketPekko.scala) Pekko Future * [Connect to & interact with a WebSocket](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/ws/WebSocketZio.scala) HttpClient ZIO * [Connect to & interact with a WebSocket](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/ws/WebSocketSynchronous.scala) HttpClient Direct * [Connect to & interact with a WebSocket](https://github.com/softwaremill/sttp/tree/master/examples-ce2/src/main/scala/sttp/client4/examples/WebSocketMonix.scala) HttpClient Monix * [Connect to & interact with a WebSocket, using Ox channels for streaming](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/ws/wsOxExample.scala) HttpClient Direct * [Connect to & interact with a WebSocket, using fs2 streaming](https://github.com/softwaremill/sttp/tree/master/examples/src/main/scala/sttp/client4/examples/ws/WebSocketStreamFs2.scala) HttpClient cats-effect :parser: markdown ``` # Migrating to sttp-client4 ## Top-level package The top-level package for sttp-client4 is `sttp.client4`. This means that sttp-client3 and sttp-client4 can be used side-by-side in the same project. They do share sttp-model and sttp-shared libraries, but these did not see a major version change, and are designed to be binary-compatible. ## Request & backend type changes The `RequestT` type is retired, being replaced by `PartialRequest` and `GenericRequest`. `PartialRequest` is the type of requests before the method & uri are set. As before, it allows setting the headers and body, creating a reusable base for defining full requests. After the uri & method are set, the request description becomes a `Request`, which can be sent. Additionally, methods to set the body or handle the response as a non-blocking, asynchronous stream become available, as well as converting the request to a web socket one. This yields requests of type `StreamRequest`, `WebSocketRequest` and `WebSocketStreamRequest`. Hence, `GenericRequest` is never used directly in user code. A parallel change is from a fully-parametrized `SttpBackend` to a family of traits: `SyncBackend`, `Backend`, `StreamBackend`, `WebSocketBackend`, `WebSocketSyncBackend`, `WebSocketStreamBackend`. These specialize the backend type to the capabilities they support, and are used to send requests of the corresponding type. These changes are introduced to simplify the types, improve error reporting and enhance IDE completions. As a tradeoff, defining generic backend wrappers requires defining constructors for each of the backend subtype. Moreover, only `request.send(backend)` should be used, instead of `backend.send(request)`; both work, but the first variant is more IDE-friendly, and for the sake of consistency, it's the encouraged one, and the only one used in the documentation. ## Removed implicit `BodySerializer` All request bodies have now to be explicitly converted to a `BasicBody` or one of the basic types (`String`, `InputStream`, `File`, `Array[Byte]`, `ByteBuffer`, `Map[String, String]`). This change has the highest impact when it comes to JSON bodies, which now have to be set using the `asJson` method. That is, importing JSON integration (e.g. through `import sttp.client4.jsoniter.*`), brings into scope both `asJson` response handling descriptions, as well as a `asJson(T): BasicBody` methods. A body can be set on a request using `request.body(asJson(...))`. Previously, the high level type -> `BasicBody` conversion was implicit, and the JSON integrations contained implicit conversions from high-level types to the JSON string (provided the library-specific encoders were in scope). While this design did save some keystrokes, it also provided poor error reporting, and the format to which the body was converted was not always clear when reading the code. This motivated the change to explicitly calling body conversions. ## `...OrFailed` response handling New response handling descriptions are added, which fail (throw an exception / return a failed effect) if the response is not successful (2xx status code). These include `stringOrFail`, `asByteArrayOrFail`, `asJsonOrFail` etc. Note that this is different from e.g. `asStringAlways`, which always handles the response as a string, regardless of the status code. Any `Either`-based response description can be converted to a failing one using `.orFail` and `.orFailDeserialization`. This replaces the `.getRight` / `.getEither` extension methods. ## Backend stub changes When defining responses in a backend stub, it's now possible to describe more precisely, if the given body should be adjusted to what's defined in the request's response description, or returned as-is. Instead of `.thenRespond` methods, there are now `.thenRespondAdjust` and `.thenRespondExact` methods. In most use-cases adjustments are used, hence when migrating any `.thenRespond` invocations should be updated to `.thenRespondAdjust`. If more control over the returned body is needed, `Response[StubBody]` instances can be created using `ResponseStub.adjust` or `ResponseStub.exact` methods. Note that in most cases `ResponseStub.ok` and `ResponseStub.apply` methods should **not** be used when defining the behavior of the backend stub. However, they might be used in other scenarios. ## Other changes * `BackendOptions` replaces `SttpBackendOptions` * `Backend.monad` replaces `SttpBackend.responseMonad` * `DefaultSyncBackend` and `DefaultHttpBackend` are provided. They allow limited customization, but are a good entry-level backend for many use-cases. * `HttpClientBackend` is move to a dedicated package * `SimpleHttpClient` is removed, `import sttp.client4.quick.*` provides a default backend instance with a no-arg `request.send()` extension method * documentation & examples use Scala 3 * the `autoDecompressionDisabled` option is superseded by `autoDecompressionEnabled` * `async-http-client` backends are removed (as there's no reactive streams support in v3, making integration difficult) * request attributes replace request tags (same mechanism as in Tapir) * the parametrization of `ResponseException` is simplified, `DeserializationException` does not have a type parameter, always requiring an `Exception` as the cause instead * when a `ResponseException` is thrown in the response handling specification, this will be logged as a successful response (as the response was received correctly), and counted as a success in the metrics as well * `HttpError` is renamed to `UnexpectedStatusCode`, and along with `DeserializationException`, both types are nested within `ResponseException` * the `opentelemetry-metrics-backend` module is renamed to `opentelemetry-backend` * `ListenerBackend` has been refactored, with improvements in naming, and additional callback methods # Model classes [sttp model](https://github.com/softwaremill/sttp-model) is a stand-alone project which provides a basic HTTP model, along with constants for common HTTP header names, media types, and status codes. The basic model classes are: `Header`, `Cookie`, `CookieWithMeta`, `MediaType`, `Method`, `StatusCode`, `CacheDirective`, `ETag` and `Uri`. The `.toString` methods of these classes returns a representation as in a HTTP request/response. See the ScalaDoc for more information. Companion objects provide methods to construct model class instances, following these rules: * `.parse(serialized: String): Either[String, ModelClass]`: returns an error message, or an instance of the model class * `.unsafeParse(serialized: String): Sth`: returns an instance of the model class or in case of an error, throws an exception. * `.unsafeApply(values)`: creates an instance of the model class; validates the input values and in case of an error, throws an exception. An error could be e.g. that the input values contain characters outside the allowed range * `.safeApply(...): Either[String, ModelClass]`: same as above, but doesn't throw exceptions. Instead, returns an error message, or the model class instance * `.apply(...): ModelClass`: creates the model type, without validation, and without throwing exceptions Moreover, companion objects provide constants and/or constructor methods for well-know model class instances. For example, there's `StatusCode.Ok`, `Method.POST`, `MediaType.ImageGif` and `Header.contentType(MediaType)`. These constants are also available as traits: `StatusCodes`, `MediaTypes` and `HeaderNames`. The model also contains aggregate/helper classes such as `Headers` and `QueryParams`. Example with objects: ```scala import sttp.client4.* import sttp.model.* object Example: val request = basicRequest.header(Header.contentType(MediaType.ApplicationJson)) .get(uri"https://httpbin.org") val backend = DefaultSyncBackend() val response = request.send(backend) if response.code == StatusCode.Ok then println("Ok!") ``` Example with traits: ```scala import sttp.client4.* import sttp.model.* object Example extends HeaderNames with MediaTypes with StatusCodes: val request = basicRequest.header(ContentType, ApplicationJson.toString) .get(uri"https://httpbin.org") val backend = DefaultSyncBackend() val response = request.send(backend) if response.code == Ok then println("Ok!") ``` For more information see * [Wikipedia: list of http header fields](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields) * [Wikipedia: media type](https://en.wikipedia.org/wiki/Media_type) * [Wikipedia: list of http status codes](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) # URIs A request can only be sent if the request method & URI are defined. To represent URIs, sttp comes with a `Uri` case class, which captures all the parts of an address. To specify the request method and URI, use one of the methods on the request definition corresponding to the name of the desired HTTP method: `.post`, `.get`, `.put` etc. All of them accept a single parameter, the URI to which the request should be sent (these methods only modify the request definition; they don't send the requests). The `Uri` class is immutable, and can be constructed by hand, but in many cases the URI interpolator will be easier to use. ## URI interpolator Using the URI interpolator it's possible to conveniently create `Uri` instances, for example: ```scala import sttp.client4.* import sttp.model.* val user = "Mary Smith" val filter = "programming languages" val endpoint: Uri = uri"http://example.com/$user/skills?filter=$filter" assert(endpoint.toString == "http://example.com/Mary%20Smith/skills?filter=programming+languages") ``` Note the `uri` prefix before the string and the standard Scala string-embedding syntax (`$user`, `$filter`). Any values embedded in the URI will be URL-encoded, taking into account the context (e.g., the whitespace in `user` will be %-encoded as `%20D`, while the whitespace in `filter` will be query-encoded as `+`). On the other hand, parts of the URI given as literal strings (not embedded values), are assumed to be URL-encoded and thus will be decoded when creating a `Uri` instance. All components of the URI can be embedded from values: scheme, username/password, host, port, path, query and fragment. The embedded values won't be further parsed, except the `:` in the host part, which is commonly used to pass in both the host and port: ```scala import sttp.client4.* // the embedded / is escaped println(uri"http://example.org/${"a/b"}") // http://example.org/a%2Fb // the literal / is not escaped println(uri"http://example.org/${"a"}/${"b"}") // http://example.org/a/b // the embedded : is not escaped println(uri"http://${"example.org:8080"}") // http://example.org:8080 ``` Both the `Uri` class, and the interpolator can be used stand-alone, without using the rest of sttp. Conversions are available both from and to `java.net.URI`; `Uri.toString` returns the URI as a `String`. ## Optional values The URI interpolator supports optional values for hosts (subdomains), query parameters and the fragment. If the value is `None`, the appropriate URI component will be removed. For example: ```scala val v1 = None val v2 = Some("v2") ``` ```scala println(uri"http://example.com?p1=$v1&p2=v2") // http://example.com?p2=v2 println(uri"http://$v1.$v2.example.com") // http://v2.example.com println(uri"http://example.com#$v1") // http://example.com ``` ## Maps and sequences Maps, sequences of tuples and sequences of values can be embedded in the query part. They will be expanded into query parameters. Maps and sequences of tuples can also contain optional values, for which mappings will be removed if `None`. For example: ```scala val ps = Map("p1" -> "v1", "p2" -> "v2") ``` ```scala println(uri"http://example.com?$ps&p3=p4") // http://example.com?p1=v1&p2=v2&p3=p4 ``` Sequences in the host part will be expanded to a subdomain sequence, and sequences in the path will be expanded to path components: ```scala val params = List("a", "b", "c") ``` ```scala println(uri"http://example.com/$params") // http://example.com/a/b/c ``` ## Special cases If a string containing the protocol is embedded *at the very beginning*, it will not be escaped, allowing to embed entire addresses as prefixes, e.g.: `uri"$endpoint/login"`, where `val endpoint = "http://example.com/api"`. This is useful when a base URI is stored in a value, and can then be used as a base for constructing more specific URIs. ## Relative URIs The `Uri` class can represent both relative and absolute URIs. Hence, in terms of [rfc3986](https://tools.ietf.org/html/rfc3986), it is in fact a URI reference. Relative URIs can be created using the interpolator, same as absolute ones, e.g.: ```scala println(uri"/api/$params") // /api/a/b/c ``` When sending requests using relative URIs, the [`ResolveRelativeUrisBackend`](../backends/summary.md) backend wrapper might be useful to resolve them. ## All features combined A fully-featured example: ```scala import sttp.client4.* val secure = true val scheme = if (secure) "https" else "http" val subdomains = List("sub1", "sub2") val vx = Some("y z") val paramMap = Map("a" -> 1, "b" -> 2) val jumpTo = Some("section2") ``` ```scala println(uri"$scheme://$subdomains.example.com?x=$vx&$paramMap#$jumpTo") // https://sub1.sub2.example.com?x=y+z&a=1&b=2#section2 ``` ## FAQ: encoding & decoding URI components A common question about sttp's `Uri` is why certain characters in various components (path segment, query parameters) are **not** encoded, even if they are given as encoded when creating the URI. The first thing to keep in mind is that internally the URI class stores all components in a **decoded** form. Hence if you have an URI which in an encoded form has some special characters, such as `/a%20b`, the `Uri` data structure, which is an ordinary `case class`, will contain a path segment with `a b`. When parsing, that includes creating URIs from constant strings e.g. `uri"http://example.com/a%20b"`, all of the components are decoded and stored in this form. This means that `Uri` might **not** exactly preserve the original form, in which path segments or query parameters have been written down (this might change in a future major release, though). When serialising the `Uri` back to a `String`, the code follows the escaping rules defined in [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986), which specifies the syntax of URIs. There, you may find that for example `&` don't have to be escaped in path components, or that `/` don't have to be escaped in query parameters. Refer to the `Rfc3986` class or the specification for a set of characters, which are allowed in each context. This is often surprising to users, as other libraries and frameworks often escape everything, even if it's not necessary, regardless of context. If for some reason you do require changing the way certain components are encoded, this can be done individually for every segment (by manually changing the `Uri` case class), or with the `[component]SegmentsEncoding` methods. For example, to always encode every non-standard character in query segments: ```scala import sttp.model._ println(uri"http://example.com?a=b/?c%26d".querySegmentsEncoding(Uri.QuerySegmentEncoding.All)) // http://example.com?a=b%2F%3Fc%26d // compare to: println(uri"http://example.com?a=b/?c%26d") // http://example.com?a=b/?c%26d ``` # Request definition basics As mentioned in the [quickstart](../quickstart.md), the following import will be needed: ```scala import sttp.client4.* ``` This brings into scope `basicRequest`, the starting request. This request can be customised, each time yielding a new, immutable request definition (unless a mutable body is set on the request, such as a byte array). As the request definition is immutable, it can be freely stored in values, shared across threads, and customized multiple times in various ways. For example, we can set a cookie, `String` -body and specify that this should be a `POST` request to a given URI: ```scala val request = basicRequest .cookie("login", "me") .body("This is a test") .post(uri"http://endpoint.com/secret") ``` The request parameters (headers, cookies, body etc.) can be specified **in any order**. It doesn't matter if the request method, the body, the headers or connection options are specified in this sequence or another. This way you can build arbitrary request templates, capturing all that's common among your requests, and customizing as needed. Remember that each time a modifier is applied to a request, you get a new immutable object. There's a lot of ways in which you can customize a request, which are covered in this guide. Another option is to just explore the API: most of the methods are self-explanatory and carry scaladocs if needed. Using the modifiers, each time we get a new request definition, but it's just a description: a data object; nothing is sent over the network until the `send(backend)` method is invoked. ## Query parameters and URIs Query parameters are specified as part of the URI, to which the request should be sent. The URI can only be set together with the request method (using `.get(Uri)`, `.post(Uri)`, etc.). The URI can be created programmatically (by calling methods on the `Uri` class), or using the `uri` interpolator, which also allows embedding (and later escaping) values from the environment. See the documentation on [creating URIs](../model/uri.md) for more details. ## Sending a request A request definition can be created without knowing how it will be sent. But to send a request, a backend is needed. A default, synchronous backend based on Java's `HttpClient` is provided in the `core` jar. To invoke the `send(backend)` method on a request description, you'll need an instance of `Backend`: ```scala val backend = DefaultSyncBackend() val response: Response[Either[String, String]] = request.send(backend) ``` The default backend invokes any effects synchronously. Other, asynchronous backends, use "wrapper" effect types, such as `Future` or `IO`. See the section on [backends](../backends/summary.md) for more details. ```{note} Only requests with the request method and uri can be sent. When trying to send a request without these components specified, a compile-time error will be reported. On how this is implemented, see the documentation on the [type of request definitions](type.md). ``` ## Initial requests sttp provides three initial requests: * `basicRequest`, which is an empty request with the `Accept-Encoding: gzip, deflate` header added. That's the one that is most commonly used. By default reads the response as a `Either[String, String]` (indicating HTTP 4xx/5xx failure or 2xx success). * `emptyRequest`, a completely empty request, with no headers at all. * `quickRequest`, which always reads the response as a `String`, regardless of the status code How the response body is handled can be (and very often is) customized. See the section on [response body specifications](../responses/body.md) for more details. # Authentication ## Supported schemes sttp supports basic, bearer-token based authentication and digest authentication. Two first cases are handled by adding an `Authorization` header with the appropriate credentials. Basic authentication, using which the username and password are encoded using Base64, can be added as follows: ```scala import sttp.client4.* val username = "mary" val password = "p@assword" basicRequest.auth.basic(username, password) ``` A bearer token can be added using: ```scala val token = "zMDjRfl76ZC9Ub0wnz4XsNiRVBChTYbJcE3F" basicRequest.auth.bearer(token) ``` ## Important Note on the `Authorization` Header and Redirects The `Authorization` header is by default removed during redirects. See [redirects](../conf/redirects.md) for more details. ## Digest authentication This type of authentication works differently. In its assumptions it is based on an additional message exchange between client and server. Due to that a special wrapping backend is need to handle that additional logic. In order to add digest authentication support just wrap other backend as follows: ```scala import sttp.client4.wrappers.DigestAuthenticationBackend val myBackend: SyncBackend = DefaultSyncBackend() DigestAuthenticationBackend(myBackend) ``` Then only thing which we need to do is to pass our credentials to the relevant request: ```scala val secureRequest = basicRequest.auth.digest(username, password) ``` It is also possible to use digest authentication against proxy: ```scala val secureProxyRequest = basicRequest.proxyAuth.digest(username, password) ``` Both of above methods can be combined with different values if proxy and target server use digest authentication. To learn more about digest authentication visit [wikipedia](https://en.wikipedia.org/wiki/Digest_access_authentication) Also keep in mind that there are some limitations with the current implementation: * there is no caching so each request will result in an additional round-trip (or two in case of proxy and server) * authorizationInfo is not supported * scalajs supports only md5 algorithm ## OAuth2 You can use sttp with OAuth2. Looking at the [OAuth2 protocol flow](https://tools.ietf.org/html/rfc6749#section-1.2), sttp might be helpful in the second and third step of the process: 1. (A)/(B) - Your UI needs to enable the user to authenticate. Your application will then receive a callback from the authentication server, which will include an authentication code. 2. (C)/(D) - You need to send a request to the authentication server, passing in the authentication code from step 1. You'll receive an access token in response (and optionally a refresh token). For example, if you were using GitHub as your authentication server, you'd need to take the values of `clientId` and `clientSecret` from the GitHub settings, then take the `authCode` received in step 1 above, and send a request like this: ```scala import sttp.client4.circe.* import io.circe.* import io.circe.generic.semiauto.* val authCode = "SplxlOBeZQQYbYS6WxSbIA" val clientId = "myClient123" val clientSecret = "s3cret" case class MyTokenResponse(access_token: String, scope: String, token_type: String, refresh_token: Option[String]) implicit val tokenResponseDecoder: Decoder[MyTokenResponse] = deriveDecoder[MyTokenResponse] val backend = DefaultSyncBackend() val tokenRequest = basicRequest .post(uri"https://github.com/login/oauth/access_token?code=$authCode&grant_type=authorization_code") .auth .basic(clientId, clientSecret) .header("accept","application/json") val authResponse = tokenRequest.response(asJson[MyTokenResponse]).send(backend) val accessToken = authResponse.body.map(_.access_token) ``` 3. (E)/(F) - Once you have the access token, you can use it to request the protected resource from the resource server, depending on its specification. # Request body ## Text data In its simplest form, the request's body can be set as a `String`. By default, this method will: * use the UTF-8 encoding to convert the string to a byte array * if not specified before, set `Content-Type: text/plain` * if not specified before, set `Content-Length` to the number of bytes in the array A `String` body can be set on a request as follows: ```scala import sttp.client4.* basicRequest.body("Hello, world!") ``` It is also possible to use a different character encoding: ```scala import sttp.client4.* basicRequest.body("Hello, world!", "utf-8") ``` ## Binary data To set a binary-data body, the following methods are available: ```scala import sttp.client4.* val bytes: Array[Byte] = ??? basicRequest.body(bytes) import java.nio.ByteBuffer val byteBuffer: ByteBuffer = ??? basicRequest.body(byteBuffer) import java.io.ByteArrayInputStream val inputStream: ByteArrayInputStream = ??? basicRequest.body(inputStream) ``` If not specified before, these methods will set the content type to `application/octet-stream`. When using a byte array, additionally the content length will be set to the length of the array (unless specified explicitly). ```{note} While the object defining a request is immutable, setting a mutable request body will make the whole request definition mutable as well. With `InputStream`, the request can be moreover sent only once, as input streams can be consumed once. ``` ## Uploading files To upload a file, simply set the request body as a `File` or `Path`: ```scala import sttp.client4.* import java.io.File basicRequest.body(new File("data.txt")) import java.nio.file.Path basicRequest.body(Path.of("data.txt")) ``` Note that on JavaScript only a `Web/API/File` is allowed. As with binary body methods, the content type will default to `application/octet-stream`, and the content length will be set to the length of the file (unless specified explicitly). See also [multi-part](multipart.md) and [streaming](streaming.md) support. ## Form data If you set the body as a `Map[String, String]` or `Seq[(String, String)]`, it will be encoded as form-data (as if a web form with the given values was submitted). The content type will default to `application/x-www-form-urlencoded`; content length will also be set if not specified. By default, the `UTF-8` encoding is used, but can be also specified explicitly: ```scala import sttp.client4.* basicRequest.body(Map("k1" -> "v1")) basicRequest.body(Map("k1" -> "v1"), "utf-8") basicRequest.body("k1" -> "v1", "k2" -> "v2") basicRequest.body(Seq("k1" -> "v1", "k2" -> "v2"), "utf-8") ``` ## Custom serializers It is also possible to write custom serializers, which return arbitrary body representations. These should be methods/functions which return instances of `BasicBody`, which is a wrapper for one of the supported request body types: a `String`, byte array, an input stream, etc. For example, here's how to write a custom serializer for a case class, with serializer-specific default content type: ```scala import sttp.client4.* import sttp.model.MediaType case class Person(name: String, surname: String, age: Int) // for this example, assuming names/surnames can't contain commas def serializePerson(p: Person): BasicBody = { val serialized = s"${p.name},${p.surname},${p.age}" StringBody(serialized, "UTF-8", MediaType.TextCsv) } basicRequest.body(serializePerson(Person("mary", "smith", 67))) ``` See the implementations of the `BasicBody` trait for more options. ## Compressing bodies Request bodies can be compressed, using an algorithm that's supported by the backend. By default, all backends support the `gzip` and `deflate` compression algorithms. To compress a request body, use the `request.compressBody(encoding)` method. This will set the the `Content-Encoding` header on the request, as well as compress the body when the request is sent. If the given encoding is not supported by the backend, an exception will be thrown / a failed effect will be returned. Support for custom compression algorithms can be added at backend creation time, by customising the `compressionHandlers` parameter, and adding a `Compressor` implementation. Such an implementation has to specify the encoding, which it handles, as well as appropriate body transformation (which is backend-specific). Note that clients often don't know upfront which compression algorithms (if at all) the server supports, and that's why requests are often sent uncompressed. Sending an encoded (compressed) body, when the server doesn't support decompression, might lead to 4xx or 5xx errors. # Cookies ## Cookies on requests Cookies sent in requests are key-value pairs contained in the `Cookie` header. They can be set on a request in a couple of ways. The first is using the `.cookie(name: String, value: String)` method. This will yield a new request definition which, when sent, will contain the given cookie. Cookies are currently only available on the JVM. Cookies can also be set using the following methods: ```scala import sttp.client4.* import sttp.model.headers.CookieWithMeta basicRequest .cookie("k1", "v1") .cookie("k2" -> "v2") .cookies("k3" -> "v3", "k4" -> "v4") .cookies(Seq(CookieWithMeta("k5", "k5"), CookieWithMeta("k6", "k6"))) ``` ## Cookies from responses It is often necessary to copy cookies from a response, e.g. after a login request is sent, and a successful response with the authentication cookie received. Having an object `response: Response[_]`, cookies on a request can be copied: ```scala import sttp.client4.* val backend = DefaultSyncBackend() val loginRequest = basicRequest .cookie("login", "me") .body("This is a test") .post(uri"http://endpoint.com") val response = loginRequest.send(backend) basicRequest.cookies(response) ``` Or, it's also possible to store only the `sttp.model.CookieWithMeta` objects (a sequence of which can be obtained from a response), and set the on the request: ```scala import sttp.client4.* val backend = DefaultSyncBackend() val loginRequest = basicRequest .cookie("login", "me") .body("This is a test") .post(uri"http://endpoint.com") val response = loginRequest.send(backend) val cookiesFromResponse = response.unsafeCookies basicRequest.cookies(cookiesFromResponse) ``` ## Cookies across redirects The `Cookie` header is a sensitive header, so by default it is stripped when following a redirect; cookies set via `Set-Cookie` during a redirect chain are not carried over to subsequent requests either. To opt into a cookie jar that does this, attach a `CookieStorage` to the request. The `FollowRedirectsBackend` (applied to all backends by default) then, for each request in a redirect chain, sends the stored cookies that domain/path-match the request and collects the response's `Set-Cookie` cookies into the storage: ```scala import sttp.client4.* import sttp.client4.wrappers.CookieStorage val backend = DefaultSyncBackend() basicRequest .cookieStorage(CookieStorage.empty) .get(uri"https://endpoint.com") .send(backend) ``` Matching follows a subset of [RFC 6265](https://www.rfc-editor.org/rfc/rfc6265): domain, path and the `Secure` attribute. Time-based expiry is not tracked, but a `Set-Cookie` with `Max-Age` <= 0 removes a matching cookie. # Headers ## Arbitrary headers Arbitrary headers can be set on the request using the `.header` method: ```scala import sttp.client4.* basicRequest.header("User-Agent", "myapp") ``` As with any other request definition modifier, this method will yield a new request, which has the given header set. The headers can be set at any point when defining the request, arbitrarily interleaved with other modifiers. While most headers should be set only once on a request, HTTP allows setting a header multiple times, or with multiple values. That's why the `header` method has an additional parameter, `onDuplicate`, which by default is set to `DuplicateHeaderBehavior.Replace`. This way, if the same header is specified twice, only the last value will be included in the request. Alternatively: * if previous values should be preserved, set this parameter to `DuplicateHeaderBehavior.Add` * if the values of the headers should be combined using `,`, or in case of cookies with `;`, use `DuplicateHeaderBehavior.Combine` There are also variants of this method accepting a number of headers: ```scala import sttp.client4.* import sttp.model.* basicRequest.header(Header("k1", "v1"), onDuplicate = DuplicateHeaderBehavior.Add) basicRequest.header("k2", "v2") basicRequest.header("k3", "v3", DuplicateHeaderBehavior.Combine) basicRequest.headers(Map("k4" -> "v4", "k5" -> "v5")) basicRequest.headers(Header("k9", "v9"), Header("k10", "v10"), Header("k11", "v11")) ``` ## Common headers For some common headers, dedicated methods are provided: ```scala import sttp.client4.* basicRequest.contentType("application/json") basicRequest.contentType("application/json", "iso-8859-1") basicRequest.contentLength(128) basicRequest.acceptEncoding("gzip, deflate") ``` See also documentation on setting [cookies](cookies.md) and [authentication](authentication.md). ## Header names To avoid using string literals for header names, the `HeaderNames` object from sttp model can be used: ```scala import sttp.client4.* import sttp.model.HeaderNames import sttp.model.headers.CacheDirective import scala.concurrent.duration.* basicRequest.header(HeaderNames.CacheControl, CacheDirective.MaxAge(1.day).toString) ``` # Multipart requests To set a multipart body on a request, the `multipartBody` method should be used (instead of `body`). Each body part is represented as an instance of `Part[BasicBodyPart]`, which can be conveniently constructed using `multipart` methods coming from the `sttp.client4` package. A single part of a multipart request consist of a mandatory name and a payload of type: * `String` * `Array[Byte]` * `ByteBuffer` * `InputStream` * `Map[String, String]` * `Seq[(String, String)]` To add a file part, the `multipartFile` method (also from the `com.softwaremill.sttp` package) should be used. This method is overloaded and supports `File`/`Path` objects on the JVM, and `Web/API/File` on JS. The content type of each part is by default the same as when setting simple bodies: `text/plain` for parts of type `String`, `application/x-www-form-urlencoded` for parts of key-value pairs (form data) and `application/octet-stream` otherwise (for binary data). The parts can be specified using either a `Seq[Multipart]` or by using multiple arguments: ```scala import sttp.client4.* basicRequest.multipartBody(Seq(multipart("p1", "v1"), multipart("p2", "v2"))) basicRequest.multipartBody(multipart("p1", "v1"), multipart("p2", "v2")) ``` For example: ```scala import sttp.client4.* import java.io.* val someFile = new File("/sample/path") basicRequest.multipartBody( multipart("text_part", "data1"), multipartFile("file_part", someFile), // someFile: File multipart("form_part", Map("x" -> "10", "y" -> "yes")) ) ``` ## Customising part meta-data For each part, an optional filename can be specified, as well as a custom content type and additional headers. For example: ```scala import sttp.client4.* import java.io.* val logoFile = new File("/sample/path/logo123.jpg") val docFile = new File("/sample/path/doc123.doc") basicRequest.multipartBody( multipartFile("logo", logoFile).fileName("logo.jpg").contentType("image/jpg"), multipartFile("text", docFile).fileName("text.doc") ) ``` # Streaming Some backends (see [backends summary](../backends/summary.md)) support streaming bodies, as described by the `Streams[S]` capability. If that's the case, you can set a stream of the supported type as a request body using the `streamBody` method, instead of the usual `body` method. ```{note} Here, streaming refers to (usually) non-blocking, asynchronous streams of data. To send data which is available as an `InputStream`, or a file from local storage (which is available as a `File` or `Path`), no special backend support is needed. See the documenttation on [setting the request body](body.md). ``` An implementation of the `Streams[S]` capability must be passed to the `.streamBody` method, to determine the type of streams that are supported. These implementations are provided by the backend implementations, e.g. `PekkoStreams` or `Fs2Streams[F]`. For example, using the [pekko-http backend](../backends/pekko.md), a request with a streaming body can be defined as follows: ```scala import sttp.client4.* import sttp.capabilities.pekko.PekkoStreams import org.apache.pekko.stream.scaladsl.Source import org.apache.pekko.util.ByteString val chunks = "Streaming test".getBytes("utf-8").grouped(10).to(Iterable) val source: Source[ByteString, Any] = Source.apply(chunks.toList.map(ByteString(_))) basicRequest .post(uri"...") .streamBody(PekkoStreams)(source) ``` ```{note} A request with the body set as a stream can only be sent using a backend supporting exactly the given type of streams. ``` It's also possible to specify that the [response body should be a stream](../responses/body.md). # The type of request definitions Most request definitions have the type `Request[T]`. The `T` specifies the type to which the response will be read. By default, this is `Either[String, String]`. But it can also be e.g. `Array[Byte]` or `Unit`, if the response should be ignored. Response body handling can be changed by calling the `.response` method. With backends which support streaming, this can also be a supported stream type. See [response body specifications](../responses/body.md) for more details. If you're using streaming [request](streaming.md)/[response](../responses/body.md) bodies, you'll get a `StreamRequest[T, R]`. The `R` type parameter specifies the requirements of this request. In case of streaming, this will be an implementation of `Streams[S]`, such as `Fs2Streams[F]` or `PekkoStreams`. You can only send such requests using backends, which also support this type of streaming (this is verified at compile-time)! If you're describing a web socket request, you'll get a `WebSocketRequest[F[_], T]`. The `F[_]` type parameter describes the effect, using which the web socket is processed (by default, the request then also contains the entire logic to interact with the web socket, although this is not strictly required). For synchronous web sockets, `F[_]` will be `Identity` ("no wrapper", direct-style). For asynchronous web sockets, `F[_]` might be `Future` or `IO` (or any other effect type). As with streams, such requests can only be sent using backends which support web sockets, and the effect type used to process the web socket. Finally, if you're processing the websocket using streaming, you'll work with a `WebSocketStreamRequest[T, S]`. As before, `S` will contain the required streaming capability. However, it will also contain the effect type, which is used to interact with the web socket. An example value for `S` might be `PekkoStreams & Effect[Future]`. # Responses Responses are represented as instances of the case class `Response[T]`, where `T` is the type of the response body. When sending a request, the response might be return directly, or wrapped with an effect containing the response. For example, for asynchronous backends, we can get a `Future[Response[T]]`, while for the default synchronous backend, there's no wrapper at all. If sending the request fails, either due to client or connection errors, an exception will be thrown (synchronous backends), or a failed effect will be returned (e.g. a failed future). ```{note} If the request completes, but results in a non-2xx return code, the request is still considered successful, that is, a `Response[T]` will be returned. See [response body specifications](body.md) for details on how such cases are handled. ``` ## Response code The response code is available through the `.code` property. There are also methods such as `.isSuccess` or `.isServerError` for checking specific response code ranges. ## Response headers Response headers are available through the `.headers` property, which gives all headers as a sequence (not as a map, as there can be multiple headers with the same name). Individual headers can be obtained using the methods: ```scala import sttp.model.* import sttp.client4.* val backend = DefaultSyncBackend() val request = basicRequest.get(uri"https://httpbin.org/get") val response = request.send(backend) val singleHeader: Option[String] = response.header(HeaderNames.Server) val multipleHeaders: Seq[String] = response.headers(HeaderNames.Allow) ``` There are also helper methods available to read some commonly accessed headers: ```scala val contentType: Option[String] = response.contentType val contentLength: Option[Long] = response.contentLength ``` Finally, it's possible to parse the response cookies into a sequence of the `CookieWithMeta` case class: ```scala import sttp.model.headers.CookieWithMeta val cookies: Seq[CookieWithMeta] = response.unsafeCookies ``` If the cookies from a response should be set without changes on the request, this can be done directly; see the [cookies](../requests/cookies.md) section in the request definition documentation. ## Obtaining the response body The response body can be obtained through the `.body: T` property. `T` is the type of the body to which it's deserialized, as specified in the request description - see the next section on [response body specifications](body.md). # Response body By default, the received response body will be read as a `Either[String, String]`, using the encoding specified in the `Content-Type` response header (and if none is specified, using `UTF-8`). This is of course configurable: response bodies can be ignored, deserialized into custom types, received as a stream or saved to a file. When using `basicRequest`, the default `response.body` will be a: * `Left(errorMessage)` if the request is successful, but response code is not 2xx. * `Right(body)` if the request is successful, and the response code is 2xx. How the response body will be read is part of the request description, as already when sending the request, the backend needs to know what to do with the response. The type to which the response body should be deserialized is a type parameter of `Request`. It's used in request definition in the `request.response: ResponseAs[T]` property. ## Basic response descriptions To conveniently specify how to deserialize the response body, a number of `as(...Type...)` methods are available. They can be used to provide a value for the request description's `response` property: ```scala import sttp.client4.* basicRequest.response(asByteArray) ``` When the above request is completely described and sent, it will result in a `Response[Either[String, Array[Byte]]]` (where the left and right correspond to non-2xx and 2xx status codes, as above). Other possible response descriptions include: ```scala import sttp.client4.* import java.io.File import java.nio.file.Path def ignore: ResponseAs[Unit] = ??? def asString: ResponseAs[Either[String, String]] = ??? def asStringAlways: ResponseAs[String] = ??? def asStringOrFail: ResponseAs[String] = ??? def asString(encoding: String): ResponseAs[Either[String, String]] = ??? def asStringAlways(encoding: String): ResponseAs[String] = ??? def asStringOrFail(encoding: String): ResponseAs[String] = ??? def asByteArray: ResponseAs[Either[String, Array[Byte]]] = ??? def asByteArrayAlways: ResponseAs[Array[Byte]] = ??? def asByteArrayOrFail: ResponseAs[Array[Byte]] = ??? def asParams: ResponseAs[Either[String, Seq[(String, String)]]] = ??? def asParamsAlways: ResponseAs[Seq[(String, String)]] = ??? def asParamsOrFail: ResponseAs[Seq[(String, String)]] = ??? def asParams(encoding: String): ResponseAs[Either[String, Seq[(String, String)]]] = ??? def asParamsAlways(encoding: String): ResponseAs[Seq[(String, String)]] = ??? def asFile(file: File): ResponseAs[Either[String, File]] = ??? def asFileAlways(file: File): ResponseAs[File] = ??? def asPath(path: Path): ResponseAs[Either[String, Path]] = ??? def asPathAlways(path: Path): ResponseAs[Path] = ??? def asEither[A, B](onError: ResponseAs[A], onSuccess: ResponseAs[B]): ResponseAs[Either[A, B]] = ??? def fromMetadata[T](default: ResponseAs[T], conditions: ConditionalResponseAs[T]*): ResponseAs[T] = ??? def asBoth[A, B](l: ResponseAs[A], r: ResponseAs[B]): ResponseAs[(A, B)] = ??? def asBothOption[A, B](l: ResponseAs[A], r: ResponseAs[B]): ResponseAs[(A, Option[B])] = ??? ``` Hence, to discard the response body, the request description should include the following: ```scala import sttp.client4.* basicRequest.response(ignore) ``` And to save the response to a file: ```scala import sttp.client4.* import java.io.* val someFile = new File("some/path") basicRequest.response(asFile(someFile)) ``` ```{note} As the handling of response is specified upfront, there's no need to "consume" the response body. It can be safely discarded if not needed. ``` The `as...Always` response descriptions will read the response body as the target type always, regardless of the status code. ## Failing when the response code is not 2xx Sometimes it's convenient to get a failed effect (or an exception thrown) when the response status code is not successful. In such cases, you should use the `...OrFail` response description, or modify an existing response description using the `.orFail` combinator: ```scala import sttp.client4.* basicRequest.response(asStringOrFail): PartialRequest[String] ``` The `.orFail` combinator works in all cases where the response body is specified to be deserialized as an `Either`. If the left is already an exception, it will be thrown unchanged. Otherwise, the left-value will be wrapped in an `UnexpectedStatusCode`. ```{note} While both `asStringAlways` and `asStringOrFail` have the type `ResponseAs[String]`, they are different. The first will return the response body as a string always, regardless of the responses' status code. The second will return a failed effect / throw a `UnexpectedStatusCode` exception for non-2xx status codes, and the string as body only for 2xx status codes. ``` There's also a variant of the combinator, `.orFailDeserialization`, which can be used to extract typed errors and fail the effect if there's a deserialization error. ## Custom body deserializers It's possible to define custom body deserializers by taking any of the built-in response descriptions and mapping over them. Each `ResponseAs` instance has `map` and `mapWithMetadata` methods, which can be used to transform it to a description for another type (optionally using response metadata, such as headers or the status code). Each such value is immutable and can be used multiple times. ```{note} Alternatively, response descriptions can be modified directly from the request description, by using the ``request.mapResponse(...)`` method. That's equivalent to calling ``request.response(request.response.map(...))``, that is setting a new response description, to a modified old response description; but with shorter syntax. ``` As an example, to read the response body as an `Int`, the following response description can be defined (warning: this ignores the possibility of exceptions!): ```scala import sttp.client4.* val asInt: ResponseAs[Either[String, Int]] = asString.mapRight((_: String).toInt) basicRequest .get(uri"http://example.com") .response(asInt) ``` To integrate with a third-party JSON library, and always parse the response as JSON (regardless of the status code): ```scala import sttp.client4.* type JsonError type JsonAST def parseJson(json: String): Either[JsonError, JsonAST] = ??? val asJson: ResponseAs[Either[JsonError, JsonAST]] = asStringAlways.map(parseJson) basicRequest .response(asJson) ``` A number of JSON libraries are supported out-of-the-box, see [json support](../other/json.md). ## Response-metadata dependent deserializers Using the `fromMetadata` combinator, it's possible to dynamically specify how the response should be deserialized, basing on the response status code and response headers. The default `asString`, `asByteArray` response descriptions use this method to return a `Left` in case of non-2xx responses, and a `Right` otherwise. A more complex case, which uses Circe for deserializing JSON, choosing to which model to deserialize to depending on the status code, can look as following: ```scala import sttp.client4.* import sttp.model.* import sttp.client4.circe.* import io.circe.* import io.circe.generic.auto.* sealed trait MyModel case class SuccessModel(name: String, age: Int) extends MyModel case class ErrorModel(message: String) extends MyModel val myRequest: Request[Either[ResponseException[String], MyModel]] = basicRequest .get(uri"https://example.com") .response(fromMetadata( asJson[ErrorModel], ConditionalResponseAs(_.code == StatusCode.Ok, asJson[SuccessModel]) )) ``` The above example assumes that success and error models are part of one hierarchy (`MyModel`). Sometimes http errors are modelled independently of success. In this case, we can use `asJsonEither`, which uses `asEitherDeserialized` under the covers: ```scala import sttp.client4.* import sttp.model.* import sttp.client4.circe.* import io.circe.* import io.circe.generic.auto.* case class MyModel(p1: Int) sealed trait MyErrorModel case class Conflict(message: String) extends MyErrorModel case class BadRequest(message: String) extends MyErrorModel case class GenericError(message: String) extends MyErrorModel basicRequest .get(uri"https://example.com") .response(asJsonEither[MyErrorModel, MyModel]) ``` ## Streaming ### Blocking streaming (InputStream) Some backends on the JVM support receiving the response body as a `java.io.InputStream`. This is possible either using the safe `asInputStream(f)` description, where the entire stream has to be consumed by the provided `f` function, and is then closed by sttp client. Alternatively, there's `asInputStreamUnsafe`, which returns the stream directly to the user, who is then responsible for closing it. `InputStream`s have two limitations. First, they operate on the relatively low `byte`-level. The consumer is responsible for any decoding, chunking etc. Moreover, all `InputStream` operations are blocking, hence using them in a non-virtual-threads environment may severely limit performance. If you're using a functional effect system, see below on how to use non-blocking streams instead. ### Non-blocking streaming If the backend used supports non-blocking, asynchronous streaming (see "Supported stream type" in the [backends summary](../backends/summary.md)), it's possible to receive responses as a stream. This can be described using the following methods: ```scala import sttp.capabilities.{Effect, Streams} import sttp.client4.* import sttp.model.ResponseMetadata def asStream[F[_], T, S](s: Streams[S])(f: s.BinaryStream => F[T]): StreamResponseAs[Either[String, T], Effect[F] with S] = ??? def asStreamOrFail[F[_], T, S](s: Streams[S])(f: s.BinaryStream => F[T]): StreamResponseAs[T, S with Effect[F]] = ??? def asStreamWithMetadata[F[_], T, S](s: Streams[S])( f: (s.BinaryStream, ResponseMetadata) => F[T] ): StreamResponseAs[Either[String, T], Effect[F] with S] = ??? def asStreamAlways[F[_], T, S](s: Streams[S])(f: s.BinaryStream => F[T]): StreamResponseAs[T, Effect[F] with S] = ??? def asStreamAlwaysWithMetadata[F[_], T, S](s: Streams[S])( f: (s.BinaryStream, ResponseMetadata) => F[T] ): StreamResponseAs[T, Effect[F] with S] = ??? def asStreamUnsafe[S](s: Streams[S]): StreamResponseAs[Either[String, s.BinaryStream], S] = ??? def asStreamUnsafeAlways[S](s: Streams[S]): StreamResponseAs[s.BinaryStream, S] = ??? ``` All of these descriptions require the streaming capability to be passed as a parameter, an implementation of `Streams[S]`. This is used to determine the type of binary streams that are supported, and to require that the backend used to send the request supports the given type of streams. These implementations are provided by the backend implementations, e.g. `PekkoStreams` or `Fs2Streams[F]`. The first two "safe" variants pass the response stream to the user-provided function, which should consume the stream entirely. Once the effect returned by the function is complete, the backend will try to close the stream (if the streaming implementation allows it). The "unsafe" variants return the stream directly to the user, and then it's up to the user of the code to consume and close the stream, releasing any resources held by the HTTP connection. For example, when using the [Pekko backend](../backends/pekko.md): ```scala import org.apache.pekko.stream.scaladsl.Source import org.apache.pekko.util.ByteString import scala.concurrent.Future import sttp.capabilities.pekko.PekkoStreams import sttp.client4.* import sttp.client4.pekkohttp.PekkoHttpBackend val backend: StreamBackend[Future, PekkoStreams] = PekkoHttpBackend() val response: Future[Response[Either[String, Source[ByteString, Any]]]] = basicRequest .post(uri"...") .response(asStreamUnsafe(PekkoStreams)) .send(backend) ``` It's also possible to parse the received stream as server-sent events (SSE), using an implementation-specific mapping function. Refer to the documentation for particular backends for more details. ## Decompressing bodies (handling the Content-Encoding header) If the response body is compressed using `gzip` or `deflate` algorithms, it will be decompressed if the `decompressResponseBody` request option is set. By default this is set to `true`, and can be disabled using the `request.disableAutoDecompression` method. The encoding of the response body is determined by the encodings that are accepted by the client. That's why `basicRequest` and `quickRequest` both have the `Accept-Encoding` header set to `gzip, deflate`. That's in contrast to `emptyRequest`, which has no headers set by default. If you'd like to use additional decompression algorithms, you'll need to: * amend the `Accept-Encoding` header that's set on the request * add a decompression algorithm to the backend; that can be done on backend creation time, by customizing the `compressionHandlers` parameter, and adding a `Decompressor` implementation. Such an implementation has to specify the encoding, which it handles, as well as appropriate body transformation (which is backend-specific). ## Limiting the response body size To limit the size of the response body, use the `maxResponseBodyLength` method on the request description. This modified the `RequestOption`s associated with the request. By default, there's no limit set. When a limit is set and it is exceed, sending the request will fail with a `SttpClientException.ReadException`. This feature is currently available only for JVM backends. # Exceptions HTTP requests might fail in a variety of ways! There are two basic types of failures that might occur: * network-level failure, such as the invalid/unroutable hosts, inability to establish a TCP connection, or broken sockets * protocol-level failure, represented as 4xx and 5xx responses The first type of failures is represented by exceptions, which are thrown when sending the request (using `request.send(backend)`). The second type of failure is represented as a `Response[T]`, with the appropriate response code. The response body might depend on the status code; by default the response is read as a `Either[String, String]`, where the left side represents protocol-level failure, and the right side: success. Exceptions might be thrown directly (synchronous, direct-style backends), or returned as failed effects (other backends, e.g. failed `scala.concurrent.Future`). Backends will try to categorize these exceptions into a `SttpClientException`, which has two main subclass branches: * `ConnectException`: when a connection (tcp socket) can't be established to the target host * `ReadException`: when a connection has been established, but there's any kind of problem receiving the response (e.g. a broken socket) Read exceptions might be further categorized into: * `TimeoutException`: any kind of timeout when reading * `TooManyRedirectsException`: throw by the [follow-redirects](../conf/redirects.md) backend * `ResponseHandlingException`: wrapping a [[ResponseException]] (exception that occurs when handling the response body, using e.g. `asJson`) In general, it's safe to assume that the request hasn't been sent in case of connect exceptions. With read exceptions, the target host might or might not have received and processed the request. Unknown exceptions aren't categorized and are re-thrown unchanged. ## Response-handling, deserialization errors Exceptions might be thrown when deserializing the response body - depending on the specification of how to handle response bodies. The built-in deserializers (see e.g. [json](../other/json.md)) return errors represented as `ResponseException[HE]`, which can either be a `UnexpectedStatusCode` (protocol-level failures, containing a potentially deserialized body value) or a `DeserializationException` (containing a deserialization-library-specific exception). This means that a typical `asJson` response specification will result in the body being read as: ```scala import sttp.client4.* def asJson[T]: ResponseAs[Either[ResponseException[String], T]] = ??? ``` There are also the `asJsonOrFail`, `asStringOrFail` etc. response descriptions, which handle http errors or deserialization exceptions by throwing an exception / returning a failed effect. Alternatively, you can use the `.orFail` and `.orFailDeserialization` methods on any response description that is an `Either`. ## Possible outcomes Summing up, when the response is deserialized (e.g. to json), sending a request can have these outcomes, and can be represented in the following ways: * network-level success (HTTP request sent and response received) * http error (4xx, 5xx), successfully parsed (**a value wrapped in `Left`, or a failed effect**) * http success (2xx), successfully parsed (**a value possibly wrapped in `Right`**) * deserialization error (**a value wrapped in `Left`, or a failed effect**) * network-level failure (invalid host, broken socket): **failed effect** # WebSockets One of the optional capabilities that a backend can support are websockets (see [backends summary](../backends/summary.md)). Websocket requests are described exactly the same as regular requests, starting with `basicRequest`, adding headers, specifying the request method and uri. A websocket request will be sent instead of a regular one if the response specification includes handling the response as a websocket. Depending on the backend you are using, there are three variants of websocket response specifications: synchronous, asynchronous and streaming. To use them, add one of the following imports: * `import sttp.client4.ws.sync.*` if you are using a synchronous backend (such as `DefaultSyncBackend`), without any effect wrappers * `import sttp.client4.ws.async.*` if you are using an asynchronous backend (e.g. based on `Future`s or `IO`s) * `import sttp.client4.ws.stream.*` if you want to handle web socket messages using a non-blocking stream (e.g. `fs2.Stream` or `akka.stream.scaladsl.Source`) The above imports will bring into scope a number of `asWebSocket(...)` methods, giving a couple of variants of working with websockets. Alternatively, you can extend the `SttpWebSocketSyncApi`, `SttpWebSocketAsyncApi` or `SttpWebSocketStreamApi` traits, to group all used sttp client features within a single object. Refer to the documentation of individual backends for additional notes, or restrictions, when using WebSockets. ## Using `WebSocket` The first possibility of interacting with web sockets is using `sttp.client4.ws.SyncWebSocket` (sync variant), or `sttp.ws.WebSocket[F]` (async variant), where `F` is the backend-specific effects wrapper, such as `Future` or `IO`. These classes contain two basic methods: * `def receive: WebSocketFrame` (optionally wrapped with `F[_]` in the async variant) which will complete once a message is available, and return the next incoming frame (which can be a data, ping, pong or close) * `def send(f: WebSocketFrame, isContinuation: Boolean = false): Unit` (again optionally wrapped with `F[_]`), which sends a message to the websocket. The `WebSocketFrame` companion object contains methods for creating binary/text messages. When using fragmentation, the first message should be sent using `finalFragment = false`, and subsequent messages using `isContinuation = true`. The `SyncWebSocket` / `WebSocket` classes also contain other methods for receiving only text/binary messages, as well as automatically sending `Pong` responses when a `Ping` is received. The following response specifications which use `SyncWebSocket` are available in the `sttp.client4.ws.sync` object (the second type parameter of `WebSocketResponseAs` specifies the type returned as the response body): ```scala import sttp.client4.* import sttp.client4.ws.SyncWebSocket import sttp.model.ResponseMetadata import sttp.shared.Identity // when using import sttp.client4.ws.sync.* def asWebSocket[T](f: SyncWebSocket => T): WebSocketResponseAs[Identity, Either[String, T]] = ??? def asWebSocketOrFail[T](f: SyncWebSocket => T): WebSocketResponseAs[Identity, T] = ??? def asWebSocketWithMetadata[T]( f: (SyncWebSocket, ResponseMetadata) => T ): WebSocketResponseAs[Identity, Either[String, T]] = ??? def asWebSocketUnsafe: WebSocketResponseAs[Identity, Either[String, SyncWebSocket]] = ??? def asWebSocketAlwaysUnsafe: WebSocketResponseAs[Identity, SyncWebSocket] = ??? ``` The first variant, `asWebSocket`, passes an open `SyncWebSocket` to the user-provided function. This function should only return once interaction with the websocket is finished. The backend can then safely close the websocket. The value that's returned as the response body is either an error (represented as a `String`), in case the websocket upgrade didn't complete successfully, or the value returned by the websocket-interacting method. The second (`asWebSocketOrFail`) is similar, but any errors due to failed websocket protocol upgrades are represented as exceptions. The remaining two variants return the open `SyncWebSocket` directly, as the response body. It is then the responsibility of the client code to close the websocket, once it's no longer needed. Similar response specifications, but using an effect wrapper and `WebSocket[F]`, are available in the `sttp.client4.ws.async` objet. See also the [examples](../examples.md), which include examples involving websockets. ## Using non-blocking, asynchronous streams Another possibility is to work with websockets by providing a streaming stage, which transforms incoming data frames into outgoing frames. This can be e.g. a [Pekko](../backends/pekko.md) `Flow` or a [fs2](../backends/fs2.md) `Pipe`. The following response specifications are available: ```scala import sttp.client4.* import sttp.capabilities.{Streams, WebSockets} import sttp.ws.WebSocketFrame // when using import sttp.client4.ws.stream._ def asWebSocketStream[S](s: Streams[S])(p: s.Pipe[WebSocketFrame.Data[_], WebSocketFrame]): WebSocketStreamResponseAs[Either[String, Unit], S] = ??? def asWebSocketStreamOrFail[S](s: Streams[S])(p: s.Pipe[WebSocketFrame.Data[_], WebSocketFrame]): WebSocketStreamResponseAs[Unit, S] = ??? ``` Using streaming websockets requires the backend to support the given streaming capability (see also [streaming](../requests/streaming.md)). Streaming capabilities are described as implementations of `Streams[S]`, and are provided by backend implementations, e.g. `PekkoStreams` or `Fs2Streams[F]`. When working with streams of websocket frames keep in mind that a text payload may be fragmented into multiple frames. sttp provides two useful methods (`fromTextPipe`, `fromTextPipeF`) for each backend to aggregate these fragments back into complete messages. These methods can be found in corresponding WebSockets classes for given effect type: ```{eval-rst} ================ ========================================== effect type class name ================ ========================================== ``monix.Task`` ``sttp.client4.impl.monix.MonixWebSockets`` ``ZIO`` ``sttp.client4.impl.zio.ZioWebSockets`` ``fs2.Stream`` ``sttp.client4.impl.fs2.Fs2WebSockets`` ================ ========================================== ``` ## Using blocking, synchronous Ox streams [Ox](https://ox.softwaremill.com) is a Scala 3 toolkit that allows you to handle concurrency and resiliency in direct style, leveraging Java's 21+ virtual threads. If you're using Ox with `sttp`, you can use the `DefaultSyncBackend` from `sttp-core` for HTTP communication. The `ox` integration module allows handling WebSockets as Ox's `Flow`: ``` // sbt dependency "com.softwaremill.sttp.client4" %% "ox" % "4.0.25" ``` The `runWebSocketPipe` function from that module accepts a `SyncWebSocket`, as well as a function, which takes a `Flow` of messages received from the server, and produces a combined `Flow` which should both consume the incoming messages, and produce outgoing messages to be sent: ```scala import sttp.client4.* import sttp.client4.impl.ox.ws.runWebSocketPipe import sttp.client4.ws.sync.* val backend = DefaultSyncBackend() basicRequest .get(uri"wss://ws.postman-echo.com/raw") // you need to provide a Flow[WebSocketFrame] => Flow[WebSocketFrame] function .response(asWebSocket(ws => runWebSocketPipe(ws)(incoming => ???))) .send(backend) ``` See the [full example here](https://github.com/softwaremill/sttp/blob/master/examples/src/main/scala/sttp/client4/examples/ws/wsOxExample.scala). Read more about Ox, structured concurrency and Flows on the [project website](https://ox.softwaremill.com). ### Using channels Alternatively, you can obtain a lower-level `Source` + `Sink`, which allows directly reading/writing from the WebSocket using Ox's channels: ```scala import ox.* import sttp.client4.ws.SyncWebSocket import sttp.client4.impl.ox.ws.asSourceAndSink def useWebSocket(ws: SyncWebSocket): Unit = supervised { // (Source[WebSocketFrame], Sink[WebSocketFrame]) val (wsSource, wsSink) = asSourceAndSink(ws) // ... } ``` Make sure that the `Source` is continually read. This will guarantee that server-side `Close` signal is received and handled. If you don't want to process frames from the server, you can at least handle it with a `fork { source.drain() }`. You don't need to manually call `ws.close()` when using this approach, this will be handled automatically underneath, according to following rules: - If the request `Sink` is closed due to an upstream error, a `Close` frame is sent, and the `Source` with incoming responses gets completed as `Done`. - If the request `Sink` completes as `Done`, a `Close` frame is sent, and the response `Sink` keeps receiving responses until the server closes communication. - If the response `Source` is closed by a `Close` frame from the server or due to an error, the request Sink is closed as `Done`, which will still send all outstanding buffered frames, and then finish. ## Compression For those who plan to use a lot of websocket traffic, you could consider websocket compression, however it's often not supported: ### OkHttp based backends * supports compression (default: not enabled) ### akka-http backend Compression is not yet available, to track Akka developments in this area, see [this issue](https://github.com/akka/akka-http/issues/659). # Other Scala HTTP clients * [akka-http client](http://doc.akka.io/docs/akka-http/current/scala/http/client-side/index.html) * [play ws](https://github.com/playframework/play-ws) * [http4s](http://http4s.org/v0.17/client/) * [Gigahorse](http://eed3si9n.com/gigahorse/) * [Requests-Scala](https://github.com/lihaoyi/requests-scala) Also, check the [comparison by Marco Firrincieli](https://github.com/mfirry/scala-http-clients) on how to implement a simple request using a number of Scala HTTP libraries. # Request body progress callback When using the `HttpClient`-based backends (which includes the `DefaultSyncBackend` and `DefaultFutureBackend` on the JVM), it is possible to register a callback that keeps track of the progress of sending the request body. This feature is not available in other backends, and setting the attribute described below will have no effect. The callback is defined through an instance of the `BodyProgressCallback` trait. When a request is sent, the `BodyProgressCallback.onInit` method is invoked exactly once with the content length (if it is known). This is followed by arbitrary number of `onNext` calls. Finally, either `onComplete` or `onError` are called exactly once. ```{note} `onNext` is called when a part of the request body is ready to be sent over the network, that is, before it is actually being transferred. ``` All of the methods in the `BodyProgressCallback` implementation should be non-blocking and complete as fast as possible, so as not to obstruct sending data over the network. To register a callback, set the `BodyProgressCallback.RequestAttribute` on a request. For example: ```scala import sttp.client4.* import sttp.client4.httpclient.{HttpClientSyncBackend, BodyProgressCallback} import java.io.File val backend = HttpClientSyncBackend() val fileToSend: File = ??? val callback = new BodyProgressCallback { override def onInit(contentLength: Option[Long]): Unit = println(s"expected content length: $contentLength") override def onNext(bytesCount: Long): Unit = println(s"next, bytes: $bytesCount") override def onComplete(): Unit = println(s"complete") override def onError(t: Throwable): Unit = println(s"error: ${t.getMessage}") } val response = basicRequest .get(uri"http://example.com") .body(fileToSend) .attribute(BodyProgressCallback.RequestAttribute, callback) .send(backend) ``` # JSON Adding support for JSON (or other format) bodies in requests/responses is a matter of providing a [body serializer](../requests/body.md) and/or a [response body specification](../responses/body.md). Both are quite straightforward to implement, so integrating with your favorite JSON library shouldn't be a problem. However, there are some integrations available out-of-the-box. Each integration is available as an import, which brings `asJson` methods into scope. Alternatively, these values are grouped intro traits (e.g. `sttp.client4.circe.SttpCirceApi`), which can be extended to group multiple integrations in one object, and thus reduce the number of necessary imports. The following variants of `asJson` methods are available: * `asJson(b: B)` - to be used when specifying the body of a request: serializes the body so that it can be used as a request's body, e.g. using `basicRequest.body(asJson(myValue))` * `asJson[B]` - to be used when specifying how the response body should be handled: specifies that the body should be deserialized to json, but only if the response is successful (2xx); otherwise, a `Left` is returned, with body as a string * `asJsonOrFail[B]` - specifies that the body should be deserialized to json, if the response is successful (2xx); throws an exception/returns a failed effect if the response code is other than 2xx, or if deserialization fails * `asJsonAlways[B]` - specifies that the body should be deserialized to json, regardless of the status code * `asJsonEither[E, B]` - specifies that the body should be deserialized to json, using different deserializers for error and successful (2xx) responses * `asJsonEitherOrFail[E, B]` - specifies that the body should be deserialized to json, using different deserializers for error and successful (2xx) responses; throws an exception/returns a failed effect, if deserialization fails The type signatures vary depending on the underlying library (required implicits and error representation differs), but they obey the following pattern: ```scala import sttp.client4.* import sttp.client4.ResponseException.DeserializationException // request bodies def asJson[B](b: B): StringBody = ??? // response handling description def asJson[B]: ResponseAs[Either[ResponseException[String], B]] = ??? def asJsonOrFail[B]: ResponseAs[B] = ??? def asJsonAlways[B]: ResponseAs[Either[DeserializationException, B]] = ??? def asJsonEither[E, B]: ResponseAs[Either[ResponseException[E], B]] = ??? def asJsonEitherOrFail[E, B]: ResponseAs[Either[E, B]] = ??? ``` The response specifications can be further refined using `.orFail` and `.orFailDeserialization`, see [response body specifications](../responses/body.md). Following data class will be used through the next few examples: ```scala case class RequestPayload(data: String) case class ResponsePayload(data: String) ``` ## Circe JSON encoding of bodies and decoding of responses can be handled using [Circe](https://circe.github.io/circe/) by the `circe` module. To use add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "circe" % "4.0.25" ``` This module adds a body serialized, so that json payloads can be sent as request bodies. To send a payload of type `T` as json, a `io.circe.Encoder[T]` implicit value must be available in scope. Automatic and semi-automatic derivation of encoders is possible by using the [circe-generic](https://circe.github.io/circe/codec.html) module. Response can be parsed into json using `asJson[T]`, provided there's an implicit `io.circe.Decoder[T]` in scope. The decoding result will be represented as either a http/deserialization error, or the parsed value. For example: ```scala import sttp.client4.* import sttp.client4.circe.* val backend: SyncBackend = DefaultSyncBackend() import io.circe.generic.auto._ val requestPayload = RequestPayload("some data") val response: Response[Either[ResponseException[String], ResponsePayload]] = basicRequest .post(uri"...") .body(asJson(requestPayload)) .response(asJson[ResponsePayload]) .send(backend) ``` Arbitrary JSON structures can be traversed by parsing the result as `io.circe.Json`, and using the [circe-optics](https://circe.github.io/circe/optics.html) module. ## Json4s To encode and decode json using json4s, add the following dependencies to your project: ``` "com.softwaremill.sttp.client4" %% "json4s" % "4.0.25" "org.json4s" %% "json4s-native" % "3.6.0" ``` Note that in this example we are using the json4s-native backend, but you can use any other json4s backend. Using this module it is possible to set request bodies and read response bodies as case classes, using the implicitly available `org.json4s.Formats` (which defaults to `org.json4s.DefaultFormats`), and by bringing an implicit `org.json4s.Serialization` into scope. Usage example: ```scala import org.json4s.Formats import org.json4s.Serialization import sttp.client4.* import sttp.client4.json4s.* val backend: SyncBackend = DefaultSyncBackend() val requestPayload = RequestPayload("some data") given Serialization = org.json4s.native.Serialization given Formats = org.json4s.DefaultFormats val response: Response[Either[ResponseException[String], ResponsePayload]] = basicRequest .post(uri"...") .body(asJson(requestPayload)) .response(asJson[ResponsePayload]) .send(backend) ``` ## spray-json To encode and decode JSON using [spray-json](https://github.com/spray/spray-json), add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "spray-json" % "4.0.25" ``` Using this module it is possible to set request bodies and read response bodies as your custom types, using the implicitly available instances of `spray.json.JsonWriter` / `spray.json.JsonReader` or `spray.json.JsonFormat`. Usage example: ```scala import sttp.client4.* import sttp.client4.sprayJson.* import spray.json.* val backend: SyncBackend = DefaultSyncBackend() implicit val payloadJsonFormat: RootJsonFormat[RequestPayload] = ??? implicit val myResponseJsonFormat: RootJsonFormat[ResponsePayload] = ??? val requestPayload = RequestPayload("some data") val response: Response[Either[ResponseException[String], ResponsePayload]] = basicRequest .post(uri"...") .body(asJson(requestPayload)) .response(asJson[ResponsePayload]) .send(backend) ``` ## play-json To encode and decode JSON using [play-json](https://www.playframework.com), add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "play-json" % "4.0.25" ``` If you use older version of play (2.9.x), add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "play29-json" % "4.0.25" ``` To use, add an import: `import sttp.client4.playJson._`. ## zio-json To encode and decode JSON using the high-performance [zio-json](https://zio.github.io/zio-json/) library, one add the following dependency to your project. The `zio-json` module depends on ZIO 2.x and on the JVM requires Java 21+. For ZIO 1.x support, use `zio1-json`. ```scala "com.softwaremill.sttp.client4" %% "zio-json" % "4.0.25" // for ZIO 2.x "com.softwaremill.sttp.client4" %% "zio1-json" % "4.0.25" // for ZIO 1.x ``` or for ScalaJS (cross build) projects: ```scala "com.softwaremill.sttp.client4" %%% "zio-json" % "4.0.25" // for ZIO 2.x "com.softwaremill.sttp.client4" %%% "zio1-json" % "4.0.25" // for ZIO 1.x ``` To use, add an import: `import sttp.client4.ziojson._` (or extend `SttpZioJsonApi`), define an implicit `JsonCodec`, or `JsonDecoder`/`JsonEncoder` for your datatype. Usage example: ```scala import sttp.client4.* import sttp.client4.ziojson.* import zio.json.* val backend: SyncBackend = DefaultSyncBackend() implicit val payloadJsonEncoder: JsonEncoder[RequestPayload] = DeriveJsonEncoder.gen[RequestPayload] implicit val myResponseJsonDecoder: JsonDecoder[ResponsePayload] = DeriveJsonDecoder.gen[ResponsePayload] val requestPayload = RequestPayload("some data") val response: Response[Either[ResponseException[String], ResponsePayload]] = basicRequest .post(uri"...") .body(asJson(requestPayload)) .response(asJson[ResponsePayload]) .send(backend) ``` ## Jsoniter-scala To encode and decode JSON using the [high(est)-performant](https://plokhotnyuk.github.io/jsoniter-scala/) [jsoniter-scala](https://github.com/plokhotnyuk/jsoniter-scala) library, one add the following dependency to your project. ```scala "com.softwaremill.sttp.client4" %% "jsoniter" % "4.0.25" ``` or for ScalaJS (cross build) projects: ```scala "com.softwaremill.sttp.client4" %%% "jsoniter" % "4.0.25" ``` To use, add an import: `import sttp.client4.jsoniter._` (or extend `SttpJsonIterJsonApi`), define an implicit `JsonCodec`, or `JsonDecoder`/`JsonEncoder` for your datatype. Note that jsoniter does not support implicits `def`s so every `Either` instance has to be generated separately. However, an `implicit def` has been made for `Option` and is shipped in the `SttpJsonIterJsonApi` class. Usage example: ```scala import sttp.client4.* import sttp.client4.jsoniter.* import com.github.plokhotnyuk.jsoniter_scala.core.* import com.github.plokhotnyuk.jsoniter_scala.macros.* val backend: SyncBackend = DefaultSyncBackend() implicit val payloadJsonCodec: JsonValueCodec[RequestPayload] = JsonCodecMaker.make //note that the jsoniter doesn't support 'implicit defs' and so either has to be generated seperatly implicit val jsonEitherDecoder: JsonValueCodec[ResponsePayload] = JsonCodecMaker.make val requestPayload = RequestPayload("some data") val response: Response[Either[ResponseException[String], ResponsePayload]] = basicRequest .post(uri"...") .body(asJson(requestPayload)) .response(asJson[ResponsePayload]) .send(backend) ``` ## uPickle To encode and decode JSON using the [uPickle](https://github.com/com-lihaoyi/upickle) library, add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "upickle" % "4.0.25" ``` or for ScalaJS (cross build) projects: ```scala "com.softwaremill.sttp.client4" %%% "upickle" % "4.0.25" ``` To use, add an import: `import sttp.client4.upicklejson.default._` and define an implicit `ReadWriter` (or separately `Reader` and `Writer`) for your datatype. Usage example: ```scala import sttp.client4.* import sttp.client4.upicklejson.default.* import upickle.default.* val backend: SyncBackend = DefaultSyncBackend() implicit val requestPayloadRW: ReadWriter[RequestPayload] = macroRW[RequestPayload] implicit val responsePayloadRW: ReadWriter[ResponsePayload] = macroRW[ResponsePayload] val requestPayload = RequestPayload("some data") val response: Response[Either[ResponseException[String], ResponsePayload]] = basicRequest .post(uri"...") .body(asJson(requestPayload)) .response(asJson[ResponsePayload]) .send(backend) ``` If you have a customised version of upickle, with [custom configuration](https://com-lihaoyi.github.io/upickle/#CustomConfiguration), you'll need to create a dedicated object, which provides the upickle <-> sttp integration. There, you'll need to provide the implementation of `upickle.Api` that you are using. Moreover, the type of the overridden `upickleApi` needs to be the singleton type of the value (as in the example below). That's needed as the `upickle.Api` contains the `read`/`write` methods to serialize/deserialize the JSON; moreover, `ReadWriter` isn't a top-level type, but path-dependent one. For example, if you want to use the `legacy` upickle configuration, the integration might look as follows: ```scala import upickle.legacy.* // get access to ReadWriter type, macroRW derivation, etc. object legacyUpickle extends sttp.client4.upicklejson.SttpUpickleApi { override val upickleApi: upickle.legacy.type = upickle.legacy } import legacyUpickle.* // use upickle as in the above examples ``` # OpenAPI sttp-client [request definitions](../requests/basics.md) can be automatically generated from [openapi](https://swagger.io/specification/) `.yaml` specifications using the `scala-sttp` code generator, included in the [openapi-generator](https://github.com/OpenAPITools/openapi-generator) project. ## Using the openapi-generator There are two generators that you can use: * [scala-sttp4-jsoniter](https://openapi-generator.tech/docs/generators/scala-sttp4-jsoniter), using sttp-client 4, Scala 3 and jsoniter-scala for JSON * [scala-sttp](https://openapi-generator.tech/docs/generators/scala-sttp), using sttp-client 4, json4s or circe for JSON ### Standalone setup This is the simplest setup which relies on calling openapi-generator manually and generating a complete sbt project from it. First, you will need to install/download openapi-generator. Follow openapi-generator's [official documentation](https://github.com/OpenAPITools/openapi-generator#1---installation) on how to do this. Next, call the generator with the following options: ```bash openapi-generator-cli generate \ -i petstore.yaml \ --generator-name scala-sttp4-jsoniter \ -o samples/client/petstore/ ``` ### Maven managed For maven project use plugin [openapi-generator-maven-plugin](https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator-maven-plugin) ### Sbt managed In this setup openapi-generator is plugged into sbt project through the [sbt-openapi-generator](https://github.com/OpenAPITools/sbt-openapi-generator/) plugin. Sttp requests and models are automatically generated upon compilation. To have your openapi descriptions automatically turned into classes first define a new module in your project: ```scala lazy val petstoreApi: Project = project .in(file("petstore-api")) .settings( openApiInputSpec := s"${baseDirectory.value.getPath}/petstore.yaml", openApiGeneratorName := "scala-sttp4-jsoniter", openApiOutputDir := baseDirectory.value.name, libraryDependencies ++= Seq( "com.softwaremill.sttp.client4" %% "core" % "4.0.25", "com.softwaremill.sttp.client4" %% "json4s" % "4.0.25", "org.json4s" %% "json4s-jackson" % "3.6.8" ) ) ``` As this will generate code into `petstore-api/src` you might want to add this folder to the `.gitignore`. Since this plugin is still in a very early stage it requires some additional configuration. First we need to connect generation with compilation. Add following line into petstore module settings: ```scala (compile in Compile) := ((compile in Compile) dependsOn openApiGenerate).value, ``` Now we have to attach our generated source code directory into cleaning process. Add following line into petstore module settings: ```scala cleanFiles += baseDirectory.value / "src" ``` Last but not least we need to tell openapi-generator not to generate whole project but only the source files (without the sbt build file): Add following line into petstore module settings: ```scala openApiIgnoreFileOverride := s"${baseDirectory.in(ThisBuild).value.getPath}/openapi-ignore-file", ``` and create `openapi-ignore-file` file in project's root directory with following content: ``` * **/* !**/src/main/scala/**/* ``` Final petstore module configuration: ```scala lazy val petstoreApi: Project = project .in(file("petstore-api")) .settings( openApiInputSpec := s"${baseDirectory.value.getPath}/petstore.yaml", openApiGeneratorName := "scala-sttp4-jsoniter", openApiOutputDir := baseDirectory.value.name, openApiIgnoreFileOverride := s"${baseDirectory.in(ThisBuild).value.getPath}/openapi-ignore-file", libraryDependencies ++= Seq( "com.softwaremill.sttp.client4" %% "core" % "4.0.25", "com.softwaremill.sttp.client4" %% "json4s" % "4.0.25", "org.json4s" %% "json4s-jackson" % "3.6.8" ), (compile in Compile) := ((compile in Compile) dependsOn openApiGenerate).value, cleanFiles += baseDirectory.value / "src" ) ``` Full demo project is available on [github](https://github.com/softwaremill/sttp-openapi-example). #### Additional notes Although recent versions of the IntelliJ IDEA IDE come with "OpenApi Specification" plugin bundled into it, this plugin doesn't seem to support latest versions of generator and so, it might be impossible to generate sttp bindings from it. # Resilience Resilience covers areas such as retries, circuit breaking and rate limiting. sttp client doesn't have the above built-in, as these concepts are usually best handled on a higher level. Sending a request (that is, invoking `myRequest.send(backend)`), can be viewed as a: * `() => Response[T]` function for synchronous backends * `() => Future[Response[T]]` for `Future`-based asynchronous backends * `IO[Response[T]]`/`Task[Response[T]]` process description All of these are lazily evaluated, and can be repeated. Such a representation allows to integrate the `send()` side-effect with a stack-dependent resilience tool. There's a number of libraries that implement the above mentioned resilience functionalities, hence there's no sense for sttp client to reimplement any of those. That's simply not the scope of this library. Still, the input for a particular resilience model might involve both the result (either an exception, or a response) and the original description of the request being sent. E.g. retries can depend on the request method; circuit-breaking can depend on the host, to which the request is sent; same for rate limiting. ## Retries Handling retries is a complex problem when it comes to HTTP requests. When is a request retryable? There are a couple of things to take into account: * connection exceptions are generally good candidates for retries * only idempotent HTTP methods (such as `GET`) could potentially be retried * some HTTP status codes might also be retryable (e.g. `500 Internal Server Error` or `503 Service Unavailable`) In some cases it's possible to implement a generic retry mechanism; such a mechanism should take into account logging, metrics, limiting the number of retries and a backoff mechanism. These mechanisms could be quite simple, or involve e.g. retry budgets (see [Finagle's](https://twitter.github.io/finagle/guide/Clients.html#retries) documentation on retries). In sttp, it's possible to recover from errors using the `monad`. sttp client contains a default implementation of a predicate, which allows deciding if a request is retriable: if the body can be sent multiple times, and if the HTTP method is idempotent. This predicate is available as `RetryWhen.Default` and has type `(GenericRequest[_, _], Either[Throwable, Response[_]]) => Boolean`. Here's an incomplete list of libraries which can be used to manage retries in various Scala stacks: * for synchornous/direct-style: [Ox](https://github.com/softwaremill/ox) * for `Future`: [retry](https://github.com/softwaremill/retry) * for ZIO: [schedules](https://zio.dev/reference/schedule/), [rezilience](https://github.com/svroonland/rezilience) * for Monix/cats-effect: [cats-retry](https://github.com/cb372/cats-retry) * for Monix: `.restart` methods See also the "resiliency" and "backend wrapper" [examples](../examples.md). ### Backend-specific retries Some backends have built-in retry mechanisms: * [akka-http](https://doc.akka.io/docs/akka-http/current/scala/http/client-side/host-level.html#retrying-a-request) * [OkHttp](http://square.github.io/okhttp) (see the builder's `retryOnConnectionFailure` method) ## Circuit breaking * for Monix & cats-effect: [monix-catnap](https://monix.io/docs/3x/#monix-catnap) * for Akka/`Future`: [akka circuit breaker](https://doc.akka.io/docs/akka/current/common/circuitbreaker.html) * for ZIO: [rezilience](https://github.com/svroonland/rezilience) ## Rate limiting * for synchornous/direct-style: [Ox](https://github.com/softwaremill/ox) * for Akka Streams: [throttle in akka streams](https://doc.akka.io/docs/akka/current/stream/operators/Source-or-Flow/throttle.html) * for ZIO: [rezilience](https://github.com/svroonland/rezilience) ## Java libraries * [resilience4j](https://github.com/resilience4j/resilience4j) (rate limiting, circuit breaking, retries, other resilience patterns) # Server-sent events All backends that support [streaming](../requests/streaming.md) also support [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events). Moreover, synchronous backends can also support SSE, when combined with [Ox](https://ox.softwaremill.com). Refer to the documentation of individual backends, for more information on how to use SSE with a given backend, as well as usage examples. # XML Adding XML encoding/decoding support is a matter of providing a [body serializer](https://sttp.softwaremill.com/en/latest/requests/body.html) and/or a [response body specification](https://sttp.softwaremill.com/en/latest/responses/body.html) (similarly as it is done for JSON format). The process of adding integrations is fairly easy, and for now, one guide on how to use scalaxb tool is provided. ## scalaxb If you possess the XML Schema definition file (`.xsd` file) consider using the scalaxb tool, which would generate needed models and serialization/deserialization logic. To use the tool please follow the documentation on [setting up](https://scalaxb.org/setup) and [running](https://scalaxb.org/running-scalaxb) scalaxb. After code generation, create an `SttpScalaxbApi` trait (or trait with another name of your choosing) and add the following code snippet: ```scala import generated.defaultScope // import may differ depending on location of generated code import scalaxb.`package`.{fromXML, toXML} // import may differ depending on location of generated code import scalaxb.{CanWriteXML, XMLFormat} // import may differ depending on location of generated code import sttp.client4.{ResponseAs, ResponseException, StringBody, asString} import sttp.model.MediaType import scala.xml.{NodeSeq, XML} trait SttpScalaxbApi: case class XmlElementLabel(label: String) // request body def asXml[B](b: B)(implicit format: CanWriteXML[B], label: XmlElementLabel): StringBody = val nodeSeq: NodeSeq = toXML[B](obj = b, elementLabel = label.label, scope = defaultScope) StringBody(nodeSeq.toString(), "utf-8", MediaType.ApplicationXml) private def deserializeXml[B](implicit decoder: XMLFormat[B]): String => Either[Exception, B] = (s: String) => try Right(fromXML[B](XML.loadString(s))) catch case e: Exception => Left(e) // response body handling description def asXml[B: XMLFormat]: ResponseAs[Either[ResponseException[String], B], Any] = asString.mapWithMetadata(ResponseAs.deserializeRightWithError(deserializeXml[B])) .showAs("either(as string, as xml)") ``` This would add `asXml` methods needed for serialization and deserialization. Please notice, that `fromXML`, `toXML`, `CanWriteXML`, `XMLFormat` and `defaultScope` are members of code generated by scalaxb. Next to this trait, you might want to introduce `sttpScalaxb` package object to simplify imports. ```scala package object sttpScalaxb extends SttpScalaxbApi ``` From now on, XML serialization/deserialization would work for all classes generated from `.xsd` file as long as `XMLFormat` for the type in the question and `XmlElementLabel` for the top XML node would be implicitly provided in the scope. Usage example: ```scala val backend: SyncBackend = DefaultSyncBackend() // `Outer` and `Inner` classes are generated by scalaxb from xsd file val requestPayload = Outer(Inner(42, b = true, "horses"), "cats") // imports sttp related serialization / deserialization logic import sttpScalaxb.* // gives needed XmlElementLabel for the top XML node given XmlElementLabel = XmlElementLabel("outer") // imports member of code generated by scalaxb, that provides `XMLFormat` for `Outer` type; // this import may differ depending on location of generated code import generated.Generated_OuterFormat val response: Response[Either[ResponseException[String], Outer]] = basicRequest .post(uri"...") .body(asXml(requestPayload)) .response(asXml[Outer]) .send(backend) ``` # Supported backends sttp supports a number of synchronous and asynchronous backends. It's the backends that take care of managing connections, sending requests and receiving responses: sttp defines only the API to describe the requests to be sent and handle the response data. Backends do all the heavy-lifting. Typically, a single backend instance is created for the lifetime of the application. Choosing the right backend depends on a number of factors: whether you are using sttp to explore some data, or is it a production system; are you using a synchronous, blocking architecture, or an asynchronous one; do you work mostly with Scala's `Future`, or maybe you use some form of a `Task` abstraction; finally, if you want to stream requests/responses, or not. ## Default backends As a starting point, the default backends are good choice. Depending on the platform, the following are available: * on the JVM, `DefaultSyncBackend` and `DefaultFutureBackend`: both based on Java's HTTP client * on JS, `DefaultFutureBackend`, based on Fetch * on Native, `DefaultSyncBackend`, based on curl These default backends provide limited customisation options, hence for any more advanced use-cases, simply substitute them with a specific implementation. E.g. the `HttpClientSyncBackend` backend, which is the underlying implementation of `DefaultSyncBackend`, offers customisation options not available in the default one. ## JVM backends Which one to choose? * for simple exploratory requests, use the [synchronous](synchronous.md) `DefaultSyncBackend` / `HttpClientSyncBackend`. * if you have Akka in your stack, use the [Akka backend](akka.md) * if you have Pekko in your stack, use the [Pekko backend](pekko.md) * if you are using `Future` without Akka, use the `DefaultFutureBackend` / `HttpClientFutureBackend` * finally, if you are using a functional effect wrapper, use one of the "functional" backends, for [ZIO](zio.md), [Monix](monix.md), [Scalaz](scalaz.md), [cats-effect](catseffect.md) or [fs2](fs2.md). Below is a summary of all the JVM backends; see the sections on individual backend implementations for more information: ```{eval-rst} ==================================== ================================ ============================================================ ========================== =================== Class Effect type Supported stream type Supports websockets Fully non-blocking ==================================== ================================ ============================================================ ========================== =================== ``DefaultSyncBackend`` None (``Identity``) ``java.io.InputStream`` (blocking) & ``ox.flow.Flow`` ¹ yes (regular & streaming¹) no ``HttpClientSyncBackend`` None (``Identity``) ``java.io.InputStream`` (blocking) & ``ox.flow.Flow`` ¹ yes (regular & streaming¹) no ``DefaultFutureBackend`` ``scala.concurrent.Future`` ``java.io.InputStream`` (blocking) yes (regular) no ``HttpClientFutureBackend`` ``scala.concurrent.Future`` ``java.io.InputStream`` (blocking) yes (regular) no ``HttpClientMonixBackend`` ``monix.eval.Task`` ``monix.reactive.Observable[ByteBuffer]`` yes (regular & streaming) yes ``HttpClientFs2Backend`` ``F[_]: cats.effect.Concurrent`` ``fs2.Stream[F, Byte]`` yes (regular & streaming) yes ``HttpClientZioBackend`` ``zio.Task`` ``zio.stream.Stream[Throwable, Byte]`` yes (regular & streaming) yes ``HttpURLConnectionBackend`` None (``Identity``) ``java.io.InputStream`` (blocking) & ``ox.flow.Flow`` ¹ no no ``TryHttpURLConnectionBackend`` ``scala.util.Try`` ``java.io.InputStream`` (blocking) & ``ox.flow.Flow`` ¹ no no ``AkkaHttpBackend`` ``scala.concurrent.Future`` ``akka.stream.scaladsl.Source[ByteString, Any]`` yes (regular & streaming) yes ``PekkoHttpBackend`` ``scala.concurrent.Future`` ``org.apache.pekko.stream.scaladsl.Source[ByteString, Any]`` yes (regular & streaming) yes ``ArmeriaFutureBackend`` ``scala.concurrent.Future`` n/a no yes ``ArmeriaScalazBackend`` ``scalaz.concurrent.Task`` n/a no yes ``ArmeriaZioBackend`` ``zio.Task`` ``zio.stream.Stream[Throwable, Byte]`` no yes ``ArmeriaMonixBackend`` ``monix.eval.Task`` ``monix.reactive.Observable[HttpData]`` no yes ``ArmeriaCatsBackend`` ``F[_]: cats.effect.Concurrent`` n/a no yes ``ArmeriaFs2Backend`` ``F[_]: cats.effect.Concurrent`` ``fs2.Stream[F, Byte]`` no yes ``OkHttpSyncBackend`` None (``Identity``) ``java.io.InputStream`` (blocking) & ``ox.flow.Flow`` ¹ yes (regular & streaming¹) no ``OkHttpFutureBackend`` ``scala.concurrent.Future`` ``java.io.InputStream`` (blocking) yes (regular) no ``OkHttpMonixBackend`` ``monix.eval.Task`` ``monix.reactive.Observable[ByteBuffer]`` yes (regular & streaming) no ``Http4sBackend`` ``F[_]: cats.effect.Effect`` ``fs2.Stream[F, Byte]`` no no ``FinagleBackend`` ``com.twitter.util.Future`` n/a no no ==================================== ================================ ============================================================ ========================== =================== ``` The backends work with Scala 2.12, 2.13 and 3. Backends supporting cats-effect are available in versions for cats-effect 2.x (dependency artifacts have the `-ce2` suffix) and 3.x. All backends that support asynchronous/non-blocking streams, also support server-sent events. ¹ The synchronous backends support streaming & streaming web sockets through Ox `Flow`s, which is available on Java 21+. See section on [synchronous backends](synchronous.md) for more details. ## Backend wrappers There are also backends which wrap other backends to provide additional functionality. These include: * `TryBackend`, which safely wraps any exceptions thrown by a synchronous backend in `scala.util.Try` * `EitherBackend`, which represents exceptions as the left side of an `Either` * `OpenTelemetryTracingBackend`, for OpenTelemetry-compatible distributed tracing. See the [dedicated section](wrappers/opentelemetry.md). * `OpenTelemetryMetricsBackend`, for OpenTelemetry-compatible metrics. See the [dedicated section](wrappers/opentelemetry.md). * `PrometheusBackend`, for gathering Prometheus-format metrics. See the [dedicated section](wrappers/prometheus.md). * extendable logging backends (with an slf4j implementation) backends. See the [dedicated section](wrappers/logging.md). * `ResolveRelativeUrisBackend` to resolve relative URIs given a base URI, or an arbitrary effectful function * `ListenerBackend` to listen for backend lifecycle events. See the [dedicated section](wrappers/custom.md). * `FollowRedirectsBackend`, which handles redirects. All implementation backends are created wrapped with this one. * `CachingBackend`, which caches responses. See the [dedicated section](wrappers/cache.md). ## Scala.JS backends In addition, there are also backends for Scala.JS: ```{eval-rst} ================================ ================================ ========================================= =================== Class Effect type Supported stream type Supports websockets ================================ ================================ ========================================= =================== ``DefaultFutureBackend`` ``scala.concurrent.Future`` n/a yes (regular) ``FetchBackend`` ``scala.concurrent.Future`` n/a yes (regular) ``FetchMonixBackend`` ``monix.eval.Task`` ``monix.reactive.Observable[ByteBuffer]`` yes (regular & streaming) ``FetchZioBackend`` ``zio.Task`` ``zio.stream.Stream[Throwable, Byte]`` yes (regular & streaming) ``FetchCatsBackend`` ``F[_]: cats.effect.Concurrent`` n/a yes (regular) ================================ ================================ ========================================= =================== ``` ## Scala Native backends And a backend for scala-native: ```{eval-rst} ================================ ============================ ========================================= =================== Class Effect type Supported stream type Supports websockets ================================ ============================ ========================================= =================== ``DefaultSyncBackend`` None (``Identity``) n/a no ``CurlBackend`` None (``Identity``) n/a no ================================ ============================ ========================================= =================== ``` ## Backend types Depending on the capabilities that a backend supports, the exact backend type differs: * `SyncBackend` are backends which are synchronous, blocking, and don't support streaming. * `Backend[F]` are backends which don't support streaming or web sockets, and use `F` to represent side effects (e.g. obtaining a response for a request) * `StreamBackend[F, S]` are backends which support streaming, use `F` to represent side effects, and `S` to represent streams of bytes. * `WebSocketBackend[F]` are backends which support web sockets, use `F` to represent side effects * `WebSocketStreamBackend[F, S]` are backends which support web sockets and streaming, use `F` to represent side effects, and `S` to represent streams of bytes. Each backend type extends `GenericBackend` has two type parameters: * `F[_]`, the type constructor used to represent side effects. That is, when you invoke `send(backend)` on a request description, do you get a `Response[_]` directly, or is it wrapped in a `Future` or a `Task`? * `P`, the capabilities supported by the backend, in addition to `Effect[F]`. If `Any`, no additional capabilities are provided. Might include `Streams` (the ability to send and receive streaming bodies) and `WebSockets` (the ability to handle websocket requests). # Akka backend This backend is based on [akka-http](http://doc.akka.io/docs/akka-http/current/scala/http/). To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "akka-http-backend" % "4.0.25" ``` A fully **asynchronous** backend. Uses the `Future` effect to return responses. There are also [other `Future`-based backends](future.md), which don't depend on Akka. Note that you'll also need an explicit dependency on akka-streams, as akka-http doesn't depend on any specific akka-streams version. So you'll also need to add, for example: ``` "com.typesafe.akka" %% "akka-stream" % "2.6.20" ``` Next you'll need to add create the backend instance: ```scala import sttp.client4.akkahttp._ val backend = AkkaHttpBackend() ``` or, if you'd like to use an existing actor system: ```scala import sttp.client4.akkahttp._ import akka.actor.ActorSystem val actorSystem: ActorSystem = ??? val backend = AkkaHttpBackend.usingActorSystem(actorSystem) ``` This backend supports sending and receiving [akka-streams](http://doc.akka.io/docs/akka/current/scala/stream/index.html) streams. The streams capability is represented as `sttp.client4.akkahttp.AkkaStreams`. To set the request body as a stream: ```scala import sttp.capabilities.akka.AkkaStreams import sttp.client4._ import akka.stream.scaladsl.Source import akka.util.ByteString val source: Source[ByteString, Any] = ??? basicRequest .post(uri"...") .streamBody(AkkaStreams)(source) ``` To receive the response body as a stream: ```scala import scala.concurrent.Future import sttp.capabilities.akka.AkkaStreams import sttp.client4._ import sttp.client4.akkahttp.AkkaHttpBackend import akka.stream.scaladsl.Source import akka.util.ByteString val backend = AkkaHttpBackend() val response: Future[Response[Either[String, Source[ByteString, Any]]]] = basicRequest .post(uri"...") .response(asStreamUnsafe(AkkaStreams)) .send(backend) ``` The akka-http backend support both regular and streaming [websockets](../other/websockets.md). ## Testing Apart from testing using [the stub](../testing/stub.md), you can create a backend using any `HttpRequest => Future[HttpResponse]` function, or an akka-http `Route`. That way, you can "mock" a server that the backend will talk to, without starting any actual server or making any HTTP calls. If your application provides a client library for its dependants to use, this is a great way to ensure that the client actually matches the routes exposed by your application: ```scala import sttp.client4.akkahttp._ import akka.http.scaladsl.server.Route import akka.actor.ActorSystem val route: Route = ??? implicit val system: ActorSystem = ??? val backend = AkkaHttpBackend.usingClient(system, http = AkkaHttpClient.stubFromRoute(route)) ``` ## WebSockets Non-standard behavior: * akka always automatically responds with a `Pong` to a `Ping` message * `WebSocketFrame.Ping` and `WebSocketFrame.Pong` frames are ignored; instead, you can configure automatic [keep-alive pings](https://doc.akka.io/docs/akka-http/current/client-side/websocket-support.html#automatic-keep-alive-ping-support) ## Server-sent events Received data streams can be parsed to a stream of server-sent events (SSE): ```scala import scala.concurrent.Future import akka.stream.scaladsl.Source import sttp.capabilities.akka.AkkaStreams import sttp.client4.akkahttp.AkkaHttpServerSentEvents import sttp.model.sse.ServerSentEvent import sttp.client4._ def processEvents(source: Source[ServerSentEvent, Any]): Future[Unit] = ??? basicRequest .get(uri"...") .response(asStream(AkkaStreams)(stream => processEvents(stream.via(AkkaHttpServerSentEvents.parse)))) ``` # cats-effect backend The [Cats Effect](https://github.com/typelevel/cats-effect) backend is **asynchronous**. It can be created for any type implementing the `cats.effect.kernel.Async` typeclass, such as `cats.effect.IO`. Sending a request is a non-blocking, lazily-evaluated operation and results in a wrapped response. There's a transitive dependency on `cats-effect`. Note that all [fs2](fs2.md) backends also support any cats-effect effect, additionally supporting request & response streaming. Also note that the [http4s](http4s.md) backend can also be created for a type implementing the cats-effect’s `Async` typeclass, and supports streaming as in [fs2](fs2.md). ## Using HttpClient Firstly, add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "cats" % "4.0.25" ``` Obtain a cats-effect `Resource` which creates the backend, and closes the thread pool after the resource is no longer used: ```scala import cats.effect.IO import sttp.client4.httpclient.cats.HttpClientCatsBackend HttpClientCatsBackend.resource[IO]().use { backend => ??? } ``` or, by providing a custom `Dispatcher`: ```scala import cats.effect.IO import cats.effect.std.Dispatcher import sttp.client4.httpclient.cats.HttpClientCatsBackend val dispatcher: Dispatcher[IO] = ??? HttpClientCatsBackend[IO](dispatcher).flatMap { backend => ??? } ``` or, if you'd like to instantiate the `HttpClient` yourself: ```scala import cats.effect.IO import cats.effect.std.Dispatcher import java.net.http.HttpClient import sttp.client4.httpclient.cats.HttpClientCatsBackend val httpClient: HttpClient = ??? val dispatcher: Dispatcher[IO] = ??? val backend = HttpClientCatsBackend.usingClient[IO](httpClient, dispatcher) ``` or, obtain a cats-effect `Resource` with a custom instance of the `HttpClient`: ```scala import cats.effect.IO import java.net.http.HttpClient import sttp.client4.httpclient.cats.HttpClientCatsBackend val httpClient: HttpClient = ??? HttpClientCatsBackend.resourceUsingClient[IO](httpClient).use { backend => ??? } ``` This backend is based on the built-in `java.net.http.HttpClient` available from Java 11 onwards. Host header override is supported in environments running Java 12 onwards, but it has to be enabled by system property: ``` -Djdk.httpclient.allowRestrictedHeaders=host ``` ## Using Armeria Creation of the backend can be done in two basic ways: * by creating an effect, which describes how the backend is created, or instantiating the backend directly. In this case, you'll need to close the backend manually * by creating a `Resource`, which will instantiate the backend and close it after it has been used. Firstly, add the following dependency to your project: ```scala // for cats-effect 3.x "com.softwaremill.sttp.client4" %% "armeria-backend-cats" % "4.0.25" // or for cats-effect 2.x "com.softwaremill.sttp.client4" %% "armeria-backend-cats-ce2" % "4.0.25" ``` create client: ```scala import cats.effect.IO import sttp.client4.armeria.cats.ArmeriaCatsBackend val backend = ArmeriaCatsBackend[IO]() // You can use the default client which reuses the connection pool of ClientFactory.ofDefault() ArmeriaCatsBackend.usingDefaultClient[IO]() ``` or, if you'd like the backend to be wrapped in cats-effect `Resource`: ```scala import cats.effect.IO import sttp.client4.armeria.cats.ArmeriaCatsBackend ArmeriaCatsBackend.resource[IO]().use { backend => ??? } ``` or, if you'd like to instantiate the [WebClient](https://armeria.dev/docs/client-http) yourself: ```scala import cats.effect.IO import com.linecorp.armeria.client.WebClient import com.linecorp.armeria.client.circuitbreaker._ import sttp.client4.armeria.cats.ArmeriaCatsBackend // Fluently build Armeria WebClient with built-in decorators val client = WebClient.builder("https://my-service.com") // Open circuit on 5xx server error status .decorator(CircuitBreakerClient.newDecorator(CircuitBreaker.ofDefaultName(), CircuitBreakerRule.onServerErrorStatus())) .build() val backend = ArmeriaCatsBackend.usingClient[IO](client) ``` ```{note} A WebClient could fail to follow redirects if the WebClient is created with a base URI and a redirect location is a different URI. ``` This backend is build on top of [Armeria](https://armeria.dev/docs/client-http). Armeria's [ClientFactory](https://armeria.dev/docs/client-factory) manages connections and protocol-specific properties. Please visit [the official documentation](https://armeria.dev/docs/client-factory) to learn how to configure it. ## Streaming This backend doesn't support non-blocking [streaming](../requests/streaming.md). ## Websockets The backend doesn't support [websockets](../other/websockets.md). # Twitter future (Finagle) backend To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "finagle-backend" % "4.0.25" ``` Next you'll need to add an implicit value: ```scala import sttp.client4.finagle.FinagleBackend val backend = FinagleBackend() ``` This backend depends on [finagle](https://twitter.github.io/finagle/), and offers an asynchronous backend, which wraps results in Twitter's `Future`. Please note that: * the backend does not support non-blocking [streaming](../requests/streaming.md) or [websockets](../other/websockets.md). # fs2 backend The [fs2](https://github.com/functional-streams-for-scala/fs2) backends are **asynchronous**. They can be created for any type implementing the `cats.effect.kernel.Async` typeclass, such as `cats.effect.IO`. Sending a request is a non-blocking, lazily-evaluated operation and results in a wrapped response. There's a transitive dependency on `cats-effect`. ## Using HttpClient Creation of the backend can be done in two basic ways: * by creating a `Resource`, which will instantiate the backend and close it after it has been used. * by creating an effect, which describes how a backend is created, or instantiating the backend directly. In this case, you'll need to close the backend manually, as well as provide a `Dispatcher` instance Firstly, add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "fs2" % "4.0.25" // for cats-effect 3.x & fs2 3.x // or "com.softwaremill.sttp.client4" %% "fs2ce2" % "4.0.25" // for cats-effect 2.x & fs2 2.x ``` Obtain a cats-effect `Resource` which creates the backend, and closes the thread pool after the resource is no longer used: ```scala import cats.effect.IO import sttp.client4.httpclient.fs2.HttpClientFs2Backend HttpClientFs2Backend.resource[IO]().use { backend => ??? } ``` or, by providing a custom `Dispatcher`: ```scala import cats.effect.IO import cats.effect.std.Dispatcher import sttp.client4.httpclient.fs2.HttpClientFs2Backend val dispatcher: Dispatcher[IO] = ??? HttpClientFs2Backend[IO](dispatcher).flatMap { backend => ??? } ``` or, if you'd like to instantiate the `HttpClient` yourself: ```scala import cats.effect.IO import cats.effect.std.Dispatcher import java.net.http.HttpClient import sttp.client4.httpclient.fs2.HttpClientFs2Backend val httpClient: HttpClient = ??? val dispatcher: Dispatcher[IO] = ??? val backend = HttpClientFs2Backend.usingClient[IO](httpClient, dispatcher) ``` or, obtain a cats-effect `Resource` with a custom instance of the `HttpClient`: ```scala import cats.effect.IO import java.net.http.HttpClient import sttp.client4.httpclient.fs2.HttpClientFs2Backend val httpClient: HttpClient = ??? HttpClientFs2Backend.resourceUsingClient[IO](httpClient).use { backend => ??? } ``` This backend is based on the built-in `java.net.http.HttpClient` available from Java 11 onwards. Host header override is supported in environments running Java 12 onwards, but it has to be enabled by system property: ``` -Djdk.httpclient.allowRestrictedHeaders=host ``` ## Using Armeria To use, add the following dependency to your project: ```scala // for cats-effect 3.x & fs2 3.x "com.softwaremill.sttp.client4" %% "armeria-backend-fs2" % "4.0.25" // or for cats-effect 2.x & fs2 2.x "com.softwaremill.sttp.client4" %% "armeria-backend-fs2-ce2" % "4.0.25" ``` create client: ```scala import cats.effect.IO import cats.effect.std.Dispatcher import sttp.client4.armeria.fs2.ArmeriaFs2Backend ArmeriaFs2Backend.resource[IO]().use { backend => ??? } val dispatcher: Dispatcher[IO] = ??? // You can use the default client which reuses the connection pool of ClientFactory.ofDefault() ArmeriaFs2Backend.usingDefaultClient[IO](dispatcher) ``` or, if you'd like to instantiate the [WebClient](https://armeria.dev/docs/client-http) yourself: ```scala import cats.effect.IO import cats.effect.std.Dispatcher import com.linecorp.armeria.client.WebClient import com.linecorp.armeria.client.circuitbreaker.* import sttp.client4.armeria.fs2.ArmeriaFs2Backend val dispatcher: Dispatcher[IO] = ??? // Fluently build Armeria WebClient with built-in decorators val client = WebClient.builder("https://my-service.com") // Open circuit on 5xx server error status .decorator(CircuitBreakerClient.newDecorator(CircuitBreaker.ofDefaultName(), CircuitBreakerRule.onServerErrorStatus())) .build() val backend = ArmeriaFs2Backend.usingClient[IO](client, dispatcher) ``` ```{note} A WebClient could fail to follow redirects if the WebClient is created with a base URI and a redirect location is a different URI. ``` This backend is built on top of [Armeria](https://armeria.dev/docs/client-http). Armeria's [ClientFactory](https://armeria.dev/docs/client-factory) manages connections and protocol-specific properties. Please visit [the official documentation](https://armeria.dev/docs/client-factory) to learn how to configure it. ## Using JavaScript The Fs2 backend is also available for the JS platform, see [the `FetchBackend` documentation](javascript/fetch.md). The `FetchFs2Backend` companion object contains methods to create the backend directly. ## Streaming The fs2 backends support streaming for any instance of the `cats.effect.Effect` typeclass, such as `cats.effect.IO`. If `IO` is used then the type of supported streams is `fs2.Stream[IO, Byte]`. The streams capability is represented as `sttp.client4.fs2.Fs2Streams`. Requests can be sent with a streaming body like this: ```scala import cats.effect.IO import fs2.Stream import sttp.capabilities.fs2.Fs2Streams import sttp.client4.* import sttp.client4.httpclient.fs2.HttpClientFs2Backend val effect = HttpClientFs2Backend.resource[IO]().use { backend => val stream: Stream[IO, Byte] = ??? basicRequest .post(uri"...") .streamBody(Fs2Streams[IO])(stream) .send(backend) } // run the effect ``` Responses can also be streamed: ```scala import cats.effect.IO import fs2.Stream import sttp.capabilities.fs2.Fs2Streams import sttp.client4.* import sttp.client4.httpclient.fs2.HttpClientFs2Backend import scala.concurrent.duration.Duration val effect = HttpClientFs2Backend.resource[IO]().use { backend => val response: IO[Response[Either[String, Stream[IO, Byte]]]] = basicRequest .post(uri"...") .response(asStreamUnsafe(Fs2Streams[IO])) .readTimeout(Duration.Inf) .send(backend) response } // run the effect ``` ## Websockets The fs2 backends support both regular and streaming [websockets](../other/websockets.md). ## Server-sent events Received data streams can be parsed to a stream of server-sent events (SSE): ```scala import cats.effect.* import fs2.Stream import sttp.client4.* import sttp.capabilities.fs2.Fs2Streams import sttp.client4.impl.fs2.Fs2ServerSentEvents import sttp.model.sse.ServerSentEvent def processEvents(source: Stream[IO, ServerSentEvent]): IO[Unit] = ??? basicRequest .get(uri"") .response(asStream(Fs2Streams[IO])(stream => processEvents(stream.through(Fs2ServerSentEvents.parse)))) ``` # Future-based backends There are several backend implementations which are `scala.concurrent.Future`-based. These backends are **asynchronous**, sending a request is a non-blocking operation and results in a response wrapped in a `Future`. Apart from the ones described below, also the [Pekko](pekko.md) & [Akka](akka.md) backends are `Future`-based. ```{eval-rst} ===================================== ================================================= ========================== Class Supported stream type Websocket support ===================================== ================================================= ========================== ``HttpClientFutureBackend`` n/a yes (regular) ``PekkoHttpBackend`` ``pekko.stream.scaladsl.Source[ByteString, Any]`` yes (regular & streaming) ``AkkaHttpBackend`` ``akka.stream.scaladsl.Source[ByteString, Any]`` yes (regular & streaming) ``OkHttpFutureBackend`` n/a yes (regular) ``ArmeriaFutureBackend`` n/a n/a ===================================== ================================================= ========================== ``` ## Using HttpClient To use, you don't need any extra dependencies, `core` is enough: ``` "com.softwaremill.sttp.client4" %% "core" % "4.0.25" ``` You'll need the following imports: ```scala import sttp.client4.httpclient.HttpClientFutureBackend import scala.concurrent.ExecutionContext.Implicits.global ``` Create the backend using: ```scala val backend = HttpClientFutureBackend() ``` or, if you'd like to instantiate the HttpClient yourself: ```scala import java.net.http.HttpClient val client: HttpClient = ??? val backend = HttpClientFutureBackend.usingClient(client) ``` This backend is based on the built-in `java.net.http.HttpClient` available from Java 11 onwards. Host header override is supported in environments running Java 12 onwards, but it has to be enabled by system property: ``` -Djdk.httpclient.allowRestrictedHeaders=host ``` ## Using OkHttp To use, add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "okhttp-backend" % "4.0.25" ``` and some imports: ```scala import sttp.client4.okhttp.OkHttpFutureBackend import scala.concurrent.ExecutionContext.Implicits.global ``` Create the backend using: ```scala val backend = OkHttpFutureBackend() ``` or, if you'd like to instantiate the OkHttpClient yourself: ```scala import okhttp3.OkHttpClient val okHttpClient: OkHttpClient = ??? val backend = OkHttpFutureBackend.usingClient(okHttpClient) ``` This backend depends on [OkHttp](http://square.github.io/okhttp/) and fully supports HTTP/2. ## Using Armeria To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "armeria-backend" % "4.0.25" ``` add imports: ```scala import sttp.client4.armeria.future.ArmeriaFutureBackend ``` create client: ```scala val backend = ArmeriaFutureBackend() // You can use the default client which reuses the connection pool of ClientFactory.ofDefault() ArmeriaFutureBackend.usingDefaultClient() ``` or, if you'd like to instantiate the [WebClient](https://armeria.dev/docs/client-http) yourself:: ```scala import com.linecorp.armeria.client.circuitbreaker.* import com.linecorp.armeria.client.WebClient // Fluently build Armeria WebClient with built-in decorators val client = WebClient.builder("https://my-service.com") // Open circuit on 5xx server error status .decorator(CircuitBreakerClient.newDecorator(CircuitBreaker.ofDefaultName(), CircuitBreakerRule.onServerErrorStatus())) .build() val backend = ArmeriaFutureBackend.usingClient(client) ``` ```{note} A WebClient could fail to follow redirects if the WebClient is created with a base URI and a redirect location is a different URI. ``` This backend is build on top of [Armeria](https://armeria.dev/docs/client-http) and doesn't support host header override. Armeria's [ClientFactory](https://armeria.dev/docs/client-factory) manages connections and protocol-specific properties. Please visit [the official documentation](https://armeria.dev/docs/client-factory) to learn how to configure it. ## Streaming The [Akka backend](akka.md) supports streaming using akka-streams. Other backends don't support non-blocking [streaming](../requests/streaming.md). ## Websockets Some of the backends (see above) support regular and streaming [websockets](../other/websockets.md). # Http4s backend This backend is based on [http4s](https://http4s.org) (client) and is **asynchronous**. To use, add the following dependency to your project: ```scala // for cats-effect 3.x & http4s 1.0.0-Mx "com.softwaremill.sttp.client4" %% "http4s-backend" % "4.0.25" // or for cats-effect 2.x & http4s 0.21.x "com.softwaremill.sttp.client4" %% "http4s-ce2-backend" % "4.0.25" ``` The backend can be created in a couple of ways, e.g.: ```scala import cats.effect.* import sttp.capabilities.fs2.Fs2Streams import sttp.client4.* import sttp.client4.http4s.* // the "org.http4s" %% "http4s-ember-client" % http4sVersion dependency needs to be explicitly added Http4sBackend.usingDefaultEmberClientBuilder[IO](): Resource[IO, StreamBackend[IO, Fs2Streams[IO]]] // the "org.http4s" %% "http4s-blaze-client" % http4sVersion dependency needs to be explicitly added Http4sBackend.usingDefaultBlazeClientBuilder[IO](): Resource[IO, StreamBackend[IO, Fs2Streams[IO]]] ``` Sending a request is a non-blocking, lazily-evaluated operation and results in a wrapped response. There's a transitive dependency on `http4s`. There are also [other cats-effect-based backends](catseffect.md), which don't depend on http4s. ## Scala Native Support When using the http4s backend on Scala Native, the `EmberClientBuilder` methods require a `Network[F]` constraint in addition to `Async[F]`: ```scala import cats.effect.* import fs2.io.net.Network import sttp.capabilities.fs2.Fs2Streams import sttp.client4.* import sttp.client4.http4s.* // On Scala Native, this requires an implicit Network[IO] in scope Http4sBackend.usingDefaultEmberClientBuilder[IO](): Resource[IO, StreamBackend[IO, Fs2Streams[IO]]] ``` The `Network[IO]` instance is provided automatically when extending `cats.effect.IOApp` or can be summoned explicitly using `Network[IO]` where needed. This constraint is specific to Scala Native; on the JVM, only the `Async[F]` constraint is required. Note that the Blaze client is JVM-only and not available on Scala Native. ## Notes * the backend contains **optional** dependencies on `http4s-ember-client` and `http4s-blaze-client`, to provide the `Http4sBackend.usingEmberClientBuilder`, `Http4sBackend.usingBlazeClientBuilder`, `Http4sBackend.usingDefaultEmberClientBuilder` and `Http4sBackend.usingDefaultBlazeClientBuilder` methods. This makes the client usable with other http4s client implementations, without the need to depend on ember or blaze. * the backend does not support `SttpBackendOptions`, that is specifying proxy settings (proxies are not implemented in http4s, see [this issue](https://github.com/http4s/http4s/issues/251)), as well as configuring the connect timeout * the backend does not support the `RequestT.options.readTimeout` option Instead, all custom timeout configuration should be done by creating a `org.http4s.client.Client[F]`, using e.g. `org.http4s.client.blaze.BlazeClientBuilder[F]` and passing it to the appropriate method of the `Http4sBackend` object. The backend supports streaming using fs2. For usage details, see the documentation on [streaming using fs2](fs2.md). The backend doesn't support [websockets](../other/websockets.md). # Scala.js (Fetch) backend A JavaScript backend with web socket support. Implemented using the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API). ## `Future`-based This is the default backend, available in the main jar for JS. To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %%% "core" % "4.0.25" ``` And create the backend instance: ```scala val backend = FetchBackend() ``` Timeouts are handled via the new [AbortController](https://developer.mozilla.org/en-US/docs/Web/API/AbortController) class. As this class only recently appeared in browsers you may need to add a [polyfill](https://www.npmjs.com/package/abortcontroller-polyfill). As browsers do not allow access to redirect responses, if a request sets `followRedirects` to false then a redirect will cause the response to return an error. Note that `Fetch` does not pass cookies by default. If your request needs cookies then you will need to pass a `FetchOptions` instance with `credentials` set to either `RequestCredentials.same-origin` or `RequestCredentials.include` depending on your requirements. ## Monix-based To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %%% "monix" % "4.0.25" ``` And create the backend instance: ```scala val backend = FetchMonixBackend() ``` ## ZIO-based To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %%% "zio" % "4.0.25" ``` And create the backend instance: ```scala val backend = FetchZioBackend() ``` ## cats-effect-based Any effect implementing the cats-effect `Concurrent` typeclass can be used. To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %%% "cats" % "4.0.25" ``` If you are on Cats Effect 2 (CE2) you will need to add the CE2 specific dependency instead: ``` "com.softwaremill.sttp.client4" %%% "catsce2" % "4.0.25" ``` And create the backend instance: ```scala val backend = FetchCatsBackend[IO]() ``` ## fs2-based Any effect implementing the cats-effect `Async` typeclass can be used. To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %%% "fs2" % "4.0.25" ``` And create the backend instance: ```scala val backend = FetchFs2Backend[IO]() ``` ## Node.js ### CommonJS module Using `FetchBackend` is possible with [node-fetch](https://www.npmjs.com/package/node-fetch) module and [ws with isomorphic-ws](https://www.npmjs.com/package/ws) module for web sockets. The latest version of node fetch (3) is not available as a CommonJS module and you must hence use the version 2. ``` npm install --save node-fetch@2 isomorphic-ws ws ``` It has to be loaded into your runtime. This can be done in your main method as such: ```scala val g = scalajs.js.Dynamic.global.globalThis val nodeFetch = g.require("node-fetch") g.fetch = nodeFetch g.Headers = nodeFetch.Headers g.Request = nodeFetch.Request g.WebSocket = g.require("isomorphic-ws") ``` ### ESModule If your Scala.js application is bundled inside an [ESModule](https://www.scala-js.org/doc/project/module.html) ``` npm install --save node-fetch isomorphic-ws ws ``` It has to be loaded into your runtime as well. This can be done using the following code: ```scala @js.native @JSImport("node-fetch", JSImport.Namespace) val nodeFetch: js.Dynamic = js.native val g = scalajs.js.Dynamic.global.globalThis g.fetch = nodeFetch.default g.Headers = nodeFetch.Headers g.Request = nodeFetch.Request ``` And for the web sockets: ```scala @js.native @JSImport("isomorphic-ws", JSImport.Namespace) val ws: js.Dynamic = js.native g.WebSocket = ws.default ``` ## Streaming Streaming support is provided via `FetchMonixBackend`. Note that streaming support on Firefox is hidden behind a flag, see [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) for more information. To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %%% "monix" % "4.0.25" ``` An example of streaming a response: ```scala import sttp.client4.* import sttp.client4.impl.monix.* import java.nio.ByteBuffer import monix.eval.Task import monix.reactive.Observable val backend = FetchMonixBackend() val response: Task[Response[Observable[ByteBuffer]]] = sttp .post(uri"...") .response(asStreamUnsafe(MonixStreams)) .send(backend) ``` ```{note} Currently no browsers support passing a stream as the request body. As such, using the `Fetch` backend with a streaming request will result in it being converted into an in-memory array before being sent. Response bodies are returned as a "proper" stream. ``` ## Websockets The backend supports both regular and streaming [websockets](../../other/websockets.md). ## Server-sent events Received data streams can be parsed to a stream of server-sent events (SSE), when using the [Monix](../monix.md), [ZIO](../zio.md), or [Fs2](../fs2.md) variants - the respective documentation pages contain appropriate SSE examples. # Monix backends There are several backend implementations which are `monix.eval.Task`-based. These backends are **asynchronous**. Sending a request is a non-blocking, lazily-evaluated operation and results in a response wrapped in a `Task`. ## Using HttpClient Creation of the backend can be done in two basic ways: * by creating a `Task`, which describes how the backend is created, or instantiating the backend directly. In this case, you'll need to close the backend manually. * by creating a `Resource`, which will instantiate the backend and close it after it has been used. Firstly, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "monix" % "4.0.25" ``` and create the backend using: ```scala import sttp.client4.httpclient.monix.HttpClientMonixBackend HttpClientMonixBackend().flatMap { backend => ??? } // or, if you'd like the backend to be wrapped in cats-effect Resource: HttpClientMonixBackend.resource().use { backend => ??? } // or, if you'd like to instantiate the HttpClient yourself: import java.net.http.HttpClient val httpClient: HttpClient = ??? val backend = HttpClientMonixBackend.usingClient(httpClient) // or, obtain a cats-effect Resource with a custom instance of the HttpClient: import java.net.http.HttpClient val httpClient: HttpClient = ??? HttpClientMonixBackend.resourceUsingClient(httpClient).use { backend => ??? } ``` This backend is based on the built-in `java.net.http.HttpClient` available from Java 11 onwards. Host header override is supported in environments running Java 12 onwards, but it has to be enabled by system property: ``` -Djdk.httpclient.allowRestrictedHeaders=host ``` ## Using OkHttp To use, add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "okhttp-backend-monix" % "4.0.25" ``` Create the backend using: ```scala import sttp.client4.okhttp.monix.OkHttpMonixBackend OkHttpMonixBackend().flatMap { backend => ??? } // or, if you'd like the backend to be wrapped in cats-effect Resource: OkHttpMonixBackend.resource().use { backend => ??? } // or, if you'd like to instantiate the OkHttpClient yourself: import okhttp3._ val okHttpClient: OkHttpClient = ??? val backend = OkHttpMonixBackend.usingClient(okHttpClient) ``` This backend depends on [OkHttp](http://square.github.io/okhttp/) and fully supports HTTP/2. ## Using Armeria To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "armeria-backend-monix" % "4.0.25" ``` add imports: ```scala import sttp.client4.armeria.monix.ArmeriaMonixBackend ``` create client: ```scala import monix.execution.Scheduler.Implicits.global val backend = ArmeriaMonixBackend() // You can use the default client which reuses the connection pool of ClientFactory.ofDefault() ArmeriaMonixBackend.usingDefaultClient() ``` or, if you'd like to instantiate the [WebClient](https://armeria.dev/docs/client-http) yourself: ```scala import com.linecorp.armeria.client.circuitbreaker.* import com.linecorp.armeria.client.WebClient // Fluently build Armeria WebClient with built-in decorators val client = WebClient.builder("https://my-service.com") // Open circuit on 5xx server error status .decorator(CircuitBreakerClient.newDecorator(CircuitBreaker.ofDefaultName(), CircuitBreakerRule.onServerErrorStatus())) .build() val backend = ArmeriaMonixBackend.usingClient(client) ``` ```{note} A WebClient could fail to follow redirects if the WebClient is created with a base URI and a redirect location is a different URI. ``` This backend is build on top of [Armeria](https://armeria.dev/docs/client-http). Armeria's [ClientFactory](https://armeria.dev/docs/client-factory) manages connections and protocol-specific properties. Please visit [the official documentation](https://armeria.dev/docs/client-factory) to learn how to configure it. ## Streaming The Monix backends support streaming. The streams capability is represented as `sttp.client4.impl.monix.MonixStreams`. The type of supported streams in this case is `Observable[ByteBuffer]`. That is, you can set such an observable as a request body (using the http-client backend as an example, but any of the above backends can be used): ```scala import sttp.capabilities.monix.MonixStreams import sttp.client4.* import sttp.client4.httpclient.monix.HttpClientMonixBackend import monix.reactive.Observable HttpClientMonixBackend().flatMap { backend => val obs: Observable[Array[Byte]] = ??? basicRequest .streamBody(MonixStreams)(obs) .post(uri"...") .send(backend) } ``` And receive responses as an observable stream: ```scala import sttp.capabilities.monix.MonixStreams import sttp.client4.* import sttp.client4.httpclient.monix.HttpClientMonixBackend import monix.eval.Task import monix.reactive.Observable import scala.concurrent.duration.Duration HttpClientMonixBackend().flatMap { backend => val response: Task[Response[Either[String, Observable[Array[Byte]]]]] = basicRequest .post(uri"...") .response(asStreamUnsafe(MonixStreams)) .readTimeout(Duration.Inf) .send(backend) response } ``` ## Websockets The Monix backend supports both regular and streaming [websockets](../other/websockets.md). ## Server-sent events Received data streams can be parsed to a stream of server-sent events (SSE): ```scala import monix.reactive.Observable import monix.eval.Task import sttp.capabilities.monix.MonixStreams import sttp.client4.impl.monix.MonixServerSentEvents import sttp.model.sse.ServerSentEvent import sttp.client4.* def processEvents(source: Observable[ServerSentEvent]): Task[Unit] = ??? basicRequest.response(asStream(MonixStreams)(stream => processEvents(stream.transform(MonixServerSentEvents.parse)))) ``` # Scala Native (curl) backend A Scala Native (0.5.x) backend implemented using [Curl](https://github.com/curl/curl/blob/master/include/curl/curl.h). To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %%% "core" % "4.0.25" ``` and initialize one of the backends: ```scala import sttp.client4.curl.* val backend = CurlBackend() val tryBackend = CurlTryBackend() ``` You need to have an environment with Scala Native [setup](https://scala-native.readthedocs.io/en/latest/user/setup.html) with additionally installed `libcrypto` (included in OpenSSL) and `curl` in version `7.56.0` or newer. ## scala-cli example Try the following example: ```scala // hello.scala //> using platform native //> using dep com.softwaremill.sttp.client4::core_native0.5:4.0.25 import sttp.client4.* import sttp.client4.curl.CurlBackend @main def run(): Unit = val backend = CurlBackend() println(basicRequest.get(uri"http://httpbin.org/ip").send(backend)) ``` ## ZIO-based To use in an sbt project, add the following dependency: ``` "com.softwaremill.sttp.client4" %%% "zio" % 4.0.25 ``` Create the backend instance for example via `scoped()` which will also ensure that acquired resources (if any) are released once out of `Scope`: ```scala //> using platform native //> using nativeVersion 0.5.10 //> using scala 3 //> using dep com.softwaremill.sttp.client4::zio::4.0.25 import sttp.client4.* import sttp.client4.curl.zio.CurlZioBackend import zio.* object Main extends ZIOAppDefault: def run = for backend <- CurlZioBackend.scoped() res <- basicRequest.get(uri"http://httpbin.org/ip").send(backend) _ <- Console.printLine(res) yield () ``` # Pekko backend This backend is based on [pekko-http](https://pekko.apache.org/docs/pekko-http/current/). To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "pekko-http-backend" % "4.0.25" ``` A fully **asynchronous** backend. Uses the `Future` effect to return responses. There are also [other `Future`-based backends](future.md), which don't depend on Pekko. Note that you'll also need an explicit dependency on pekko-streams, as pekko-http doesn't depend on any specific pekko-streams version. So you'll also need to add, for example: ``` "org.apache.pekko" %% "pekko-stream" % "1.6.0" ``` Next you'll need to add create the backend instance: ```scala import sttp.client4.pekkohttp.* val backend = PekkoHttpBackend() ``` or, if you'd like to use an existing actor system: ```scala import sttp.client4.pekkohttp.* import org.apache.pekko.actor.ActorSystem val actorSystem: ActorSystem = ??? val backend = PekkoHttpBackend.usingActorSystem(actorSystem) ``` This backend supports sending and receiving [pekko-streams](https://pekko.apache.org/docs/pekko/current/stream/index.html) streams. The streams capability is represented as `sttp.client4.pekkohttp.PekkoStreams`. To set the request body as a stream: ```scala import sttp.capabilities.pekko.PekkoStreams import sttp.client4.* import org.apache.pekko import pekko.stream.scaladsl.Source import pekko.util.ByteString val source: Source[ByteString, Any] = ??? basicRequest .post(uri"...") .streamBody(PekkoStreams)(source) ``` To receive the response body as a stream: ```scala import scala.concurrent.Future import sttp.capabilities.pekko.PekkoStreams import sttp.client4.* import sttp.client4.pekkohttp.PekkoHttpBackend import org.apache.pekko import pekko.stream.scaladsl.Source import pekko.util.ByteString val backend = PekkoHttpBackend() val response: Future[Response[Either[String, Source[ByteString, Any]]]] = basicRequest .post(uri"...") .response(asStreamUnsafe(PekkoStreams)) .send(backend) ``` The pekko-http backend support both regular and streaming [websockets](../other/websockets.md). ## Testing Apart from testing using [the stub](../testing/stub.md), you can create a backend using any `HttpRequest => Future[HttpResponse]` function, or an pekko-http `Route`. That way, you can "mock" a server that the backend will talk to, without starting any actual server or making any HTTP calls. If your application provides a client library for its dependants to use, this is a great way to ensure that the client actually matches the routes exposed by your application: ```scala import sttp.client4.pekkohttp.* import org.apache.pekko import pekko.http.scaladsl.server.Route import pekko.actor.ActorSystem val route: Route = ??? implicit val system: ActorSystem = ??? val backend = PekkoHttpBackend.usingClient(system, http = PekkoHttpClient.stubFromRoute(route)) ``` ## WebSockets Non-standard behavior: * pekko always automatically responds with a `Pong` to a `Ping` message * `WebSocketFrame.Ping` and `WebSocketFrame.Pong` frames are ignored; instead, you can configure automatic [keep-alive pings](https://pekko.apache.org/docs/pekko-http/current/client-side/websocket-support.html#automatic-keep-alive-ping-support) ## Server-sent events Received data streams can be parsed to a stream of server-sent events (SSE): ```scala import scala.concurrent.Future import org.apache.pekko.stream.scaladsl.Source import sttp.capabilities.pekko.PekkoStreams import sttp.client4.pekkohttp.PekkoHttpServerSentEvents import sttp.model.sse.ServerSentEvent import sttp.client4._ def processEvents(source: Source[ServerSentEvent, Any]): Future[Unit] = ??? basicRequest .get(uri"...") .response(asStream(PekkoStreams)(stream => processEvents(stream.via(PekkoHttpServerSentEvents.parse)))) ``` # Scalaz backend The [Scalaz](https://github.com/scalaz/scalaz) backend is **asynchronous**. Sending a request is a non-blocking, lazily-evaluated operation and results in a response wrapped in a `scalaz.concurrent.Task`. There's a transitive dependency on `scalaz-concurrent`. ## Using Armeria To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "armeria-backend-scalaz" % "4.0.25" ``` add imports: ```scala import sttp.client4.armeria.scalaz.ArmeriaScalazBackend ``` create client: ```scala val backend = ArmeriaScalazBackend() // You can use the default client which reuses the connection pool of ClientFactory.ofDefault() ArmeriaScalazBackend.usingDefaultClient() ``` or, if you'd like to instantiate the [WebClient](https://armeria.dev/docs/client-http) yourself: ```scala import com.linecorp.armeria.client.circuitbreaker._ import com.linecorp.armeria.client.WebClient // Fluently build Armeria WebClient with built-in decorators val client = WebClient.builder("https://my-service.com") // Open circuit on 5xx server error status .decorator(CircuitBreakerClient.newDecorator(CircuitBreaker.ofDefaultName(), CircuitBreakerRule.onServerErrorStatus())) .build() val backend = ArmeriaScalazBackend.usingClient(client) ``` ```{note} A WebClient could fail to follow redirects if the WebClient is created with a base URI and a redirect location is a different URI. ``` This backend is build on top of [Armeria](https://armeria.dev/docs/client-http). Armeria's [ClientFactory](https://armeria.dev/docs/client-factory) manages connections and protocol-specific properties. Please visit [the official documentation](https://armeria.dev/docs/client-factory) to learn how to configure it. ## Streaming This backend doesn't support non-blocking [streaming](../requests/streaming.md). ## Websockets The backend doesn't support [websockets](../other/websockets.md). # Starting & cleaning up In case of most backends, you should only instantiate a backend once per application, as a backend typically allocates resources such as thread or connection pools. When ending the application, make sure to call `backend.close()`, which results in an effect which frees up resources used by the backend (if any). If the effect wrapper for the backend is lazily evaluated, make sure to include it when composing effects! Note that only resources allocated by the backends are freed. For example, if you use the `PekkoHttpBackend()` the `close()` method will terminate the underlying actor system. However, if you have provided an existing actor system upon backend creation (`PekkoHttpBackend.usingActorSystem`), the `close()` method will be a no-op. # Synchronous backends There are several synchronous backend implementations. Sending a request using these backends is a blocking operation, and results in a `sttp.client4.Response[T]`. ## Using HttpClient The default **synchronous** backend. To use, you don't need any extra dependencies, `core` is enough: ``` "com.softwaremill.sttp.client4" %% "core" % "4.0.25" ``` Create the backend using: ```scala import sttp.client4.httpclient.HttpClientSyncBackend val backend = HttpClientSyncBackend() ``` or, if you'd like to instantiate the HttpClient yourself: ```scala import sttp.client4.httpclient.HttpClientSyncBackend import java.net.http.HttpClient val httpClient: HttpClient = ??? val backend = HttpClientSyncBackend.usingClient(httpClient) ``` This backend is based on the built-in `java.net.http.HttpClient` available from Java 11 onwards. Host header override is supported in environments running Java 12 onwards, but it has to be enabled by system property: ``` -Djdk.httpclient.allowRestrictedHeaders=host ``` ## Using HttpURLConnection To use, you don't need any extra dependencies, `core` is enough: ``` "com.softwaremill.sttp.client4" %% "core" % "4.0.25" ``` Create the backend using: ```scala import sttp.client4.httpurlconnection.HttpURLConnectionBackend val backend = HttpURLConnectionBackend() ``` This backend supports host header override, but it has to be enabled by system property: ``` -Dsun.net.http.allowRestrictedHeaders=true ``` ## Using OkHttp To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "okhttp-backend" % "4.0.25" ``` Create the backend using: ```scala import sttp.client4.okhttp.OkHttpSyncBackend val backend = OkHttpSyncBackend() ``` or, if you'd like to instantiate the OkHttpClient yourself: ```scala import sttp.client4.okhttp.OkHttpSyncBackend import okhttp3.* val okHttpClient: OkHttpClient = ??? val backend = OkHttpSyncBackend.usingClient(okHttpClient) ``` This backend depends on [OkHttp](http://square.github.io/okhttp/) and fully supports HTTP/2. ## Streaming Synchronous backends don't support non-blocking [streaming](../requests/streaming.md). However, blocking, synchronous streaming is supported when sttp is used in combination with [Ox](https://ox.softwaremill.com): a Scala 3 toolkit that allows you to handle concurrency and resiliency in direct-style, leveraging Java 21+ virtual threads. For a start, in a virtual thread environment, often using `InputStreams` is enough. However, for more flexibility, these can be created or consumed using Ox's `Flow`s. See the [example](https://github.com/softwaremill/sttp/blob/master/examples/src/main/scala/sttp/client4/examples/streaming/streamOx.scala) where sending & receiving bodies describes as `Flow`s is demonstrated. ## Websockets Both HttpClient and OkHttp backends support regular [WebSockets](../other/websockets.md), with a blocking, synchronous streaming variant available through integration with Ox. See the linked WebSockets docs for details. ## Server-sent events If you're using Ox with `sttp`, you can handle SSE as a `Flow[ServerSentEvent]`: ``` // sbt dependency "com.softwaremill.sttp.client4" %% "ox" % "4.0.25", ``` ```scala import sttp.client4.* import sttp.client4.impl.ox.sse.OxServerSentEvents import java.io.InputStream def handleSse(is: InputStream): Unit = OxServerSentEvents.parse(is).runForeach(event => println(s"Received event: $event")) val backend = DefaultSyncBackend() basicRequest .get(uri"https://postman-echo.com/server-events/3") .response(asInputStreamAlways(handleSse)) .send(backend) ``` # Caching backend To use the caching backend, add the following dependency: ``` "com.softwaremill.sttp.client4" %% "caching-backend" % "4.0.25" ``` The backend caches responses to eligible requests, and returns them from the cache if a repeated request is made. A prerequisite for a request to be considered for caching is that its response-as description is "cache-friendly"; this excludes non-blocking streaming responses, file-based responses and WebSockets. An implementation of a `Cache` trait is required when creating the backend. The `Cache` allows storing cached values (with a TTL), and retrieving them. The cache is highly configurable, including: * determining if a request is eligible for caching (before it is sent) * computing the cache key * computing the caching duration (basing on the response) * serialization and deserialization of the response To use, wrap your backend (the below uses default configuration): ```scala import sttp.client4.caching.CachingBackend CachingBackend(delegateBackend, myCacheImplementation) ``` ## Default configuration Using `CachingConfig.Default`, caching happens if: * the request is a `GET` or `HEAD` request * the response contains a `Cache-Control` header with a `max-age` directive (standard HTTP semantics); the response is cached for the duration specified in this directive The cache key is created using the request method, URI, and the values of headers specified in the `Vary` header. For requests which might be cached, the response's body is read into a byte array. If the response is determined to be cached, it is serialized to JSON (using jsoniter-scala) and stored in the cache. See [examples](../../examples.md) for an example usage of the caching backend, using Redis. # Custom backends It is also entirely possible to write custom backends (if doing so, please consider contributing!) or wrap an existing one. One can even write completely generic wrappers for any delegate backend, as each backend comes equipped with a monad for the used effect type. This brings the possibility to `map` and `flatMap` over responses. Possible use-cases for wrapper-backend include: * logging * capturing metrics * request signing (transforming the request before sending it to the delegate) See also the section on [resilience](../../other/resilience.md) which covers topics such as retries, circuit breaking and rate limiting. ## Request attributes Each request contains a `attributes: AttributeMap` type-safe map. This map can be used to tag the request with any backend-specific information, and isn't used in any way by sttp itself. Attributes can be added to a request using the `def attribute[T](k: AttributeKey[T], v: T)` method, and read using the `def attribute[T](k: Attribute[T]): Option[T]` method. Backends, or backend wrappers can use attributes e.g. for logging, passing a metric name, using different connection pools, or even different delegate backends. ## Listener backend The `sttp.client4.listener.ListenerBackend` can make it easier to create backend wrappers which need to be notified about request lifecycle events: when a request is started, and when it completes either successfully or with an exception. This is possible by implementing a `sttp.client4.listener.RequestListener`. A request listener can associate a value with a request, which will then be passed to the request completion notification methods. A side-effecting request listener, of type `RequestListener[Identity, L]`, can be lifted to a request listener `RequestListener[F, L]` given a `MonadError[F]`, using the `RequestListener.lift` method. ## Backend wrappers and redirects See the appropriate section in docs on [redirects](../../conf/redirects.md). ## Logging backend wrapper A good example on how to implement a logging backend wrapper is the [logging](logging.md) backend wrapper implementation. It uses the `ListenerBackend` to get notified about request lifecycle events. To adjust the logs to your needs, or to integrate with your logging framework, simply copy the code and modify as needed. ## Examples backend wrappers A number of example backend wrappers can be found in [examples](../../examples.md). ## Example new backend Implementing a new backend is made easy as the tests are published in the `core` jar file under the `tests` classifier. Simply add the follow dependencies to your `build.sbt`: ``` "com.softwaremill.sttp.client4" %% "core" % "4.0.25" % Test classifier "tests" ``` Implement your backend and extend the `HttpTest` class: ```scala import sttp.client4.* import sttp.client4.testing.{ConvertToFuture, HttpTest} import scala.concurrent.Future class MyCustomBackendHttpTest extends HttpTest[Future]: override implicit val convertToFuture: ConvertToFuture[Future] = ConvertToFuture.future override val backend: Backend[Future] = ??? //new MyCustomBackend() override def timeoutToNone[T](t: Future[T], timeoutMillis: Int): Future[Option[T]] = ??? ``` ## Custom backend wrapper using cats When implementing a backend wrapper using cats, it might be useful to import: ```scala import sttp.client4.impl.cats.implicits.* ``` from the cats integration module. The module should be available on the classpath after adding following dependency: ```scala "com.softwaremill.sttp.client4" %% "cats" % "4.0.25" // for cats-effect 3.x // or "com.softwaremill.sttp.client4" %% "catsce2" % "4.0.25" // for cats-effect 2.x ``` The object contains implicits to convert a cats `MonadError` into the sttp `MonadError`, as well as a way to map the effects wrapper used with the `.mapK` extension method for the backend. # Logging The `sttp.client4.logging.LoggingBackend` can log requests and responses which end successfully or with an exception. It can be created given: * a `sttp.client4.logging.Logger`, which is an integration point with logging libraries. Two such integration are available with sttp-client: slf4j and scribe (see below), but custom ones can be easily added. * a `sttp.client4.logging.Log`, which constructs messages and performs logging actions. A custom implementation can be provided to change the message content or use dynamic log levels. By default, the following options are exposed: * `includeTimings` - should the duration of the request be included in the log message * `beforeCurlInsteadOfShow` - before sending a request, instead of a summary of the request to be sent, log the curl command which corresponds to the request * `logRequestBody` - should the request body be logged before sending the request (if the request body can be logged) * `logRequestHeaders` - should the non-sensitive request headers be logged before sending the request * `logResponseBody` - should the response body be logged after receiving a response to the request (if the response body can be replayed) * `logResponseHeaders` - should the non-sensitive response headers be logged The messages are by default logged on these levels: * `DEBUG` before the request is sent * `DEBUG` when a request completes successfully (with a 1xx/2xx status code) * `WARN` when a request completes successfully (with a 4xx/5xx status code) * `ERROR` when there's an exception when sending a request Log levels can be configured when creating the `LoggingBackend`, or specified independently in a custom implementation of `Log`. ## Using slf4j To use the [slf4j](http://www.slf4j.org) logging backend wrapper, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "slf4j-backend" % "4.0.25" ``` There are three backend wrappers available, which log request & response information using a slf4j `Logger`. To see the logs, you'll need to use an slf4j-compatible logger implementation, e.g. [logback](http://logback.qos.ch), or use a binding, e.g. [log4j-slf4j](https://logging.apache.org/log4j/2.x/log4j-slf4j-impl.html). Example usage: ```scala import sttp.client4.* import sttp.client4.logging.slf4j.Slf4jLoggingBackend val backend = Slf4jLoggingBackend(DefaultSyncBackend()) basicRequest.get(uri"https://httpbin.org/get").send(backend) // Logs: // 21:14:23.735 [main] INFO sttp.client4.logging.slf4j.Slf4jTimingBackend - Request: GET https://httpbin.org/get, took: 0.795s, response: 200 ``` To create a customized logging backend, see the section on [custom backends](custom.md). ## Using scribe To use the [scribe](https://github.com/outr/scribe) logging backend wrapper, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "scribe-backend" % "4.0.25" ``` # OpenTelemetry Currently, the following OpenTelemetry features are supported: - metrics using `OpenTelemetryMetricsBackend`, wrapping any other backend - tracing using `OpenTelemetryTracingBackend`, wrapping a synchronous backend - tracing using `OpenTelemetryTracingZioBackend`, wrapping any ZIO2 backend - tracing using [trace4cats](https://github.com/trace4cats/trace4cats), wrapping a cats-effect backend ## Metrics The backend depends only on [opentelemetry-api](https://github.com/open-telemetry/opentelemetry-java). To use add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "opentelemetry-backend" % "4.0.25" ``` Then an instance can be obtained as follows: ```scala import scala.concurrent.Future import sttp.client4.* import sttp.client4.opentelemetry.* import io.opentelemetry.api.OpenTelemetry // any effect and capabilities are supported val sttpBackend: Backend[Future] = ??? val openTelemetry: OpenTelemetry = ??? OpenTelemetryMetricsBackend(sttpBackend, openTelemetry) ``` All counters have provided default names, but the names can be customized by setting correct parameters in constructor: ```scala import scala.concurrent.Future import sttp.client4.* import sttp.client4.opentelemetry.* import io.opentelemetry.api.OpenTelemetry val sttpBackend: Backend[Future] = ??? val openTelemetry: OpenTelemetry = ??? OpenTelemetryMetricsBackend( sttpBackend, OpenTelemetryMetricsConfig( openTelemetry, responseToSuccessCounterMapper = (_, _) => Some(CollectorConfig("my_custom_counter_name")) ) ) ``` ## Tracing To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "opentelemetry-backend" % "4.0.25" ``` The backend records traces corresponding to HTTP client calls. The default span name is the HTTP method (e.g. `POST`), but this can be customized to provide more accurate (but still general) span names by providing a custom span-name-generating method (as [recommended by OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/http/http-spans/#name)). Alternative span naming strategies might include reading request's attributes (to determine the target URI template), or parts of the URI. Other aspects of the backend can be configured as well: * the `Tracer` instance and context propagators * how request, response, error attributes are computed ```{note} Relies on the built-in OpenTelemetry Java SDK `ContextStorage` mechanism of propagating the tracing context; by default, this is using `ThreadLocal`s, which works with synchronous/direct-style environments. `Future`s are supported through instrumentation provided by the OpenTelemetry javaagent. For functional effect systems, usually a dedicated integration library is required. ``` Example usage: ```scala import sttp.client4.* import sttp.client4.opentelemetry.* import io.opentelemetry.api.OpenTelemetry val sttpBackend: SyncBackend = ??? val openTelemetry: OpenTelemetry = ??? OpenTelemetryTracingBackend( sttpBackend, OpenTelemetryTracingConfig( openTelemetry, spanName = request => request.uri.pathSegments.segments.headOption.map(_.v).getOrElse("root") ) ) ``` ## Tracing (ZIO) To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "opentelemetry-tracing-zio-backend" % "4.0.25" // for ZIO 2.x ``` This backend depends on [zio-opentelemetry](https://github.com/zio/zio-telemetry). The OpenTelemetry backend wraps a `Task` based ZIO backend. In order to do that, you need to provide the wrapper with a `Tracing` from zio-telemetry. Here's how you construct `ZioTelemetryOpenTelemetryBackend`: ```scala import sttp.client4.* import zio.* import zio.telemetry.opentelemetry.tracing.* import sttp.client4.opentelemetry.zio.* val zioBackend: Backend[Task] = ??? val tracing: Tracing = ??? OpenTelemetryTracingZioBackend(zioBackend, tracing) ``` By default, the span is named after the HTTP method (e.g `POST`) as [recommended by OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#http-client) for HTTP clients, and the http method, url and response status codes are set as span attributes. You can override these defaults by supplying a custom `OpenTelemetryZioTracer`. ## Metrics (cats-effect, otel4s) Add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "opentelemetry-otel4s-metrics-backend" % "4.0.25" ``` This backend depends on [otel4s](https://github.com/typelevel/otel4s). Use `Otel4sMetricsBackend` to enable tracing of a client: ```scala import cats.effect.* import org.typelevel.otel4s.metrics.MeterProvider import sttp.client4.* import sttp.client4.opentelemetry.otel4s.* implicit val meterProvider: MeterProvider[IO] = ??? val catsBackend: Backend[IO] = ??? Otel4sMetricsBackend(catsBackend, Otel4sMetricsConfig.default) .use { backend => ??? } ``` The backend follows the OpenTelemetry [specification](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/) of HTTP metrics. The following metrics are available by default: - [http.client.request.duration](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpclientrequestduration) - [http.client.request.body.size](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpclientrequestbodysize) - [http.client.response.body.size](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpclientresponsebodysize) - [http.client.active_requests](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpclientactive_requests) You can customize histogram buckets and URL template behavior by providing a custom `Otel4sMetricsConfig`. ### URL template The `url.template` [experimental attribute](https://opentelemetry.io/docs/specs/semconv/attributes-registry/url/) is not added by default, as URL structures vary widely across APIs. To enable it, provide a `GenericRequest[_, _] => Option[String]` function via the `urlTemplate` config field. Because the function receives the full request, you can use request attributes to pass the template from the call site. A built-in implementation is available in `UrlTemplates.replaceIds`: it replaces UUIDs and numeric IDs in path segments and query values with `{id}`, and always returns `Some` (the URL unchanged when no IDs are found). ```scala import cats.effect.* import org.typelevel.otel4s.metrics.MeterProvider import sttp.client4.* import sttp.client4.opentelemetry.otel4s.* implicit val meterProvider: MeterProvider[IO] = ??? val catsBackend: Backend[IO] = ??? // Use the built-in implementation (replaces UUIDs and numeric IDs with {id}): Otel4sMetricsBackend( catsBackend, Otel4sMetricsConfig( requestDurationHistogramBuckets = Otel4sMetricsConfig.DefaultDurationBuckets, requestBodySizeHistogramBuckets = None, responseBodySizeHistogramBuckets = None, urlTemplate = UrlTemplates.replaceIds ) ) // Or provide a custom function based on request attributes: import sttp.attributes.AttributeKey val UrlTemplateKey = new AttributeKey[String]("UrlTemplateKey") Otel4sMetricsBackend( catsBackend, Otel4sMetricsConfig( requestDurationHistogramBuckets = Otel4sMetricsConfig.DefaultDurationBuckets, requestBodySizeHistogramBuckets = None, responseBodySizeHistogramBuckets = None, urlTemplate = req => req.attribute(UrlTemplateKey) ) ) // Then, at the call site: // basicRequest.get(uri"...").attribute(UrlTemplateKey, "/users/{id}") ``` ## Tracing (cats-effect, otel4s) Add the following dependency to your project: ```scala "com.softwaremill.sttp.client4" %% "opentelemetry-otel4s-tracing-backend" % "4.0.25" ``` This backend depends on [otel4s](https://github.com/typelevel/otel4s). Use `Otel4sTracingBackend` to enable tracing of a client: ```scala import cats.effect.* import org.typelevel.otel4s.trace.TracerProvider import sttp.client4.* import sttp.client4.opentelemetry.otel4s.* implicit val tracerProvider: TracerProvider[IO] = ??? val catsBackend: Backend[IO] = ??? Otel4sTracingBackend(catsBackend, Otel4sTracingConfig.default) ``` The backend follows the OpenTelemetry [specification](https://opentelemetry.io/docs/specs/semconv/http/http-spans/) of HTTP spans. You can customize span name and attached attributes by providing a custom `Otel4sTracingConfig`. ## Tracing (cats-effect, trace4cats) The [trace4cats](https://github.com/trace4cats/trace4cats) project includes sttp-client integration. # Prometheus backend To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "prometheus-backend" % "4.0.25" ``` and some imports: ```scala import sttp.client4.prometheus.* ``` This backend depends on [Prometheus JVM Client](https://github.com/prometheus/client_java). Keep in mind this backend registers histograms and gathers request times, but you have to expose those metrics to [Prometheus](https://prometheus.io/). The Prometheus backend wraps any other backend, for example: ```scala import sttp.client4.pekkohttp.* val backend = PrometheusBackend(PekkoHttpBackend()) ``` It gathers request execution times in `Histogram`. It uses by default `http_client_request_duration_seconds` name, defined in `PrometheusBackend.DefaultHistogramName`. It is possible to define custom histograms name by passing function mapping request to histogram name: ```scala import sttp.client4.pekkohttp.* val backend = PrometheusBackend( PekkoHttpBackend(), PrometheusConfig( requestToHistogramNameMapper = request => Some(HistogramCollectorConfig(request.uri.host.getOrElse("example.com"))) ) ) ``` You can disable request histograms by passing `None` returning function: ```scala import sttp.client4.pekkohttp.* val backend = PrometheusBackend(PekkoHttpBackend(), PrometheusConfig(requestToHistogramNameMapper = _ => None)) ``` This backend also offers `Gauge` with currently in-progress requests number. It uses by default `http_client_requests_active` name, defined in `PrometheusBackend.DefaultRequestsActiveCounterName`. It is possible to define custom gauge name by passing function mapping request to gauge name: ```scala import sttp.client4.pekkohttp.* val backend = PrometheusBackend( PekkoHttpBackend(), PrometheusConfig( requestToInProgressGaugeNameMapper = request => Some(CollectorConfig(request.uri.host.getOrElse("example.com"))) ) ) ``` You can disable request in-progress gauges by passing `None` returning function: ```scala import sttp.client4.pekkohttp.* val backend = PrometheusBackend(PekkoHttpBackend(), PrometheusConfig(requestToInProgressGaugeNameMapper = _ => None)) ``` # ZIO backends The [ZIO](https://github.com/zio/zio) backends are **asynchronous**. Sending a request is a non-blocking, lazily-evaluated operation and results in a response wrapped in a `zio.Task`. There's a transitive dependency on the `zio` or `zio1` modules. The `*-zio` modules depend on ZIO 2.x. For ZIO 1.x support, use modules with the `*-zio1` suffix. ## Using HttpClient To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "zio" % "4.0.25" // for ZIO 2.x "com.softwaremill.sttp.client4" %% "zio1" % "4.0.25" // for ZIO 1.x ``` Create the backend using: ```scala import sttp.client4.httpclient.zio.HttpClientZioBackend HttpClientZioBackend().flatMap { backend => ??? } // or, if you'd like the backend to be created in a Scope: HttpClientZioBackend.scoped().flatMap { backend => ??? } // or, if you'd like to instantiate the HttpClient yourself: import java.net.http.HttpClient val httpClient: HttpClient = ??? val backend = HttpClientZioBackend.usingClient(httpClient) // or, obtain a Scope with a custom instance of the HttpClient: HttpClientZioBackend.scopedUsingClient(httpClient).flatMap { backend => ??? } ``` This backend is based on the built-in `java.net.http.HttpClient` available from Java 11 onwards. The backend is fully non-blocking, with back-pressured websockets. Host header override is supported in environments running Java 12 onwards, but it has to be enabled by system property: ``` -Djdk.httpclient.allowRestrictedHeaders=host ``` ## Using Armeria To use, add the following dependency to your project: ``` "com.softwaremill.sttp.client4" %% "armeria-backend-zio" % "4.0.25" // for ZIO 2.x "com.softwaremill.sttp.client4" %% "armeria-backend-zio1" % "4.0.25" // for ZIO 1.x ``` add imports: ```scala import sttp.client4.armeria.zio.ArmeriaZioBackend ``` create client: ```scala ArmeriaZioBackend().flatMap { backend => ??? } // or, if you'd like the backend to be wrapped in a Scope: ArmeriaZioBackend.scoped().flatMap { backend => ??? } // You can use the default client which reuses the connection pool of ClientFactory.ofDefault() ArmeriaZioBackend.usingDefaultClient().flatMap { backend => ??? } ``` ```{note} The default client factory is reused to create `ArmeriaZioBackend` if a `SttpBackendOptions` is unspecified. So you only need to manage a resource when `SttpBackendOptions` is used. ``` or, if you'd like to instantiate the [WebClient](https://armeria.dev/docs/client-http) yourself: ```scala import com.linecorp.armeria.client.circuitbreaker.* import com.linecorp.armeria.client.WebClient // Fluently build Armeria WebClient with built-in decorators val client = WebClient.builder("https://my-service.com") // Open circuit on 5xx server error status .decorator(CircuitBreakerClient.newDecorator(CircuitBreaker.ofDefaultName(), CircuitBreakerRule.onServerErrorStatus())) .build() ArmeriaZioBackend.usingClient(client).flatMap { backend => ??? } ``` ```{note} A WebClient could fail to follow redirects if the WebClient is created with a base URI and a redirect location is a different URI. ``` This backend is build on top of [Armeria](https://armeria.dev/docs/client-http). Armeria's [ClientFactory](https://armeria.dev/docs/client-factory) manages connections and protocol-specific properties. Please visit [the official documentation](https://armeria.dev/docs/client-factory) to learn how to configure it. ## Using JavaScript The ZIO backend is also available for the JS platform, see [the `FetchBackend` documentation](javascript/fetch.md). The `FetchBackend` companion object contains methods to create the backend directly, as a layer or scoped. ## Using Scala Native The ZIO backend is also available for the native platform backed by curl, see [the `CurlZioBackend` documentation](native/curl.md). The `CurlZioBackend` companion object contains methods to create the backend directly, as a layer or scoped. ## ZIO layers + constructors When using constructors to express service dependencies, ZIO layers can be used to provide the `SttpBackend` instance, instead of creating one by hand. In this scenario, the lifecycle of a `SttpBackend` service is described by `ZLayer`s, which can be created using the `.layer`/`.layerUsingConfig`/... methods on `HttpClientZioBackend` / `ArmeriaZioBackend`. The layers can be used to provide an implementation of the `SttpBackend` dependency when creating services. For example: ```scala import sttp.client4.* import sttp.client4.httpclient.zio.* import zio.* class MyService(sttpBackend: Backend[Task]) { def runLogic(): Task[Response[String]] = { val request = basicRequest.response(asStringAlways).get(uri"https://httpbin.org/get") request.send(sttpBackend) } } object MyService { val live: ZLayer[Backend[Task], Any, MyService] = ZLayer.fromFunction(new MyService(_)) } ZLayer.make[MyService](MyService.live, HttpClientZioBackend.layer()) ``` ## ZIO environment As yet another alternative to effectfully or resourcefully creating backend instances, ZIO environment can be used. There are top-level `send` and `sendR` top-level methods which require a `SttpClient` to be available in the environment. The `SttpClient` itself is a type alias: ```scala package sttp.client4.httpclient.zio type SttpClient = SttpBackend[Task, ZioStreams with WebSockets] // or, when using Armeria package sttp.client4.armeria.zio type SttpClient = SttpBackend[Task, ZioStreams] ``` The lifecycle of the `SttpClient` service is described by `ZLayer`s, which can be created using the `.layer`/`.layerUsingConfig`/... methods on `HttpClientZioBackend` / `ArmeriaZioBackend`. The `SttpClient` companion object contains effect descriptions which use the `SttpClient` service from the environment to send requests or open websockets. This is different from sttp usage with other effect libraries (which require invoking `.send(backend)` on the request), but is more in line with one of the styles of using ZIO. For example: ```scala mdoc:compile-only import sttp.client4.* import sttp.client4.httpclient.zio.* import zio.* val request = basicRequest.get(uri"https://httpbin.org/get") val sent: ZIO[SttpClient, Throwable, Response[Either[String, String]]] = send(request) ``` ## Streaming The ZIO based backends support streaming using zio-streams. The following example is using the `HttpClientZioBackend`. The type of supported streams is `Stream[Throwable, Byte]`. The streams capability is represented as `sttp.client4.impl.zio.ZioStreams`. To leverage ZIO environment, use the `SttpClient` object to create request send effects. Requests can be sent with a streaming body: ```scala import sttp.capabilities.zio.ZioStreams import sttp.client4.* import zio.stream.* import zio.Task val sttpBackend: StreamBackend[Task, ZioStreams] = ??? val s: Stream[Throwable, Byte] = ??? val request = basicRequest .post(uri"...") .streamBody(ZioStreams)(s) request.send(sttpBackend) ``` And receive response bodies as a stream: ```scala import sttp.capabilities.zio.ZioStreams import sttp.client4.* import zio.* import zio.stream.* import scala.concurrent.duration.Duration val sttpBackend: StreamBackend[Task, ZioStreams] = ??? val request = basicRequest .post(uri"...") .response(asStreamUnsafe(ZioStreams)) .readTimeout(Duration.Inf) val response: ZIO[Any, Throwable, Response[Either[String, Stream[Throwable, Byte]]]] = request.send(sttpBackend) ``` ## Websockets The `HttpClient` ZIO backend supports both regular and streaming [websockets](../other/websockets.md). ## Testing A stub backend can be created through the `.stub` method on the companion object, and configured as described in the [testing](../testing/stub.md) section. A layer with the stub `Backend` can be then created by simply calling `ZLayer.succeed(backendStub)`. ## Server-sent events Received data streams can be parsed to a stream of server-sent events (SSE): ```scala import zio.* import zio.stream.* import sttp.capabilities.zio.ZioStreams import sttp.client4.impl.zio.ZioServerSentEvents import sttp.model.sse.ServerSentEvent import sttp.client4.* def processEvents(source: Stream[Throwable, ServerSentEvent]): Task[Unit] = ??? basicRequest .get(uri"...") .response(asStream(ZioStreams)(stream => processEvents(stream.viaFunction(ZioServerSentEvents.parse)))) ``` # The stub backend If you need a stub backend for use in tests instead of a "real" backend (you probably don't want to make HTTP calls during unit tests), you can use the `BackendStub` class. It allows specifying how the backend should respond to requests matching given predicates. The [pekko-http](../backends/pekko.md) or [akka-http](../backends/akka.md) backends also provide an alternative way to create a stub, from a request-transforming function. ## Creating a stub backend An empty backend stub can be created using the following ways: * by calling `.stub` on the "real" base backend's companion object, e.g. `DefaultSyncBackend.stub`, `HttpClientZioBackend.stub`, `HttpClientMonixBackend.stub` * by using one of the factory methods `BackendStub.synchronous` or `BackendStub.asynchronousFuture`, which return stubs which use the `Identity` or standard Scala's `Future` effects without streaming support * by explicitly specifying the effect and supported capabilities: * for cats-effect `BackendStub[IO](implicitly[MonadAsyncError[IO]])` * for ZIO `BackendStub[Task](new RIOMonadAsyncError[Any])` * for Monix `BackendStub[Task](TaskMonad)` * by instantiating backend stubs which support streaming or WebSockets, mirroring the hierarchy of the base backends: * `StreamBackendStub`, e.g. `StreamBackendStub[IO, Fs2Streams[IO]](implicitly[MonadAsyncError[IO]])` (for cats-effect with fs2) * `WebSocketBackendStub`, e.g. `WebSocketBackendStub[Task](new RIOMonadAsyncError[Any])` (for ZIO) * `WebSocketStreamBackendStub` * `WebSocketSyncBackendStub` * by specifying a fallback/delegate backend, see below Responses used in the stubbing are most conveniently created using factory methods from `ResponseStub`. Some code which will be reused among following examples: ```scala import sttp.client4.* import sttp.model.* import sttp.client4.testing.* import java.io.File import scala.concurrent.Future import scala.concurrent.ExecutionContext.Implicits.global case class User(id: String) ``` ## Specifying behavior Behavior of the stub can be specified using a series of invocations of the `whenRequestMatches` and `thenRespond...` methods: ```scala val testingBackend = SyncBackendStub .whenRequestMatches(_.uri.path.startsWith(List("a", "b"))) .thenRespondAdjust("Hello there!") .whenRequestMatches(_.method == Method.POST) .thenRespondServerError() val response1 = basicRequest.get(uri"http://example.org/a/b/c").send(testingBackend) // response1.body will be Right("Hello there") val response2 = basicRequest.post(uri"http://example.org/d/e").send(testingBackend) // response2.code will be 500 ``` It is also possible to match requests by partial function, returning a response. E.g.: ```scala val testingBackend = SyncBackendStub .whenRequestMatchesPartial({ case r if r.uri.path.endsWith(List("partial10")) => ResponseStub.adjust("Not found", StatusCode.NotFound) case r if r.uri.path.endsWith(List("partialAda")) => // additional verification of the request is possible assert(r.body == StringBody("z", "utf-8")) ResponseStub.adjust("Ada") }) val response1 = basicRequest.get(uri"http://example.org/partial10").send(testingBackend) // response1.body will be Right(10) val response2 = basicRequest.post(uri"http://example.org/partialAda").send(testingBackend) // response2.body will be Right("Ada") ``` ```{note} This approach to testing has one caveat: the responses are not type-safe. That is, the stub backend cannot match on or verify that the type of the response body matches the response body type, as it was requested. However, the response bodies can be adjusted, and attempted to be handled as specified by the response description. Hence, you can provide bodies as a `String`, `Array[Byte]`, `InputStream`, `File`, `WebSocket`, `WebSocketStub`, or a non-blocking binary stream, and the conversions to the desired type will happen as part of the test. ``` Another way to specify the behavior is passing response wrapped in the effect to the stub. It is useful if you need to test a scenario with a slow server, when the response should be not returned immediately, but after some time. Example with Futures: ```scala val testingBackend = BackendStub.asynchronousFuture .whenAnyRequest .thenRespondF(Future { Thread.sleep(5000) ResponseStub.adjust("OK") }) val responseFuture = basicRequest.get(uri"http://example.org").send(testingBackend) // responseFuture will complete after 5 seconds with "OK" response ``` The returned response may also depend on the request: ```scala val testingBackend = SyncBackendStub .whenAnyRequest .thenRespondF(req => ResponseStub.adjust(s"OK, got request sent to ${req.uri.host}") ) val response = basicRequest.get(uri"http://example.org").send(testingBackend) // response.body will be Right("OK, got request sent to example.org") ``` You can define consecutive raw responses that will be served: ```scala val testingBackend: SyncBackendStub = SyncBackendStub .whenAnyRequest .thenRespondCyclic( ResponseStub.adjust("first"), ResponseStub.adjust("error", StatusCode.InternalServerError) ) basicRequest.get(uri"http://example.org").send(testingBackend) // code will be 200 basicRequest.get(uri"http://example.org").send(testingBackend) // code will be 500 basicRequest.get(uri"http://example.org").send(testingBackend) // code will be 200 ``` The `sttp.client4.testing` package also contains a utility method to force the body as a string (`forceBodyAsString`) or as a byte array (`forceBodyAsByteArray`), if the body is not a stream or multipart: ```scala val testingBackend = SyncBackendStub .whenRequestMatches(_.forceBodyAsString.contains("Hello, world!")) .thenRespondAdjust("Hello back!") ``` If the stub is given a request, for which no behavior is stubbed, it will return a failed effect with an `IllegalArgumentException`. ## Simulating exceptions If you want to simulate an exception being thrown by a backend, e.g. a socket timeout exception, you can do so by using the `thenThrow` method: ```scala val testingBackend = SyncBackendStub .whenRequestMatches(_ => true) .thenThrow(new SttpClientException.ConnectException( basicRequest.get(uri"http://example.com"), new RuntimeException)) ``` ## Adjusting the response body type When using `.thenRespondAdjust` or `ResponseStub.adjust` methods, the stub will attempt to convert the body returned by the stub to the desired type. If the given body isn't in one of the supported "raw" types, an `IllegalArgumentException` will be thrown. This is to: * test code which maps a basic response body to a custom type, e.g. mapping a raw json string using a decoder to a domain type * reading a classpath resource (which results in an `InputStream`) and requesting a response of e.g. type `String` * using resource-safe response specifications for streaming and websockets The following conversions are supported: * anything to `()` (unit), when the response is ignored * `InputStream` and `Array[Byte]` to `String` * `InputStream` and `String` to `Array[Byte]` * `WebSocketStub` to `WebSocket` * `WebSocketStub` and `WebSocket` are supplied to the websocket-consuming functions, if the response specification describes such interactions * for non-blocking, asynchronous streaming responses, any provided value is treated as a raw stream value; the value should be of type `sttp.capabilities.Streams.BinaryStream`, and if that's not the case, a `ClassCastException` might be thrown * any of the above to custom types through mapped response specifications ## Example: returning JSON For example, if you want to return a JSON response, simply use `.withResponse(String)` as below: ```scala val testingBackend = SyncBackendStub .whenRequestMatches(_ => true) .thenRespondAdjust(""" {"username": "john", "age": 65 } """) def parseUserJson(a: Array[Byte]): User = ??? val response = basicRequest.get(uri"http://example.com") .response(asByteArrayAlways.map(parseUserJson)) .send(testingBackend) ``` In the example above, the stub's rules specify that a response with a `String`-body should be returned for any request; the request, on the other hand, specifies that response body should be parsed from a byte array to a custom `User` type. These type don't match, so the `BackendStub` will in this case convert the body to the desired type. ## Example: returning a file If you want to save the response to a file and have the response handler set up like this: ```scala val destination = new File("path/to/file.ext") basicRequest.get(uri"http://example.com").response(asFile(destination)) ``` With the stub created as follows: ```scala val fileResponseHandle = new File("path/to/file.ext") SyncBackendStub .whenRequestMatches(_ => true) .thenRespondAdjust(fileResponseHandle) ``` the `File` set up in the stub will be returned as though it was the `File` set up as `destination` in the response handler above. This means that the file from `fileResponseHandle` is not written to `destination`. If you actually want a file to be written you can set up the stub like this: ```scala import org.apache.commons.io.FileUtils import cats.effect.* import sttp.client4.impl.cats.implicits.* import sttp.monad.MonadAsyncError val sourceFile = new File("path/to/file.ext") val destinationFile = new File("path/to/file.ext") BackendStub(implicitly[MonadAsyncError[IO]]) .whenRequestMatches(_ => true) .thenRespondF: _ => FileUtils.copyFile(sourceFile, destinationFile) IO(ResponseStub.adjust(destinationFile, StatusCode.Ok)) ``` ## Responding with bodies as-is Alternatively, response bodies can be provided as-is, without any adjustment attempts, regardless of the request's response description. To do that, use `.thenRespondExact` or `ResponseStub.exact`. If the desired response's type is not the same as the provided one, a `ClassCastException` will be thrown. For example: ```scala case class User(name: String) val someUser: User = ??? // coming from a JSON integration def asJson[T]: ResponseAs[Either[String, T]] = ??? val testingBackend = SyncBackendStub .whenRequestMatches(_ => true) .thenRespondExact(Right(someUser)) val response = basicRequest.get(uri"http://example.com") .response(asJson[User]) .send(testingBackend) ``` ## Delegating to another backend It is also possible to create a stub backend which delegates calls to another (possibly "real") backend if none of the specified predicates match a request. This can be useful during development, to partially stub a yet incomplete API with which we integrate: ```scala val testingBackend = SyncBackendStub.withFallback(DefaultSyncBackend()) .whenRequestMatches(_.uri.path.startsWith(List("a"))) .thenRespondAdjust("I'm a STUB!") val response1 = basicRequest.get(uri"http://api.internal/a").send(testingBackend) // response1.body will be Right("I'm a STUB") val response2 = basicRequest.post(uri"http://api.internal/b").send(testingBackend) // response2 will be whatever a "real" network call to api.internal/b returns ``` ## Testing streams Streaming responses can be stubbed the same as ordinary values. The body of the response should contain the raw byte stream value, of type `sttp.capabilities.Streams.BinaryStream`. ## Testing web sockets Like streams, web sockets can be stubbed as ordinary values, by providing `WebSocket` or `WebSocketStub` instances. If the response specification is a resource-safe consumer of the web socket, the function will be invoked if the provided stubbed body is a `WebSocket` or `WebSocketStub`. The stub can be configured to return the high-level (already mapped/transformed) response body. ### WebSocketStub `WebSocketStub` allows easy creation of stub `WebSocket` instances. Such instances wrap a state machine that can be used to simulate simple WebSocket interactions. The user sets initial responses for `receive` calls as well as logic to add further messages in reaction to `send` calls. For example: ```scala import sttp.ws.testing.WebSocketStub import sttp.ws.WebSocketFrame val backend = WebSocketBackendStub.synchronous val webSocketStub = WebSocketStub .initialReceive( List(WebSocketFrame.text("Hello from the server!")) ) .thenRespondS(0): case (counter, tf: WebSocketFrame.Text) => (counter + 1, List(WebSocketFrame.text(s"echo: ${tf.payload}"))) case (counter, _) => (counter, List.empty) backend.whenAnyRequest.thenRespondAdjust(webSocketStub) ``` There is a possibility to add error responses as well. If this is not enough, using a custom implementation of the `WebSocket` trait is recommended. ### WebSocket streams When using the `asWebSocketStream` response description, you can provide the behavior to be run when this request is sent, using `WebSocketStreamConsumer` as the body. For example: ```scala import cats.effect.IO import sttp.capabilities.fs2.Fs2Streams import sttp.client4.WebSocketStreamBackend import sttp.client4.httpclient.fs2.HttpClientFs2Backend import sttp.client4.testing.WebSocketStreamConsumer import sttp.client4.ws.stream._ val backend: WebSocketStreamBackend[IO, Fs2Streams[IO]] = HttpClientFs2Backend .stub[IO] .whenAnyRequest .thenRespondAdjust( WebSocketStreamConsumer[IO](Fs2Streams[IO]) { pipe => // somehow consume the pipe: fs2.Pipe[IO, WebSocketFrame.Data, WebSocketFrame] to produce an IO[Unit] ??? }) basicRequest .get(uri"http://example.org") .response(asWebSocketStream(Fs2Streams[IO]) { // the client-side behavior, a fs2.Pipe[IO, WebSocketFrame.Data, WebSocketFrame] ??? }) .send(backend) ``` ## Verifying that a request was sent Using `RecordingSttpBackend` it's possible to capture all interactions in which a backend has been involved. The recording backend is a [backend wrapper](../backends/wrappers/custom.md), and it can wrap any backend, but it's most useful when combined with the backend stub. Example usage: ```scala import scala.util.Try val testingBackend = RecordingBackend( SyncBackendStub .whenRequestMatches(_.uri.path.startsWith(List("a", "b"))) .thenRespondAdjust("Hello there!") ) val response1 = basicRequest.get(uri"http://example.org/a/b/c").send(testingBackend) // response1.body will be Right("Hello there") testingBackend.allInteractions: List[(GenericRequest[_, _], Try[Response[_]])] // the list will contain one element and can be verified in a test ``` # Converting requests to CURL commands sttp comes with builtin request to curl converter. To convert request to curl invocation use `.toCurl` method. For example: ```scala import sttp.client4.* basicRequest.get(uri"http://httpbin.org/ip").toCurl // res0: String = """curl \ // --request GET \ // --url 'http://httpbin.org/ip' \ // --header 'Accept-Encoding: gzip, deflate' \ // --location \ // --max-redirs 32""" ``` Note that the `Accept-Encoding` header, which is added by default to all requests (`Accept-Encoding: gzip, deflate`), can make curl warn that _binary output can mess up your terminal_, when running generated command from the command line. It can be omitted by setting `omitAcceptEncoding = true` when calling `.toCurl` method. # Timeouts sttp supports read and connection timeouts: * Connection timeout - can be set globally (30 seconds by default) * Read timeout - can be set per request (1 minute by default) How to use: ```scala import sttp.client4.* import scala.concurrent.duration.* // all backends provide a constructor that allows to specify backend options val backend = DefaultSyncBackend( options = BackendOptions.connectionTimeout(1.minute)) basicRequest .get(uri"...") .readTimeout(5.minutes) // or Duration.Inf to turn read timeout off .send(backend) ``` # Proxy support sttp library by default checks for your System proxy properties ([docs](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html)): Following settings are checked: 1. `socksProxyHost` and `socksProxyPort` *(default: 1080)* 2. `http.proxyHost` and `http.proxyPort` *(default: 80)* 3. `https.proxyHost` and `https.proxyPort` *(default: 443)* Settings are loaded **in given order** and the **first existing value** is being used. Otherwise, proxy values can be specified manually when creating a backend: ```scala import sttp.client4.* val backend = DefaultSyncBackend( options = BackendOptions.httpProxy("some.host", 8080)) basicRequest .get(uri"...") .send(backend) // uses the proxy ``` Or in case your proxy requires authentication (supported by the JVM backends): ```scala import sttp.client4.* BackendOptions.httpProxy("some.host", 8080, "username", "password") ``` if you use a HttpClient-based backend (e.g. `HttpClientBackend`) and your proxy server requires Basic authentication, you will need to enable it by removing Basic from the `jdk.http.auth.tunneling.disabledSchemes` networking property, or by setting a system property of the same name to "". ```scala System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "") ``` Please be aware that enabling Basic authentication for HTTP tunneling can expose your credentials to interception, so it should only be done if you understand the risks and your network is secure. This behaviour is described in https://www.oracle.com/java/technologies/javase/8u111-relnotes.html. ## Ignoring and allowing specific hosts There are two additional settings that can be provided to via `BackendOptions`: * `nonProxyHosts`: used to define hosts for which request SHOULD NOT be proxied * `onlyProxyHosts`: used to define hosts for which request SHOULD be proxied If only `nonProxyHosts` is provided, then some hosts will be skipped when proxying. If only `onlyProxyHosts` is provided, then requests will be proxied only if host matches provided list. If both `nonProxyHosts` and `onlyProxyHosts` are provided, then `nonProxyHosts` takes precedence. Both of these options are `Nil` by default. ### Wildcards It is possible to use wildcard, but only as either prefix or suffix. E.g. `onlyProxyHosts = List("localhost", "*.local", "127.*")` # Redirects By default, sttp follows redirects. If you'd like to disable following redirects, use the `followRedirects` method: ```scala import sttp.client4.* basicRequest.followRedirects(false) ``` If a request has been redirected, the history of all followed redirects is accessible through the `response.history` list. The first response (oldest) comes first. The body of each response will be a `Left(message)` (as the status code is non-2xx), where the message is whatever the server returned as the response body. ## Redirecting POST requests If a `POST` or `PUT` request is redirected, by default it will be sent unchanged to the new address, that is using the original body and method. However, most browsers and some clients issue a `GET` request in such case, without the body. To enable this behavior, use the `redirectToGet` method: ```scala import sttp.client4.* basicRequest.redirectToGet(true) ``` Note that this only affects `301 Moved Permanently` and `302 Found` redirects. `303 See Other` redirects are always converted, while `307 Temporary Redirect` and `308 Permanent Redirect` never. ## Important Note on the `Authorization` header Most modern http clients will, by default, strip the `Authorization` header when encountering a redirect; sttp client is no different. You can disable the stripping of all sensitive headers using the following code: ```scala import sttp.client4.* import sttp.client4.wrappers.{FollowRedirectsBackend, FollowRedirectsConfig} val myBackend: SyncBackend = DefaultSyncBackend() val backend: SyncBackend = FollowRedirectsBackend( delegate = myBackend, FollowRedirectsConfig( sensitiveHeaders = Set.empty ) ) ``` If you just want to disable stripping of the `Authorization` header, you can do the following: ```scala import sttp.client4.* import sttp.model.* import sttp.client4.wrappers.{FollowRedirectsBackend, FollowRedirectsConfig} val myBackend: SyncBackend = DefaultSyncBackend() val backend: SyncBackend = FollowRedirectsBackend( delegate = myBackend, FollowRedirectsConfig( sensitiveHeaders = HeaderNames.SensitiveHeaders.filterNot(_ == HeaderNames.Authorization.toLowerCase) ) ) ``` ## Backend wrappers and redirects By default redirects are handled at a low level, using a wrapper around the main, concrete backend: each of the backend factory methods, e.g. `HttpClientSyncBackend()` returns a backend wrapped in `FollowRedirectsBackend`. This causes any further backend wrappers to handle a request which involves redirects as one whole, without the intermediate requests. However, wrappers which collects metrics, implements tracing or handles request retries might want to handle every request in the redirect chain. This can be achieved by layering another `FollowRedirectsBackend` on top of the wrapper. Only the top-level follow redirects backend will handle redirects, other follow redirect wrappers (at lower levels) will be disabled. For example: ```scala import sttp.capabilities.Effect import sttp.client4.* import sttp.client4.wrappers.FollowRedirectsBackend import sttp.monad.MonadError abstract class MyWrapper[F[_], P] private (delegate: GenericBackend[F, P]) extends GenericBackend[F, P]: def send[T](request: GenericRequest[T, P with Effect[F]]): F[Response[T]] = ??? def close(): F[Unit] = ??? def monad: MonadError[F] = ??? object MyWrapper: def apply[F[_]](delegate: Backend[F]): Backend[F] = // disables any other FollowRedirectsBackend-s further down the delegate chain FollowRedirectsBackend(new MyWrapper(delegate) with Backend[F] {}) ``` ### Custom URI encoding Whenever a redirect request is about to be created, the `FollowRedirectsBackend` uses the value provided in the `Location` header. In its simplest form, a call to `uri"$location"` is being made in order to construct these `Uri`s. The `FollowRedirectsBackend` allows modification of such `Uri` by providing a custom `transformUri: Uri => Uri` function. This might be useful if, for example, some parts of the `Uri` had been initially encoded in a more strict or lenient way. For example: ```scala import sttp.client4.* import sttp.client4.wrappers.{FollowRedirectsBackend, FollowRedirectsConfig} import sttp.model.Uri.QuerySegmentEncoding val myBackend: SyncBackend = DefaultSyncBackend() val backend: SyncBackend = FollowRedirectsBackend( delegate = myBackend, FollowRedirectsConfig( // encodes all special characters in the query segment, including the allowed ones transformUri = _.querySegmentsEncoding(QuerySegmentEncoding.All) ) ) ``` Since encoding query segments more strictly is common, there's a built-in method for that: ```scala import sttp.client4.* import sttp.client4.wrappers.FollowRedirectsBackend val myBackend: SyncBackend = DefaultSyncBackend() val backend: SyncBackend = FollowRedirectsBackend.encodeUriAll(myBackend) ``` # SSL SSL handling can be customized (or disabled) when creating a backend and is backend-specific. Depending on the underlying backend's client, you can customize SSL settings. ## SSL Context Common requirement for handling SSL is creating `SSLContext`. It's required by several backends. ### One way SSL Example assumes that you have your client key store in `.p12` format. If you have your credentials in `.pem` format covert them using: `openssl pkcs12 -export -inkey your_key.pem -in your_cert.pem -out your_cert.p12` Sample code might look like this: ```scala import java.io.FileInputStream import java.security.{KeyStore, SecureRandom} import java.security.cert.X509Certificate import javax.net.ssl.* val TrustAllCerts: X509TrustManager = new X509TrustManager(): def getAcceptedIssuers: Array[X509Certificate] = Array[X509Certificate]() override def checkServerTrusted(x509Certificates: Array[X509Certificate], s: String): Unit = () override def checkClientTrusted(x509Certificates: Array[X509Certificate], s: String): Unit = () val ks: KeyStore = KeyStore.getInstance(KeyStore.getDefaultType) ks.load(new FileInputStream("/path/to/your_cert.p12"), "password".toCharArray) val kmf: KeyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm) kmf.init(ks, "password".toCharArray) val ssl: SSLContext = SSLContext.getInstance("TLS") ssl.init(kmf.getKeyManagers, Array(TrustAllCerts), new SecureRandom) ``` ### Mutual SSL In mutual SSL you are also validating server certificate so example assumes you have it in your trust store. It can be imported to trust store with: `keytool -import -alias server_alias -file server.cer -keystore server_trust` Next, based on [one way SSL example](#one-way-ssl), add `TrustManagerFactory` to your code: ```scala ks.load(new FileInputStream("/path/to/server_trust"), "password".toCharArray) val tmf: TrustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm) tmf.init(ks) val ssl: SSLContext = SSLContext.getInstance("TLS") ssl.init(kmf.getKeyManagers, tmf.getTrustManagers, new SecureRandom) ``` ## Using HttpClient Backends using `HttpClient` provides factory methods accepting `HttpClient`. In this example we are using `IO` and `HttpClientFs2Backend`. Using `SSLContext` from [first section](#ssl-context): ```scala import cats.effect.IO import cats.effect.kernel.Resource import cats.effect.std.Dispatcher import java.net.http.HttpClient import sttp.capabilities.fs2.Fs2Streams import sttp.client4.WebSocketStreamBackend import sttp.client4.httpclient.fs2.HttpClientFs2Backend val httpClient: HttpClient = HttpClient.newBuilder().sslContext(ssl).build() val backend: Resource[IO, WebSocketStreamBackend[IO, Fs2Streams[IO]]] = HttpClientFs2Backend.resourceUsingClient[IO](httpClient) ``` ## Using HttpUrlConnection Using `SSLContext` from [first section](#ssl-context) define a function to customize connection. ```scala import sttp.client4.* import sttp.client4.httpurlconnection.HttpURLConnectionBackend import java.net.HttpURLConnection import javax.net.ssl.HttpsURLConnection def useSSL(conn: HttpURLConnection): Unit = conn match case https: HttpsURLConnection => https.setSSLSocketFactory(ssl.getSocketFactory) case _ => () val backend = HttpURLConnectionBackend(customizeConnection = useSSL) ``` It is also possible to set default `SSLContext` using `SSLContext.setDefault(ssl)`. ## Using Pekko-http Using `SSLContext` from [first section](#ssl-context) create a `HttpsConnectionContext`. ```scala import org.apache.pekko.actor.ActorSystem import org.apache.pekko.http.scaladsl.{ConnectionContext, HttpsConnectionContext} import sttp.client4.pekkohttp.* val actorSystem: ActorSystem = ActorSystem() val https: HttpsConnectionContext = ConnectionContext.httpsClient(ssl) val backend = PekkoHttpBackend.usingActorSystem(actorSystem, customHttpsContext = Some(https)) ``` For more information refer to [pekko docs](https://pekko.apache.org/docs/pekko-http/current/client-side/client-https-support.html). ## Using OkHttp Using `SSLContext` from [first section](#ssl-context) create a `OkHttpClient`. Specifying `X509TrustManager` explicitly is required for OkHttp. You can instantiate one your self, or extract one from `tmf: TrustManagerFactory` from [first section](#ssl-context). ```scala import okhttp3.OkHttpClient import sttp.client4.okhttp.OkHttpFutureBackend import javax.net.ssl.X509TrustManager val yourTrustManager: X509TrustManager = ??? val client: OkHttpClient = new OkHttpClient.Builder() .sslSocketFactory(ssl.getSocketFactory, yourTrustManager) .build() val backend = OkHttpFutureBackend.usingClient(client) ``` For more information refer to [okhttp docs](https://square.github.io/okhttp/https/). # 1. Extract stream from http4s Date: 2019-10-04 ## Context When getting the response as a stream from the http4s backend, the underlying connection is closed as soon as the effect reading the response is finished. That is, before the user had a chance to consume the stream. ## Decision In http4s, all interaction with the response body is protected: the response stream *must* be consumed entirely into an effect by a provided function `Stream[F, Byte] => F[T]`. This is different than in sttp: if the user wants to get the response as a stream, it is up to the user to consume the stream. Implicitly, there's an assumption that the underlying connection will be closed only after the whole stream is consumed. http4s has "internal" consumption of the stream, while sttp - external. This way of consuming streams is less safe, but that's a design decision of sttp's client API. Hence, we need to adjust the way http4s works and "extract" the stream from http4s, releasing the connection only when the stream is fully read (or completes with an error). To do that, in `Http4sBackend`, we create two `Deferred`s: one to signal that the request body has been fully consumed (`responseBodyCompleteVar`), and another, `responseVar`, to extract the response. When the request is read, first the stream body is adjusted, so that the completion var is filled in the stream's finalizer. Then, we create the response and fill the `responseVar` with the response value. Finally, we read from the completion, so that http4s closes the connection only when the response is read. The ordering of these effects is crucial, so that we don't deadlock! That process is started in the background (`sendRequest.start`). In the foreground, we read the content of the `responseVar` and return it the user as soon as it's available. # 2. HTTP model conventions Date: 2019-10-24 ## Context The classes which form the HTTP model (in the `sttp.model` package) vary in the conventions they use for error-handling and the parsing interface they offer. ## Decision We want to unify the behavior of the HTTP model classes so that they work in a predictable and similar way. The conventions are: * `.toString` returns a representation of the model class in a format as in an HTTP request/response. For example, for an uri this will be `http://...`, for a header `[name]: [value]`, etc. * constructors of the model classes are private; instances should be created through methods on the companion objects. * `[SthCompanionObject].parse(serialized: String): Either[String, Sth]`: returns an error message or an instance of the model class * `[SthCompanionObject].unsafeApply(values)`: creates an instance of the model class; validates the input values and in case of an error, *throws an exception*. An error could be e.g. that the input values contain characters outside of the allowed range * `[SthCompanionObject].safeApply(...): Either[String, Sth]`: same as above, but doesn't throw exceptions. Instead, returns an error message or the model class instance * `[SthCompanionObject].apply(...): Sth`: creates the model type, without validation, and without throwing exceptions # 3. Separate backend types Date: 2023-02-22 ## Context Sttp base types are [being refactored](https://github.com/softwaremill/sttp/pull/1703) to decrease the learning curve, and generally improve the developer experience. ## Decision The `SttpBackend` type is split into a type hierarchy, with new subtypes representing backends with different effects and capabilities. The root of the hierarchy is `GenericBackend`, which corresponds to the old backend: it is parametrised with the type constructor used to represent effects, and a set of capabilities that the backend supports. Additionally, there's a number of subtypes: `Backend`, `SyncBackend`, `StreamBackend`, `WebSocketBackend` and `WebSocketStreamBackend` which fix the capabilities, and in case of `SyncBackend` also the effect type. ### What do we gain Thanks to the different kinds of backends being represented by top-level types, the `send` methods on request descriptions can now have more developer-friendly and precise signatures. For example: ```scala class Request[T] { def send[F[_]](backend: Backend[F]): F[Response[T]] = backend.send(this) def send(backend: SyncBackend): Response[T] = backend.send(this) } ``` Specifically, for a request sent using a synchronous request, the result type is a `Response` directly, without the `Identity` wrapper. This improves ergonomics (e.g. when using autocomplete) and type inference. Moreover: * users are exposed to simpler types, such as `SyncBackend` instead of `SttpBackend[Identity, Any]` * error messages are more precise, pointing to the specific backend type that is required to send a request ### What do we loose Backend wrappers, which are designed to work with any delegate backend, now need to have 5 alternate constructors, for each specific backend subtype. For example: ```scala object FollowRedirectsBackend { def apply(delegate: SyncBackend): SyncBackend = new FollowRedirectsBackend(delegate) with SyncBackend {} def apply[F[_]](delegate: Backend[F]): Backend[F] = new FollowRedirectsBackend(delegate) with Backend[F] {} def apply[F[_]](delegate: WebSocketBackend[F]): WebSocketBackend[F] = new FollowRedirectsBackend(delegate) with WebSocketBackend[F] {} def apply[F[_], S](delegate: StreamBackend[F, S]): StreamBackend[F, S] = new FollowRedirectsBackend(delegate) with StreamBackend[F, S] {} def apply[F[_], S](delegate: WebSocketStreamBackend[F, S]): WebSocketStreamBackend[F, S] = new FollowRedirectsBackend(delegate) with WebSocketStreamBackend[F, S] {} } ``` Additionally, adding more capabilities will require enriching the type hierarchy with more subtypes, as well as adding new variants to any backend wrappers. However, in the course of sttp's history, no new capabilities have been added, and we do not foresee having to add more in the future. ### Alternate designs We [have explored](https://github.com/adpi2/sttp/pull/5) an alternate design, where the `GenericBackend` was a stand-alone type, to which types representing to specific capabilities (`Backend`, `StreamBackend` etc.) delegate. This partially replaced the inheritance with composition. The main goal of the change was to allow implementing generic backend wrappers in a more straightforward way. The delegating types included a `SelfType` type parameter, which was leveraged by generic backend wrappers, e.g.: ```scala trait SyncBackend extends Backend[Identity] { self => override type Capabilities <: Any override type SelfType <: SyncBackend } object FollowRedirectsBackend { def apply[F[_]](delegate: Backend[F]): delegate.SelfType = delegate.wrap(new FollowRedirectsBackend(_, FollowRedirectsConfig.Default)) } ``` However, this had two major drawbacks: * the type inference on the wrapped backends was worse, as it returned e.g. `SyncBackend.SelfType` instead of `SyncBackend` * alternatively, we could fix the self type to become type aliases instead of type bounds, but then the subtyping relation between the backend types has been lost * additionally, using two wrappers in a generic way resulted in type such as `delegate.SelfType#SelfType`, which are not only not readable, but also rejected by the Scala 3 compiler. # 4. Separate request types Date: 2023-02-23 ## Context Sttp base types are [being refactored](https://github.com/softwaremill/sttp/pull/1703) to decrease the learning curve, and generally improve the developer experience. ## Decision The `RequestT` type is being split into a type hierarchy, with new subtypes representing requests requiring different capabilities. The requests are split into request builders: top-level `PartialRequest` and traits with common methods: `PartialRequestBuilder` and `RequestBuilder`. Additionally, we've got a base trait for a generic request (that is a request description, which includes the uri and method), `GenericRequest`. This is then implemented by a number of top-level classes: `Request`, `StreamRequest`, `WebSocketRequest` and `WebSocketStreamRequest`. The capabilities supported by these requests are fixed. Setting a response description or an input stream body promotes a `Request` or `PartialRequest` to the appropriate type. ### What do we gain The main gain is that the types are simpler. Before, the `RequestT[U, T, R]` type had three type parameters: a marker type constructor, specifying if the uri/method are specified; the target type, to which the response is deserialized; and the set of required capabilities. With the new design, the base `Request[T]` type has one type parameter only (response target type). This is expanded to two type parameters in the other request types, to express the additional requirements. Moreover, the specific request types clearly specify what type of `Backend` is required to send the request. ### What do we loose It is harder to write generic methods which manipulate *any* request description (including partial request). This is still possible, by properly parametrizing the method with the subtype of `PartialRequestBuilder` used, but might not be as straightforward as before. Moreover, integration with Tapir might be more challenging, as we cannot simply accept an endpoint with a set of capabilities, and produce a request with a mirror capability set. Special-casing on streaming/websocket capabilities will be required. Finally, some flexibility is lost, as there are no partial streaming/websocket requests. That is, the method & uri need to be specified on the request *before* specifying that the body should be streamed or that the response should be a web socket one. This could be amended by introducing additional `PartialRequest` subtypes, however it is not clear right now that it is necessary.