Scala http operations

Question

How do I perform the following in Scala?

• HTTP Get
• HTTP Get With custom headers
• HTTP Post

Solution 1

You could try out Dispatch. A little difficult to grasp at first, but after a while I've started to like it. It works on top of HttpClient.

import dispatch.Http
import Http._
// Get
Http(url("http://youruri.com/yo") >>> System.out)
// Get with header
Http(url("http://youruri.com/yo") <:< Map("Accept" -> "application/json") >>> System.out)
// Post
Http(url("http://youruri.com/yo") << yourPostData >|)

Solution 2

You can simply use java.net.URL to send HTTP GET and HTTP POST requests. You can also set HTTP request headers on the HttpURLConnection like this:

val con = url.openConnection.asInstanceOf[HttpURLConnection]
con.setRequestProperty("Header", "Value")

I have written myself a utility class which does exactly this. You can see it here:

https://github.com/gruenewa/gruenewa-misc/blob/master/gruenewa-wsclient/src/main/scala/gruenewa/wsclient/Service.scala

Solution 3

this is my own implementation of a simple Http client including cookies management. Maybe it will be useful for you. But I'm not sure if header modification is directly possible (it may require your own implementation of URLConnection).

import java.io.OutputStreamWriter
import java.net.{URLConnection, URL}

class Http(userAgent: String,
           encoding: String,
           HttpRequestTimeout: Int = 15000) {

  import collection.JavaConversions._
  import Implicits.wrapInputStream
  import java.net.URLEncoder.encode

  var cookies = Map[String, String]()

  private def loadCookies(conn: URLConnection) {
    for ((name, value) <- cookies) conn.setRequestProperty("Cookie", name + "=" + value)
  }

  private def saveCookies(conn: URLConnection) {
    conn.getHeaderFields.lift("Set-Cookie") match {
      case Some(cList) => cList foreach { c =>
        val (name,value) = c span { _ != '=' }
        cookies += name -> (value drop 1)
      }
      case None =>
    }
  }

  private def encodePostData(data: Map[String, String]) =
    (for ((name, value) <- data) yield encode(name, encoding) + "=" + encode(value, encoding)).mkString("&")

  def Get(url: String) = {
    val u = new URL(url)
    val conn = u.openConnection()
    conn.setRequestProperty("User-Agent", userAgent)
    conn.setConnectTimeout(HttpRequestTimeout)

    loadCookies(conn)

    conn.connect

    saveCookies(conn)

    conn.getInputStream.mkString
  }

  def Post(url: String, data: Map[String, String]) = {
    val u = new URL(url)
    val conn = u.openConnection

    conn.setRequestProperty("User-Agent", userAgent)
    conn.setConnectTimeout(HttpRequestTimeout)

    loadCookies(conn)

    conn.setDoOutput(true)
    conn.connect

   val wr = new OutputStreamWriter(conn.getOutputStream())
    wr.write(encodePostData(data))
    wr.flush
    wr.close


    saveCookies(conn)

    conn.getInputStream.mkString
  }
}

Solution 4

While I appreciate the Dispatch library for all it's worth, the syntax still confuses me a bit.

Someone directed me to scalaj-http the other day which seems a little easier

Solution 5

Regarding simply GETting data from URL. If you don't want to use external sources, then:

  val in = scala.io.Source.fromURL("http://some.your.url/params?start&here", 
                                   "utf-8")
  for (line <- in.getLines)
    println(line)  

For all other stuff, you can choose any method you like from answers above.

View more solutions

Comments

[Andrey]

The only problem with Dispatch is that you can hardly find a method with a meaningful name. Operator methods are everywhere. Your code may easily look like a Morse code.

[thoredge]

I've got a project on github (github.com/thoraage/coffee-trader) where I tried the borders of what I could do with Dispatch and the RestHelpers of Lift. It provides some examples; which are otherwise impossible to search for on google (since it looks like morse code).

[parsa]

emoticons! someone will get really confused.

[James Moore]

github.com/gruenewa/gruenewa-wsclient/blob/master/src/main/… just returns 404

[gruenewa]

I fixed the link. Thank you, James.

[Wilfred Springer]

If you struggle with the operators, you might want to take a look at the overview page here: flotsam.nl/dispatch-periodic-table.html

[Mike]

I don't think your loadCookies implementation works for multiple cookies, here is my suggestion: conn.setRequestProperty("Cookie", cookies.map{case (name, value) => name + "=" + value}.mkString("; "))

[nemoo]

OMG, this API brings out the worst in scala. An Api that is useless unless you have a cheatsheet in front of you all the time.

[WestCoastProjects]

From whence cometh Implicits.WrapInputStream i can not find that even by googling.

[WestCoastProjects]

Anyways the same can be accomplished with @inline def readStream(is: InputStream) = scala.io.Source.fromInputStream(is).mkString -except not implicits.