Understanding the http handlerfunc wrapper technique in Go

Question

I saw an article written by Mat Ryer about how you can use a server type and http handlers of the type that are wrappers for func(http.ResponseWriter, *http.Request)

I see this as a more elegant way to build REST APIs, however I'm totally stumped on getting the wrapper to function correctly. I either get a mismatched type error at compilation or a 404 at invocation.

This is basically what I have for study purposes at the moment.

package main

import(
   "log"
   "io/ioutil"
   "encoding/json"
   "os"
   "net/http"
   "github.com/gorilla/mux"
)

type Config struct {
   DebugLevel int `json:"debuglevel"`
   ServerPort string `json:"serverport"`
}

func NewConfig() Config {

   var didJsonLoad bool = true

   jsonFile, err := os.Open("config.json")
   if(err != nil){
      log.Println(err)
      panic(err)
      recover()
      didJsonLoad = false
   }

   defer jsonFile.Close()

   jsonBytes, _ := ioutil.ReadAll(jsonFile)

   config := Config{}

   if(didJsonLoad){
      err = json.Unmarshal(jsonBytes, &config)
      if(err != nil){
         log.Println(err)
         panic(err)
         recover()
      }
   }

   return config
}

type Server struct {
   Router *mux.Router
}

func NewServer(config *Config) *Server {
   server := Server{
      Router : mux.NewRouter(),
   }

   server.Routes()

   return &server
}

func (s *Server) Start(config *Config) {
   log.Println("Server started on port", config.ServerPort)
   http.ListenAndServe(":"+config.ServerPort, s.Router)
}

func (s *Server) Routes(){
   http.Handle("/sayhello", s.HandleSayHello(s.Router))
}

func (s *Server) HandleSayHello(h http.Handler) http.Handler {
   log.Println("before")
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      w.Write([]byte("Hello."))
      h.ServeHTTP(w, r)
   })
}

func main() {
   config := NewConfig()
   server := NewServer(&config)
   server.Start(&config)
}

As this is right now, I'll only get back a 404 invoking localhost:8091/sayhello. (Yes, that is the port I've set in my config file.)

Before, since I'm using Gorilla Mux, I was setting the handler like so:

func (s *Server) Routes(){
    s.Router.HandleFunc("/sayhello", s.HandleSayHello)
}

Which gave me this error I was totally stumped on. cannot use s.HandleSayHello (type func(http.Handler) http.Handler) as type func(http.ResponseWriter, *http.Request) in argument to s.Router.HandleFunc

I saw in solution for this SO post that I should use http.Handle and pass in the router.

func (s *Server) Routes(){
   http.Handle("/sayhello", s.HandleSayHello(s.Router))
}

But now how do I prevent the actual function from executing when I set my routes? The "before" in my print statement is showing up before the server starts. I don't see it as a problem now, but it might be once I start writing more complex middleware for database queries I plan to use this for.

Researching this technique further, I found other readings that suggested I need a middleware or handler type defined.

I don't fully understand what's going on in these examples because the types they're defining don't seem to be getting used.

This resource shows how the handlers are written, but not how the routes are set up.

I did find that Gorilla Mux has built in wrappers for this stuff, but I'm having a hard time understanding the API.

The example they show is like this:

func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Do stuff here
        log.Println(r.RequestURI)
        // Call the next handler, which can be another middleware in the chain, or the final handler.
        next.ServeHTTP(w, r)
    })
}

And the routes are defined like this:

r := mux.NewRouter()
r.HandleFunc("/", handler)
r.Use(loggingMiddleware)

What is the purpose of r.Use when it's not registering the url route? How is handler being used?

When my code is written like this, I get no compilation errors, but I don't understand how my function is suppose to write back "Hello". I guess I could be using w.Write in the wrong place.

Solution 1

I think you might be mixing up "middleware" with real handlers.

http handlers

Types that implement the ServeHTTP(w http.ResponseWriter, r *http.Request) method satisfy the http.Handler interface and therefore instances of those types can, for example, be used as the second argument to the http.Handle function or the equivalent http.ServeMux.Handle method.

An example might make this more clear:

type myHandler struct {
    // ...
}

func (h myHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte(`hello world`))
}

func main() {
    http.Handle("/", myHandler{})
    http.ListenAndServe(":8080", nil)
}

---

http handler funcs

Functions with the signature func(w http.ResponseWriter, r *http.Request) are http handler funcs that can be converted to an http.Handler using the http.HandlerFunc type. Notice that the signature is the same as the signature of the http.Handler's ServeHTTP method.

For example:

func myHandlerFunc(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte(`hello world`))
}

func main() {
    http.Handle("/", http.HandlerFunc(myHandlerFunc))
    http.ListenAndServe(":8080", nil)
}

The expression http.HandlerFunc(myHandlerFunc) converts the myHandlerFunc function to the type http.HandlerFunc which implements the ServeHTTP method so the resulting value of that expression is a valid http.Handler and therefore it can be passed to the http.Handle("/", ...) function call as the second argument.

Using plain http handler funcs instead of http handler types that implement the ServeHTTP method is common enough that the standard library provides the alternatives http.HandleFunc and http.ServeMux.HandleFunc. All HandleFunc does is what we do in the above example, it converts the passed in function to http.HandlerFunc and calls http.Handle with the result.

---

http middleware

Functions with a signature similar to this func(h http.Handler) http.Handler are considered middleware. Keep in mind that the signature of the middleware isn't restricted, you could have middleware that takes more arguments than just a single handler and returns more values as well, but in general a function that takes at least one handler and retruns at least one new handler can be considered middleware.

As an example take a look at http.StripPrefix.

---

Let's now clear some of the apparent confusion.

#1

func (s *Server) HandleSayHello(h http.Handler) http.Handler {

The name of the method and the way you used it before, passing it directly to HandleFunc, suggest that you want this to be a normal http handler func, but the signature is that of middleware and that's the reason for the error you got:

cannot use s.HandleSayHello (type func(http.Handler) http.Handler) as type func(http.ResponseWriter, *http.Request) in argument to s.Router.HandleFunc

So updating your code to something like the code below will get rid of that compile error and will also properly render the "Hello." text when visiting /sayhello.

func (s *Server) HandleSayHello(w http.ResponseWriter, r *http.Request) {
      w.Write([]byte("Hello."))
}

func (s *Server) Routes(){
    s.Router.HandleFunc("/sayhello", s.HandleSayHello)
}

---

#2

As this is right now, I'll only get back a 404 invoking localhost:8091/sayhello.

The problem is in these two lines

http.Handle("/sayhello", s.HandleSayHello(s.Router))

and

http.ListenAndServe(":"+config.ServerPort, s.Router)

The http.Handle func registers the passed in handler with the default ServeMux instance, it does not register it with the gorilla router instance in s.Router as you seem to assume, and then you are passing s.Router to the ListenAndServe func which uses it to serve every request comming to localhost:8091, and since s.Router has no handler registered with it you get the 404.

---

#3

But now how do I prevent the actual function from executing when I set my routes? The "before" in my print statement is showing up before the server starts.

func (s *Server) Routes(){
   http.Handle("/sayhello", s.HandleSayHello(s.Router))
}

func (s *Server) HandleSayHello(h http.Handler) http.Handler {
   log.Println("before")
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      w.Write([]byte("Hello."))
      h.ServeHTTP(w, r)
   })
}

Depends on what you mean by "actual function". In Go you execute functions by adding parenthesis at the end of their name. So what is executed here when you're setting the routes is the http.Handle function and the HandleSayHello method.

The HandleSayHello method has essentially two statements in its body, the function-call-expression statement log.Println("before") and the return statement return http.HandlerFunc(... and both of these will be executed every time you call HandleSayHello. However the statements inside the returned function, the handler, will not be executed when you call HandleSayHello, instead they will be executed when the returned handler is called.

You don't want "before" to be printed when HandleSayHello is called but you want it to be printed when the returned handler is called? All you need to do is move the log line down to the returned handler:

func (s *Server) HandleSayHello(h http.Handler) http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
      log.Println("before")
      w.Write([]byte("Hello."))
      h.ServeHTTP(w, r)
   })
}

This code of course now makes little sense, even as an example for educational purposes it will confuse rather than clarify the concept of handlers and middleware.

Instead maybe consider something like this:

// the handler func
func (s *Server) HandleSayHello(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello."))
}

// the middleware
func (s *Server) PrintBefore(h http.Handler) http.Handler {
       return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
               log.Println("before") // execute before the actual handler
               h.ServeHTTP(w, r)     // execute the actual handler
       })
}

func (s *Server) Routes(){
        // PrintBefore takes an http.Handler but HandleSayHello is an http handler func so
        // we first need to convert it to an http.Hanlder using the http.HandlerFunc type.
        s.Router.HandleFunc("/sayhello", s.PrintBefore(http.HandlerFunc(s.HandleSayHello)))
}

---

#4

r := mux.NewRouter()
r.HandleFunc("/", handler)
r.Use(loggingMiddleware)

What is the purpose of r.Use when it's not registering the url route? How is handler being used?

Use registers the middleware at the router level which means that all handlers registered with that router will have the middleware executed before they themselves are executed.

For example the above code is equivalent to this:

r := mux.NewRouter()
r.HandleFunc("/", loggingMiddleware(handler))

Of course Use is not intended to be unnecessary and confusing, it is useful if you have many endpoints all with different handlers and all of them need a bunch of middleware to be applied to them.

Then code like this:

r.Handle("/foo", mw1(mw2(mw3(foohandler))))
r.Handle("/bar", mw1(mw2(mw3(barhandler))))
r.Handle("/baz", mw1(mw2(mw3(bazhandler))))
// ... hundreds more

Can be radically simplified:

r.Handle("/foo", foohandler)
r.Handle("/bar", barhandler)
r.Handle("/baz", bazhandler)
// ... hundreds more
r.Use(mw1, mw2, m3)

Solution 2

From gorilla mux doc file:

Middlewares are (typically) small pieces of code which take one request, do something with it, and pass it down to another middleware or the final handler.

The r.Use() is useful for registering a middleware. You can register middleware as many as possible.

r.HandleFunc("/hello", func (w http.ResponseWriter, r *http.Request) {
    fmt.Println("from handler")
    w.Write([]byte("Hello! \n"))
})

r.Use(func (next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // do something here
        fmt.Println("from middleware one")
        next.ServeHTTP(w, r)
    })
})

r.Use(func (next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // do another thing here
        fmt.Println("from middleware two")
        next.ServeHTTP(w, r)
    })
})

r.Use(func (next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // do something again but differently here
        fmt.Println("from middleware three")
        next.ServeHTTP(w, r)
    })
})

If you see the code above, on each middleware there is statement next.ServeHTTP(w, r). The statement is used to proceed the incoming request to the next step (it can be next middleware, or the actual handler).

Every middleware will always be executed before the actual handler. The execution it self happen in sequential order, depending on the order of middleware registration.

After all middleware executed successfully, the next.ServeHTTP(w, r) of the last middleware will proceed the incoming request to go to actual handler (in example above, it's handler of /hello route).

When you access the /hello, the log will print:

from middleware one
from middleware two
from middleware three
from handler

---

If you want on certain condition the incoming request will not be proceed, then simply don't call the next.ServeHTTP(w, r). Example:

r.Use(func (next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // ...

        if someCondition {
            next.ServeHTTP(w, r)
        } else {
            http.Error(w, "some error happen", http.StatusBadRequest)
        }
    })
})

---

Middleware often used to perform some process on the incoming request, before or after the handler called. For example like: CORS configuration, CRSF checking, gzip compression, logging, etc.

Comments

[GhostRavenstorm]

This is great. I get what Use does now. Unfortunately it's not the solution I'm looking for. I'm trying to write handler that don't handle requests, but return functions that do according to Mat Ryer's post medium.com/statuscode/… I'm trying to do it with Gorilla Mux which is the router I'm using.

[GhostRavenstorm]

Focusing on #1 only, I'm trying to understand what makes something satisfies the requirements for ServerHTTP. http.Handle can only register functions that are http.HandlerFunc type, right? Wouldn't registering a function returning that type work? I'm trying to follow Mat Ryer's examples as closely as possible because I have many handler dependencies I want to pass as arguments to the handling function.

[mkopriva]

@GhostRavenstorm No, http.Handle takes a value of any type that implements the http.Handler interface, http.HandlerFunc is one of those types that implements that interface, but it's not the only one. So your "http.Handle can only register functions that are http.HandlerFunc type" statement is incorrect.

[GhostRavenstorm]

Okay, I understand that more clearly. I tried having a handlerfunc return a handlerfunc, but I still get type errors. The function being passed in HandleFunc seem to must match the signature of ServerHTTP(w http.ResponseWriter, r *http.Request) exactly or it causes problems. In that case, I don't understand how this post is using valid code. There seesm to be some entry point to these examples not being shown that I'm missing to get these to work.

[mkopriva]

@GhostRavenstorm: "Wouldn't registering a function returning that type work?" again, no. If the function F takes an argument of type T then you have to pass in a value of type T there's nothing else you can pass in; it is not enough to just pass a function that returns T. Whatever you've read in Mat Ryer's article you clearly misunderstood so I suggest you re-read it, a couple times. And just to be clear I'm not trying to insult you, I had to read countless articles till I was able to wrap my head around interfaces.

[mkopriva]

@GhostRavenstorm you are correct here, http.HandleFunc takes in a function of type http.HandlerFunc. Note that http.HandleFunc is not the same as http.Hanlde.

[mkopriva]

@GhostRavenstorm also note that for the argument to http.HandleFunc the method ServeHTTP(... is not important, what's important is that the signature, ie type, of the function that you pass in is the same as defined by the argument, ie. func(http.ResponseWriter, *http.Request). The ServeHTTP method is only important for the http.Handle or anything that depends on the http.Handler interface which declares that method as one of its members.

[mkopriva]

@GhostRavenstorm in that article are you referring to the examples where he is using the HandleFunc method? If so note that the what he is passing in as the second argument is always the result of a function call, e.g. s.router.HandleFunc("/api/", s.handleAPI()) here he is not passing handleAPI, he is calling it and whatever that call returns gets passed to HandleFunc, for example if you change that code to s.router.HandleFunc("/api/", s.handleAPI) (removing the ()) then his code won't compile either.

[Kevin]

This is a great clarification of the HTTP handling API.