Encode / decode URLs

Question

What's the recommended way of encoding and decoding entire URLs in Go? I am aware of the methods url.QueryEscape and url.QueryUnescape, but they don't seem to be exactly what I am looking for. Specifically I am looking for methods like JavaScript's encodeURIComponent and decodeURIComponent.

Thanks.

Solution 1

You can do all the URL encoding you want with the net/url module. It doesn't break out the individual encoding functions for the parts of the URL, you have to let it construct the whole URL. Having had a squint at the source code I think it does a very good and standards compliant job.

Here is an example (playground link)

package main

import (
    "fmt"
    "net/url"
)

func main() {

    Url, err := url.Parse("http://www.example.com")
    if err != nil {
        panic("boom")
    }

    Url.Path += "/some/path/or/other_with_funny_characters?_or_not/"
    parameters := url.Values{}
    parameters.Add("hello", "42")
    parameters.Add("hello", "54")
    parameters.Add("vegetable", "potato")
    Url.RawQuery = parameters.Encode()

    fmt.Printf("Encoded URL is %q\n", Url.String())
}

Which prints-

Encoded URL is "http://www.example.com/some/path/or/other_with_funny_characters%3F_or_not/?vegetable=potato&hello=42&hello=54"

Solution 2

From MDN on encodeURIComponent:

encodeURIComponent escapes all characters except the following: alphabetic, decimal digits, '-', '_', '.', '!', '~', '*', ''', '(', ')'

From Go's implementation of url.QueryEscape (specifically, the shouldEscape private function), escapes all characters except the following: alphabetic, decimal digits, '-', '_', '.', '~'.

Unlike Javascript, Go's QueryEscape() will escape '!', '*', ''', '(', ')'. Basically, Go's version is strictly RFC-3986 compliant. Javascript's is looser. Again from MDN:

If one wishes to be more stringent in adhering to RFC 3986 (which reserves !, ', (, ), and *), even though these characters have no formalized URI delimiting uses, the following can be safely used:

function fixedEncodeURIComponent (str) {
  return encodeURIComponent(str).replace(/[!'()]/g, escape).replace(/\*/g, "%2A");
}

Solution 3

As of Go 1.8, this situation has changed. We now have access to PathEscape in addition to the older QueryEscape to encode path components, along with the unescape counterpart PathUnescape.

Solution 4

How about this:

template.URLQueryEscaper(path)

Solution 5

For mimicking Javascript's encodeURIComponent(), I created a string helper function.

Example: Turns "My String" to "My%20String"

https://github.com/mrap/stringutil/blob/master/urlencode.go

import "net/url"

// UrlEncoded encodes a string like Javascript's encodeURIComponent()
func UrlEncoded(str string) (string, error) {
    u, err := url.Parse(str)
    if err != nil {
        return "", err
    }
    return u.String(), nil
}

View more solutions

Comments

[thwd]

What's the difference between encodeURIComponent and url.QueryEscape?

[Ivan Rave]

it encodes "a+q" to "a+q", but encodeURIComponent('a+q') = 'a%2Bq'

[Shay Tsadok]

Worth to mention: if the parsed url already contains query params, then instead of parameters := url.Values{} you should initialize with parameters, err := url.ParseQuery(Url.RawQuery)

[tom10271]

prog.go:5:2: cannot find package "template" in any of: /usr/local/go/src/template (from $GOROOT) /go/src/template (from $GOPATH)

[TomSawyer]

it encode whole url, included domain, not only path and query strings like browser does

[Xian Shu]

@TomSawyer why do you put whole url to the method? It is your problem, not the method. Please think this: "a.b.com" and "c.d.com/?url=http://a.b.com".