Why you should avoid ioutil.ReadAll in Go

When you have an io.Reader , the most common way to read is with ioutil.ReadAll but it is not the best and most efficient way.

So here are three functions to compare the different methods to unmarshal JSON from an io.Reader …

1. With ioutil.ReadAll

func IOUtilReadAll(reader io.Reader) (map[string]interface{}, error) {
   var (
      m    map[string]interface{}
      b, _ = ioutil.ReadAll(reader)
   )

   return m, json.Unmarshal(b, &m)
}

2. With io.Copy

func IOCopy(reader io.Reader) (map[string]interface{}, error) {
   var (
      m    map[string]interface{}
      buf  bytes.Buffer
      _, _ = io.Copy(&buf, reader)
   )

   return m, json.Unmarshal(buf.Bytes(), &m)
}

3. With json.Decoder

func JsonDecoder(reader io.Reader) (map[string]interface{}, error) {
   var m map[string]interface{}

   return m, json.NewDecoder(reader).Decode(&m)
}

Now, let’s benchmark them!

var jsonStr = `{"one":"foobar","two":"foobar","three":"foobar","four":"foobar","five":"foobar","six":"foobar","seven":"foobar","eight":"foobar","nine":"foobar","ten":"foobar"}`
func BenchmarkJsonDecoder(b *testing.B) {
   b.ReportAllocs()
   for i := 0; i < b.N; i++ {
      JsonDecoder(strings.NewReader(jsonStr))
   }
}

func BenchmarkIOUtilReadAll(b *testing.B) {
   b.ReportAllocs()
   for i := 0; i < b.N; i++ {
      IOUtilReadAll(strings.NewReader(jsonStr))
   }
}

func BenchmarkIOCopy(b *testing.B) {
   b.ReportAllocs()
   for i := 0; i < b.N; i++ {
      IOCopy(strings.NewReader(jsonStr))
   }
}

I am passing strings.NewReader to all three functions to be consistent, although it is not necessary for json.Decoder since it does not drain the io.Reader .

And here are the results…

BenchmarkJsonDecoder
BenchmarkJsonDecoder-8        180230       6741 ns/op     2766 B/op       69 allocs/op
BenchmarkIOUtilReadAll
BenchmarkIOUtilReadAll-8      168019       6928 ns/op     4126 B/op       69 allocs/op
BenchmarkIOCopy
BenchmarkIOCopy-8             158901       6677 ns/op     2286 B/op       69 allocs/op

Boom! Although there are minimal differences in ns/op, B/op is a staggering 2x!

But don’t take my words. If you are using ioutil.ReadAll , run a benchmark and compare the difference. This might just be a small performance improvement in your application and the effort could be better spent elsewhere.

I hope this is helpful. Do clap for me (or rather this article) if you think so.

Do follow me if you have not already done so!