从管道读取的 Golang 读取大量数据

你的代码效率很低.它在循环中每次分配和初始化data.

for {
    data := make([]byte, 4<<20) // Read 4MB at a time
}

reader作为io.Reader的代码是错误的.例如，您忽略了_, err := reader.Read(data)读取的字节数，并且没有正确处理err个错误.

Package io个

import "io" 

type Reader个

type Reader interface {
        Read(p []byte) (n int, err error)
}

Reader是包装基本读取方法的接口.

READ最多将len(P)字节读入p.它返回字节数 读取(0<；=n<；=len(P))和遇到的任何错误.即使读了也行 返回n<；len(P)，则在执行 打电话.如果某些数据可用，但不是len(P)字节，请读取 通常返回可用的内容，而不是等待更多.

When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call. An instance of this general case is that a Reader returning a non-zero number of bytes at the end of the input stream may return either err == EOF or err == nil. The next Read should return 0, EOF regardless.

Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.

不鼓励Read的实现返回零字节 计数无错误，除非len(P)==0.来电者应该请客 返回0和零，表示什么都没有发生； 具体地说，它并不表示EOF.

实现不能保留p.

下面是一个符合io.Reader接口的模型文件读取程序:

package main

import (
    "bufio"
    "io"
    "log"
    "os"
)

func main() {
    nBytes, nChunks := int64(0), int64(0)
    r := bufio.NewReader(os.Stdin)
    buf := make([]byte, 0, 4*1024)
    for {
        n, err := r.Read(buf[:cap(buf)])
        buf = buf[:n]
        if n == 0 {
            if err == nil {
                continue
            }
            if err == io.EOF {
                break
            }
            log.Fatal(err)
        }
        nChunks++
        nBytes += int64(len(buf))
        // process buf
        if err != nil && err != io.EOF {
            log.Fatal(err)
        }
    }
    log.Println("Bytes:", nBytes, "Chunks:", nChunks)
}

输出:

2014/11/29 10:00:05 Bytes: 5589891 Chunks: 1365