File input and output in Go is the set of operations to read, write, and process persistent data using packages like os, io, and bufio.
Many programs need to read a configuration, save a log, or process a large CSV.
Go, true to its Unix roots, treats files as first-class citizens. The standard library offers two fundamental packages:
os: Provides a platform-independent interface to interact with the Operating System (open, close, permissions).io: Defines the abstract interfaces for reading and writing data streams.
We’ll look at everything from functions that load entire content to incremental processing using I/O interfaces.
Complete reading and writing in memory
If you’re working with small files (like a .json configuration or a certificate), the easiest approach is to read everything at once and store it in a variable.
Since Go 1.16, this is done directly with the os package.
Reading an entire file
package main
import (
"fmt"
"os"
)
func main() {
// Reads the ENTIRE file and loads it into memory ([]byte)
content, err := os.ReadFile("config.txt")
if err != nil {
panic(err) // Or handle the error appropriately
}
// Convert bytes to string for printing
fmt.Println(string(content))
}Writing an entire file
func main() {
message := []byte("Hello world from Go!")
// Writes the data. If it doesn't exist, creates it. If it exists, OVERWRITES it.
// 0644 are Unix permissions (Read/Write for me, Read for everyone else)
err := os.WriteFile("output.txt", message, 0644)
if err != nil {
panic(err)
}
}Watch out for RAM: These functions load all content into memory. If you try to read a 4GB log with os.ReadFile, your program will consume 4GB of RAM and the OS will likely kill it (OOM Kill).
Incremental reading with bufio
For large files or when we want to process data line by line, the correct strategy is to open the file, create a Scanner, and read a bit at a time.
The bufio (Buffered I/O) package is our best friend here.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// 1. Open the file (we get a *os.File pointer)
file, err := os.Open("giant_book.txt")
if err != nil {
panic(err)
}
// IMPORTANT: Schedule the close immediately
defer file.Close()
// 2. Create a Scanner that wraps the file
scanner := bufio.NewScanner(file)
// 3. Iterate line by line
// Scan() advances the cursor and returns true while data exists
lineNum := 1
for scanner.Scan() {
line := scanner.Text() // Get the current line as a string
fmt.Printf("Line %d: %s\n", lineNum, line)
lineNum++
}
// 4. Check for errors during reading
if err := scanner.Err(); err != nil {
fmt.Println("Error reading file:", err)
}
}This code maintains bounded memory usage relative to the total file size. However, Scanner defaults to a maximum token size of 64 KiB. For longer lines, we can increase the limit before starting:
scanner.Buffer(make([]byte, 64*1024), 1024*1024) // Up to 1 MiB per lineIf tokens can be very large or we need more control, a bufio.Reader is usually a better fit.
Controlled writing with os.OpenFile
os.WriteFile is convenient, but sometimes you need more control:
- Do you want to append text (Append) instead of overwriting?
- Do you want to create the file only if it doesn’t exist?
For that, we use os.OpenFile with Flags.
func main() {
// Flags:
// O_APPEND: Write at the end
// O_CREATE: Create if it doesn't exist
// O_WRONLY: Write-only
file, err := os.OpenFile("log.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
panic(err)
}
defer file.Close()
// Write to the open file
if _, err := file.WriteString("New log entry\n"); err != nil {
panic(err)
}
}The io.Reader and io.Writer interfaces
This pattern is one of the most beautiful ideas in the standard library.
The *os.File type satisfies the io.Reader and io.Writer interfaces.
This means any function that accepts a Reader (such as a JSON decoder, a GZIP compressor, or an HTTP response) also accepts a file.
Copying files with io.Copy
You don’t need a manual read/write loop to copy files.
import (
"errors"
"io"
"os"
)
func Copy(source string, destination string) (err error) {
// Open source
src, err := os.Open(source)
if err != nil { return err }
defer src.Close()
// Create destination
dst, err := os.Create(destination)
if err != nil { return err }
defer func() {
err = errors.Join(err, dst.Close())
}()
// Copy stream to stream
_, err = io.Copy(dst, src)
return err
}Example: Downloading a file from the internet
Since http.Response.Body is a Reader and the file is a Writer, we can connect them directly. We need to check the HTTP error, status code, and write errors.
func Download(url, destination string) (err error) {
client := http.Client{Timeout: 30 * time.Second}
resp, err := client.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("download %s: status %s", url, resp.Status)
}
file, err := os.Create(destination)
if err != nil {
return err
}
defer func() {
err = errors.Join(err, file.Close())
}()
_, err = io.Copy(file, resp.Body)
return err
}Path handling with path/filepath
Avoid concatenating paths with strings ("folder/" + "file.txt"). filepath.Join applies the path rules of the operating system and cleans up separators.
Always use the path/filepath package.
import "path/filepath"
func main() {
folder := "data"
file := "config.json"
// Build the correct path according to the OS
fullPath := filepath.Join(folder, file)
fmt.Println(fullPath)
// Windows: data\config.json
// Linux: data/config.json
}For URL or import paths, which always use /, use the path package, not path/filepath.
