Avoid hitting API rate limits using a reverse proxy server

When I was in college, I liked gin. Mostly because it came in a beautiful, square blue bottle. Today, I like gin, because it’s the name of a beautifully simple web framework.

In this post, I will demonstrate how to create a reverse proxy server using the Gin web framework.

This will allow us to request our own server as much as we want, while requesting an external server at a rate lower than the API rate limit!

Wiz Khalifa sporting a bottle of Bombay Saphhire gin

Gin gonic is a fast web framework that features a simple interface that is easy to build upon.

Gin is written in Go, a language that is gaining popularity due to its simplicity, speed, and robustness.

Setup

First, you’ll need Go, and a good IDE. I think Visual Studio Code is currently the best IDE, and it works on all major platforms.

Next, initialize the project with the following commands. I’m using github.com/nathankjer/gin-proxy as my project path but you should use your own.

go mod init
go mod tidy

Creating the proxy server

Now let’s create a proxy server. Paste the code below (source: S├ębastien Le Gall) into a file located at /proxy/proxy.go. I will be using Wikipedia as my example host, but you can use whatever you like.

package main

import (
	"fmt"
	"net/http"
	"net/http/httputil"
	"net/url"

	"github.com/gin-gonic/gin"
)

func proxy(c *gin.Context) {
	remote, err := url.Parse("https://en.wikipedia.org")
	if err != nil {
		fmt.Println(err)
	}
	proxy := httputil.NewSingleHostReverseProxy(remote)
	proxy.Director = func(req *http.Request) {
		req.Header = c.Request.Header
		req.Host = remote.Host
		req.URL.Scheme = remote.Scheme
		req.URL.Host = remote.Host
		req.URL.Path = c.Param("path")
		req.URL.RawQuery = c.Request.URL.RawQuery
	}
	proxy.ServeHTTP(c.Writer, c.Request)
}

func main() {
	r := gin.Default()
	r.Any("/*path", proxy)
	r.Run(":3001")
}

Let’s create a client that will use our proxy server. Paste the following within /client/client.go:

package main

import (
	"fmt"
	"io"
	"net/http"
)

func main() {
	res, err := http.Get("http://localhost:3001/w/api.php?action=parse&page=Proxy_server&format=json")
	if err != nil {
		fmt.Println(err.Error())
	}
	responseBody, err := io.ReadAll(res.Body)
	if err != nil {
		fmt.Println(err.Error())
	}
	fmt.Println(string(responseBody))
}

Now, run the proxy server with the following command. This will start our proxy server, which will be listening for requests on port 3001.

go run proxy/proxy.go

Next, run the following in a separate terminal window/tab:

go run client/client.go

You should get the following output:

There should be a 200 response on the proxy server:

Rate limiting

We can utilize gin’s built-in rate limiter by running the proxy server like so:

go run proxy/proxy.go -rps=100

This utilizes the leaky bucket method, ensuring that requests are made on average every 10 ms.

Creating a request server

Let’s add a request server that will forward our requests to the proxy server, or return a cached response for previously seen requests. This will allow us to never ask for the same information twice, which comes with the added benefit of 100 times faster response times.

Paste the following within server/server.go:

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"time"

	"crypto/md5"

	"github.com/gin-gonic/gin"
	"github.com/nathankjer/gin-proxy/db"
	"github.com/nathankjer/gin-proxy/models"
)

func request(c *gin.Context) {
	header, _ := json.Marshal(c.Request.Header)
	body, _ := io.ReadAll(c.Request.Body)
	request_id := fmt.Sprintf("%x", md5.Sum([]byte(c.Request.Method+c.Request.Host+c.Request.URL.String()+string(header)+string(body))))

	var request models.Request
	if err := db.Database.Where("id = ?", request_id).First(&request).Error; err != nil {

		// Perform request using proxy
		req, err := http.NewRequest(c.Request.Method, "http://localhost:3001"+c.Request.URL.Path+"?"+c.Request.URL.RawQuery, c.Request.Body)
		if err != nil {
			fmt.Println(err)
		}
		res, err := http.DefaultClient.Do(req)
		if err != nil {
			fmt.Println(err)
		}
		responseBody, err := io.ReadAll(res.Body)
		if err != nil {
			fmt.Println(err)
		}

		// Save request
		request = models.Request{
			Id:             request_id,
			Method:         c.Request.Method,
			Host:           c.Request.Host,
			Path:           c.Request.URL.Path,
			ResponseStatus: res.StatusCode,
			ResponseBody:   responseBody,
			CreatedAt:      time.Now(),
		}
		db.Database.Create(&request)
		c.Data(res.StatusCode, "", responseBody)
	} else {
		c.Data(request.ResponseStatus, "", request.ResponseBody)
	}
}

func main() {
	db.Connect()
	r := gin.Default()
	r.Any("/*path", request)
	r.Run(":3000")
}

We need a model to represent requests. Paste the following within models/requests.go:

package models

import "time"

type Request struct {
	Id             string    `json:"id"`
	Method         string    `json:"method"`
	Host           string    `json:"host"`
	Path           string    `json:"path"`
	ResponseStatus int       `json:"response_status"`
	ResponseBody   []byte    `json:"response_body"`
	CreatedAt      time.Time `json:"expires_at"`
}

Next, let’s create an interface that will allow us to save requests to a PostgreSQL database. First, let’s create a database for our project:

echo "CREATE DATABASE ginproxy;" | psql --dbname postgres

Paste the following within db/db.go. Depending on your setup, you may need to add a password to the database config string:

package db

import (
	"log"
	"os"

	"github.com/nathankjer/gin-proxy/models"
	"gorm.io/driver/postgres"
	gorm "gorm.io/gorm"
)

var Database *gorm.DB

func Connect() {
	dbURL := "host=localhost user=postgres dbname=ginproxy port=5432"
	database, err := gorm.Open(postgres.Open(dbURL), &gorm.Config{})
	if err != nil {
		panic("Failed to connect to database!")
	}
	Migrate(database)
	Database = database
}
func Migrate(database *gorm.DB) {
	database.AutoMigrate(&models.Request{})
}

Modify the port within client.go to 3000. Run proxy.go, server.go, and re-run client.go. The response for proxy and client, should be the same as before.

On server.go, you should get the following:

Run client.go again. You should now see a second request on server.go. The second request responded 100 times faster than the original request! As expected, no request was made to the proxy server.

Making requests from the browser

Not only can we use the proxy for APIs, but we can also use it to make requests from the browser! The animation shown previously is a good illustration. This can be used for reviewing traffic, blocking ads, and injecting content/styles.

We can use the code below to replace all instances of paragraph tags with one that is styled with a pink text color.

c.Data(res.StatusCode, "", []byte(strings.ReplaceAll(string(responseBody), "<p>", "<p style=\"color:pink;\">")))

Future improvements

The server does not fully support headers or cookies, or other nuances such as the user-agent. I designed it primarily to make requests to APIs but these are features that might be implemented in the future.


About the author



Hi, I'm Nathan. Thanks for reading! Keep an eye out for more content being posted soon.


Leave a Reply

Your email address will not be published. Required fields are marked *