Kazuki Higashiguchi
Posted on December 16, 2021
Series introduction
In my previous post I talked about how to establish a WebSocket connection in Go.
In this post, I will be starting to talk about how to relay TCP connection from "App" to the peer of WebSocket in Go.
- Start a WebSocket server (Part 1)
- Establish a WebSocket connection (Part 2)
- Relay TCP connection from "App" to the peer of WebSocket (Part 3 | Part 4)
- Relay TCP connection in WebSocket data to "internal API"
- Keep a established connection
Reverse HTTP proxy over WebSocket
A reverse HTTP proxy over WebSocket is a type of proxies, which retrieves resources on behalf on a client from servers and uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.
I'll use root-gg/wsp as a sample code to explain it. I'll use the forked code to explain because maintenance has stopped and the Go language and libraries version needed to be updated.
Relay TCP connection to the peer of WebSocket
In the part 1, I presented a demonstration.
HTTP communication is relayed by the following route.
app -[1]-> wsp server -[2](WebSocket)-> wsp client -> internal API
As a pre requirement, a WebSocket connection has already been established between wsp server and wsp client before relaying.
The way to establish WebSocket connection is explained in part 2, but next we need to pool the connection on the server side and use it when relaying.
These flow are divided into three parts to explain it.
- Receive requests to be proxied (
[1]in the relay flow) - Pool the WebSocket connection on the server for relaying
- Relay TCP connection to the peer WebSocket (
[2]in the relay flow)
This post describes the 1st and 2nd flow. In part 4, I'll explain the 3rd flow.
1. Receive requests to be proxied
To receive requests to be proxied to "internal API" finally, there are mainly two kinds of API interfaces.
- Expose an endpoint (i.e.
/requests) that accept HTTP requests - Work as a HTTP proxy used by "app" (i.e.
curl -x {wsp server's address} http://localhost:8081)
In the demo example, wsp server (WebSocket server) chose 1st option and exposes the endpoint whose path is /requests.
$ curl -H 'X-PROXY-DESTINATION: http://localhost:8081/hello' http://127.0.0.1:8080/request
Let's read the HTTP handler code in Go (code), which waits the request to /requests/.
func (s *Server) request(w http.ResponseWriter, r *http.Request) {
// Parse destination URL
dstURL := r.Header.Get("X-PROXY-DESTINATION")
if dstURL == "" {
wsp.ProxyErrorf(w, "Missing X-PROXY-DESTINATION header")
return
}
URL, err := url.Parse(dstURL)
if err != nil {
wsp.ProxyErrorf(w, "Unable to parse X-PROXY-DESTINATION header")
return
}
r.URL = URL
// (omit)
}
wsp defined the custom HTTP header X-PROXY-DESTINATION to specify the final destination URL. For example, when you want to access http://localhost:8081/hello, add a HTTP header X-PROXY-DESTINATION: http://localhost:8081/hello in your request.
2. Pool the WebSocket connection on the server for relaying
We need to make sure that the http handler can recognize and use the connected WebSocket connections.
In this code, an HTTP server that exposes an endpoint /request is set up at a given address. The handler mapped to /request is as follow (code):
func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
// - 1. Upgrade a received HTTP request to a WebSocket connection
// (omit)
// - 2. Wait a greeting message from the peer and parse it
// (omit)
// 3. Register the connection into server pools.
// s.lock is for exclusive control of pools operation.
s.lock.Lock()
defer s.lock.Unlock()
var pool *Pool
// There is no need to create a new pool,
// if it is already registered in current pools.
for _, p := range s.pools {
if p.id == id {
pool = p
break
}
}
if pool == nil {
pool = NewPool(s, id)
s.pools = append(s.pools, pool)
}
// update pool size
pool.size = size
// Add the WebSocket connection to the pool
pool.Register(ws)
}
First, the server needs to have a WebSocket connection pooled so that it can be used for relay. One design idea to achieve this is to prepare three entities Server, Pool, and Connection as shown below.
In this code, the Server struct have a field pools which is a pointer of the slice of Pool struct.
type Server struct {
// (omit)
pools []*Pool
// (omit)
}
The Pool struct represents the connection from ws client. The definition is as follow:
type Pool struct {
server *Server
id PoolID
size int
connections []*Connection
idle chan *Connection
done bool
lock sync.RWMutex
}
The Connection struct manages a single WebSocket connection because wsp supports multiple connections from a single wsp client at the same time.
type Connection struct {
pool *Pool
ws *websocket.Conn
status ConnectionStatus
idleSince time.Time
lock sync.Mutex
nextResponse chan chan io.Reader
}
Then, going back to the handler code, we can see that the server checks whether requesting connection is already registered, and create a new pool if it's not registered in current pools.
func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
// (omit)
var pool *Pool
// There is no need to create a new pool,
// if it is already registered in current pools.
for _, p := range s.pools {
if p.id == id {
pool = p
break
}
}
if pool == nil {
pool = NewPool(s, id)
s.pools = append(s.pools, pool)
}
// (omit)
}
Each ws client issues an unique ID, which is a common design when you want to pool and recognize external devices. By issuing an ID, the server is able to recognize meta information such as ws client source IP and so on.
At the end of the handler function, the server registers a new connection to the created pool.
func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
// (omit)
pool.Register(ws)
}
Pool.Register is as follow:
func (pool *Pool) Register(ws *websocket.Conn) {
// (omit)
log.Printf("Registering new connection from %s", pool.id)
connection := NewConnection(pool, ws)
pool.connections = append(pool.connections, connection)
}
This handler process registered an idle connection that can be used for relay. NewConnection() function mark a new connection as a usable connection for relay in the pool and start a thread (technically, start a new goroutine running) that keeps reading messages over the connected WebSocket connection from the wsp client.
func NewConnection(pool *Pool, ws *websocket.Conn) *Connection {
// Initialize a new Connection
c := new(Connection)
c.pool = pool
c.ws = ws
c.nextResponse = make(chan chan io.Reader)
c.status = Idle
// Mark that this connection is ready to use for relay
c.Release()
// Start to listen to incoming messages over the WebSocket connection
go c.read()
return c
}
At initialization, a new Connection is created by state Idle.
Enum in Go
The status Idle means it is opened but not working now. Here is a state diagram describing the behavior of Connection.
In this repository, I made the defined type ConnectionStatus and constants Idle, Busy, and Closed.
// ConnectionStatus is an enumeration type which represents the status of WebSocket connection.
type ConnectionStatus int
const (
// Idle state means it is opened but not working now.
// The default value for Connection is Idle, so it is ok to use zero-value(int: 0) for Idle status.
Idle ConnectionStatus = iota
Busy
Closed
)
By the way, if you want not to define ConnectionStatus with Zero-Value, you can skip the zero-value as follows:
// Pattern1: Assign zero-value to blank
const (
_ State = iota
A = iota
B
)
// Pattern2: Start from 1
const (
C State = iota + 1
D
)
Conclusion
This post explained how to relay TCP connection from "App" to the peer of WebSocket, especially implementation to receive requests to be proxied and to pool the WebSocket connection on the server for relaying.
I will explain the rest points in part 3 and beyond.
- Relay TCP connection from "App" to the peer of WebSocket (rest)
- Relay TCP connection in WebSocket data to "internal API"
- Keep a established connection
Posted on December 16, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.