lua-http

daurnimator

October 2017

What is lua-http?

HTTP Library for Lua
Optionally asynchronous (including DNS lookups and TLS)
Supports HTTP(S) version 1.0, 1.1 and 2
Functionality for both client and server
Compatible with Lua 5.1, 5.2, 5.3 and LuaJIT

History

Started as experiment in writing a HTTP 2 client
To test the client, I needed a server...
Realised that lua doesn't have a good HTTP 1 client: I should support that too
... and I needed to test that too
Scope creeped to be library for all things HTTP for Lua

Other projects

Every application lua gets embedded in implements their own new (non-feature complete) network protocol libraries

prosody's net.http
openresty's lua-resty-http
nmap
weechat-lua
....

Goal

Be go-to http library for all Lua developers
Support all standardised HTTP features
Does not want to be everything
Library to build your own http server/framework

I encourage anyone to build their own web frameworks on top of lua-thtp
Even if you don't want to use lua-http's connection objects, consider using e.g. http.headers module

Compatibility

Works anywhere except windows
Due to use of cqueues, can work inside of other programs and not block their main loops

Semantics

Many HTTP2 semantics are used for HTTP1
- Method/path/etc. are request headers :method, :path
- Status code is just another response header :status
Use same classes for both client and server
All potentially blocking operations take a timeout argument
Return nil, err_msg, errno on failure

Avoided pitfalls

Multiple headers e.g. Set-Cookie
Headers cannot just be implemented as key/value pairs
- Order matters
- sometimes should concat together with whitespace seperator
Timeouts apply to total operation, not parts

e.g. if :go() follows a redirect, the timeout given is for the whole operation
Buffering vs Streams

HTTP servers may return an "infinite" body

request:go() returns a stream Use stream:get_body_as_string() to get as string
Header field formats
Partial failures
- failed partial reads are :unget back into socket for retry
Redirects

Why cqueues?

Nature of HTTP2 requires a scheduler
HTTP1 does too for pipelining
Lua has no native main loop
cqueues is a composable main loop
- Will work with other main loops without blocking them.

Works Sync or Async

When run outside of a cqueues managed coroutine, is blocking => great for quick scripts or writing utils
When run inside of a cqueues managed coroutine, will yield back to controller

Features

Good/correct SSL/TLS checking
- host and IP validation
- Will avoid banned ciphers
- Uses HSTS
- ALPN to try http2
- Check purpose
Of course this is all configurable; but sensible+secure defaults!
Proxies
- Will pick up from env vars
  
  Watches out for HTTPoxy
Follows redirects
Pipelining
Websocket library
Compat APIs to help migration

Client Example

local http_request = require "http.request"
local my_req = http_request.new_from_uri("http://httpbin.org/get")
local headers, stream = assert(my_req:go())
for field, value in headers:each() do
    print(field, value)
end
local body = assert(stream:get_body_as_string())
print(body)

Server Example

server_hello.lua

Maturity

The client module is good for use anywhere today.
The server module has a couple of issues left before it should be used on the wide-open internet.
e.g. Support for MAX_CONCURRENT_STREAMS is needed to prevent resource exhaustion for a http2 server.

Future work

Automatic cookie store
Connection pooling

Installing

Available on luarocks
Available as debian package in stretch
I maintain PKGBUILD in the ArchLinux AUR

Questions?

Homepage: https://github.com/daurnimator/lua-http/

Docs: https://daurnimator.github.io/lua-http/