Post on 21-Jan-2018
transcript
Build Great Networked APIs with Swift, OpenAPI, and gRPCTim Burks, Google
Swift Cloud Workshop No. 2
September 30, 2017
What do we do?
WE WRITE CODE
What do we want?
QUALTIY!
Protocol Buffersa language-neutral, platform-neutral, extensible mechanism for serializing structured data.
Interface Builder for Data
message Person {
string name = 1;
int32 id = 2;
string email = 3;
message PhoneNumber {
string number = 1;
}
repeated PhoneNumber phone = 4;
}
Interface Builder: Developers specify their interfaces using a special tool, tooling compiles and integrates that into their apps.
Protocol Buffers: Developers specify their data structures using a special language, tooling compiles and integrates that into their apps.
Discovery Format Code Generators
github.com/google/google-api-python-client
Idiosyncracies and Lessons
Toolkit and the next generation of Google APIs
So far, so good.
REST (Discovery Format)
● 232 API descriptions at https://www.googleapis.com/discovery/v1/apis● 10 generators listed at https://developers.google.com/api-client-library/
gRPC (Toolkit)
● 30+ gRPC-based APIs● 7 target languages
OpenAPI 3.0
Industry standard format for describing for REST APIs
Originally designed for documentation, now with many other applications: API Authoring, Validation, Documentation, Analysis, Search, Testing, Mocking, Management, Code Generation
Consensus
OpenAPI-based code generators
● swagger-codegen (Open source, Smartbear + community)○ 70+ targets○ First commit: July 6, 2011○ Used by Lyft and Square to generate SDKs
● AutoRest (Open source, Microsoft)● oas-nodegen (Open source, Capital One)● APIMatic (Proprietary, APIMatic)
more?
Problems we’ve seen so far in open source code generators
● Missing or weak build system integration.● Invalid service addresses (error in OpenAPI description).● No auth.● No documentation (not even a README!).● No samples.● No test harness.● No assurance that the called service even works.● Ugly generated code.
○ Machine-generated operation names.○ Machine-generated struct names.
● Unwanted dependencies.
swagger-codegen often isn’t used “out of the box”:
“Generating client libraries involves customizing the provided language-specific templates…
The amount of modification each template needs varies by language and we’re looking forward to working with the Swagger Codegen community to share our refinements.” Val Polouchkine, Lyft
“...Swagger Codegen is a pretty active project. If you don’t check in your templates, things are gonna break unexpectedly because Swagger Codegen just uses the latest and greatest templates that are out there. So if you don’t sort of manually make sure that those things work, you’re gonna have an issue there.” Tristan Sokol, Square
Code generation pipeline
API Description GeneratableAPI Description
Language-specific model
GeneratedAPI support
code
render idiomatic API support code according
to user preferences (re. build systems,
dependencies)
verify model, define structures and
entry points,name everything
filter language-specific reserved words,
(optionally) define file structure for generated code
typical monolithiccode generation pipeline
API Description(OpenAPI)
GeneratableAPI Description(internal data
structures)
Language-specific model(internal data
structures)
GeneratedAPI support
code
All-in-one repo and package
Problems with monolithic code generators
● Long build times: changing one target requires rebuilding everything.● Long test times: new builds must be tested for every target language.● For stability, teams may prefer to archive their own generator builds.● Forks will abound.● Quality is uneven.● Versioning is hard.● Complexity and potentially unfamiliar build systems deter contributors.
protoccode generation pipeline
API Description(.proto)
GeneratableAPI Description
(binary FileDescriptors)
Language-specific model (internal data
structures)
GeneratedAPI support
code
language target-specific plugins
protoc
protoc plugin definition
Why does protoc have a plug-in architecture?
● Fast build times: changing one target only requires rebuilding its plugin.● Fast test times: new builds need only be tested for the affected targets.● For stability, teams can archive their own protoc and plugin builds.● New plugins can abound.● Separately-maintained plugins can offer different maturity levels.● Separately-maintained plugins can be appropriately versioned.● Separately-maintained plugins can be in languages that contributors
prefer.
What’s the catch?
Plugins require a well-defined interchange format.
Fortunately, we have two great tools for that.
gnostic
gnostic processed and verified protobuf representation of
OpenAPI description
protoc + pluginsOpenAPI
.proto
reusable data structures and reader for protobuf OpenAPI descriptions
gnostic apps and plugins
OpenAPIdescription
gnostic-generator
OpenAPI.proto and
compiler code
OpenAPIJSON
schema
Kubernetes OpenAPI: .json vs .pb
Format Size Deserialization time Download time (at 80 Mbps)
Json 1653 KB >500 ms 165.3 ms
Proto binary 914 KB 9.3 ms 91.4 ms
Proto binary compressed 96 KB 13.5 ms 1.3 ms
Source: mehdy@google.com
gnosticcode generation pipeline
API Description(OpenAPI v3)
GeneratableAPI Description
(Normalized/Annotated
OpenAPI v3?)
Language-specific model(Normalized/
Annotated OpenAPI v3?)
GeneratedAPI support
code
target-specificplugins
gnostic +linter
let anyone write code generation plugins...
● in their own repositories● with their own versioning● in whatever implementation language they choose
API code generation is a community problem that needs community-based solutions...
Code generators should belong to their communities.
gRPC
@grpcio
RPC: Use Cases
Direct RPCs:Microservices
RPCs to access APIs
Google APIs
OSS APIs
MobileWeb
DesktopRPCs
Datacenters Cloud
Service 1
Service 2
Service 3
Service 4Containers
@grpcio
What is gRPC?● HTTP/2 based RPC framework developed by Google● Open, Multiplatform, Secure, Performant
Multiplatform
● Idiomatic APIs in popular languages (C++, Go, Java, Python, Node.js, C#, Ruby, PHP)● Supports mobile devices (Android Java, iOS Obj-C, Swift)● Linux, Windows, Mac OS X● (web browser support in development)
OpenSource
● developed fully in open on GitHub: https://github.com/grpc/
@grpcio
● Builds on Apple’s swift-protobuf and grpc-core.● Includes:
○ gRPC framework (C and Swift components)○ generated code surface○ protoc plugin for code generation
● Full-service gRPC:○ All four gRPC API styles are supported.○ gRPC framework supports both clients and servers.○ Plugin generates client and server code in separate files.○ Testing on MacOS and Ubuntu.
● Audiences: client and server developers, inside/outside Google
gRPC for Swift (https://github.com/grpc/grpc-swift)
@grpcio
package echo;
service Echo {
// Immediately returns an echo of a request.
rpc Get(EchoRequest) returns (EchoResponse) {}
// Splits a request into words and returns each word in a stream of messages.
rpc Expand(EchoRequest) returns (stream EchoResponse) {}
// Collects a stream of messages and returns them concatenated when the caller closes.
rpc Collect(stream EchoRequest) returns (EchoResponse) {}
// Streams back messages as they are received in an input stream.
rpc Update(stream EchoRequest) returns (stream EchoResponse) {}
}
message EchoRequest {
// The text of a message to be echoed.
string text = 1;
}
message EchoResponse {
// The text of an echo response.
string text = 1;
}
gRPC Swift sample service
@grpcio
/// To build a server, implement a class that conforms to this protocol.
public protocol Echo_EchoProvider {
func get(request : Echo_EchoRequest, session : Echo_EchoGetSession) throws -> Echo_EchoResponse
func expand(request : Echo_EchoRequest, session : Echo_EchoExpandSession) throws
func collect(session : Echo_EchoCollectSession) throws
func update(session : Echo_EchoUpdateSession) throws
}
gRPC Swift server protocol
@grpcio
// get returns requests as they were received.
func get(request : Echo_EchoRequest, session : Echo_EchoGetSession) throws -> Echo_EchoResponse {
return Echo_EchoResponse(text:"Swift echo get: " + request.text)
}
...
// update streams back messages as they are received in an input stream.
func update(session : Echo_EchoUpdateSession) throws -> Void {
while true {
do {
let request = try session.Receive()
try session.Send(Echo_EchoResponse(text:"Swift echo update: \(request.text)"))
} catch Echo_EchoServerError.endOfStream {
break
}
}
try session.Close()
}
}
gRPC Swift server sample
@grpcio
gRPC Swift unary client sample // Unary
if client == "get" {
var requestMessage = Echo_EchoRequest(text:message)
let responseMessage = try service.get(requestMessage) // blocking
print("get received: " + responseMessage.text)
}
@grpcio
gRPC Swift bidirectional streaming client sample (1/2) // Bidirectional streaming
if client == "update" {
let sem = DispatchSemaphore(value: 0)
let updateCall = try service.update() // blocking
DispatchQueue.global().async {
while true {
do {
let responseMessage = try updateCall.Receive() // blocking
print("Received: \(responseMessage.text)")
} catch Echo_EchoClientError.endOfStream {
sem.signal()
break
}
}
}
...
@grpcio
gRPC Swift bidirectional streaming client sample (2/2)...
let parts = message.components(separatedBy:" ")
for part in parts {
let requestMessage = Echo_EchoRequest(text:part)
try updateCall.Send(requestMessage)
sleep(1)
}
try updateCall.CloseSend()
// Wait for the call to complete.
sem.wait()
}
gRPC-Swift TODO
● Build system integration○ Package Manager○ Cocoapods?○ Carthage?
● gRPC interoperability tests● Samples that wrap Google APIs
○ Google Cloud Speech API○ Google Datastore API
github.com/grpc/grpc-swift/issues
More gRPC Information
Website: http://grpc.io
Sources: https://github.com/grpc/grpc
Mailing list: https://groups.google.com/d/forum/grpc-io
Ecosystem: https://github.com/grpc-ecosystem
github.com/googleapis/gnosticplugins for OpenAPI-based code generation
github.com/grpc/grpc-swiftFast streaming APIs in Swift
github.com/google/auth-library-swiftOAuth support for Google Cloud
http://twitter.com/timburkstimburks@google.com