From 3560afeea669f499bf66fbcd3284788c4a8868e9 Mon Sep 17 00:00:00 2001 From: Steve Ayers Date: Wed, 14 Dec 2022 12:59:24 -0500 Subject: [PATCH] Improve context in documentation (#307) This spruces up the documentation a bit by: - expanding on the main README to give more context on what Protobuf is and how they can be used - adds some additional examples - adds additional instructions as well as a description of the packages - adds badges for NPM packages and versions - adds table of contents where appropriate To view the main README: https://github.com/bufbuild/protobuf-es/tree/sayers/improve_context_in_docs --- .github/buf-logo.svg | 1 + README.md | 128 +++++++++--- docs/migrating.md | 58 ++++-- docs/runtime_api.md | 191 ++++++++++-------- packages/protobuf-example/README.md | 6 +- packages/protobuf-test/README.md | 8 +- packages/protobuf-test/extra/example.proto | 11 +- .../src/gen/js/extra/example_pb.d.ts | 43 ++-- .../src/gen/js/extra/example_pb.js | 15 +- .../src/gen/ts/extra/example_pb.ts | 60 ++++-- .../src/iterating-fields.test.ts | 39 +++- packages/protobuf-test/src/readme.test.ts | 100 +++++++++ packages/protobuf/README.md | 50 ++--- packages/protoc-gen-es/README.md | 33 +-- packages/protoplugin-example/README.md | 7 +- packages/protoplugin-test/README.md | 2 +- packages/protoplugin/README.md | 8 +- 17 files changed, 505 insertions(+), 255 deletions(-) create mode 100644 .github/buf-logo.svg create mode 100644 packages/protobuf-test/src/readme.test.ts diff --git a/.github/buf-logo.svg b/.github/buf-logo.svg new file mode 100644 index 000000000..73ed09565 --- /dev/null +++ b/.github/buf-logo.svg @@ -0,0 +1 @@ + diff --git a/README.md b/README.md index b28834db9..cf47f72bc 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,126 @@ -Protobuf-ES -=========== +![The Buf logo](./.github/buf-logo.svg) + +# Protobuf-ES + +[![License](https://img.shields.io/github/license/bufbuild/protobuf-es?color=blue)](./LICENSE) [![NPM Version](https://img.shields.io/npm/v/@bufbuild/protobuf/latest?color=green&label=%40bufbuild%2Fprotobuf)](https://www.npmjs.com/package/@bufbuild/protobuf) [![NPM Version](https://img.shields.io/npm/v/@bufbuild/protoplugin/latest?color=green&label=%40bufbuild%2Fprotoplugin)](https://www.npmjs.com/package/@bufbuild/protoplugin) [![NPM Version](https://img.shields.io/npm/v/@bufbuild/protoc-gen-es/latest?color=green&label=%40bufbuild%2Fprotoc-gen-es)](https://www.npmjs.com/package/@bufbuild/protoc-gen-es) A complete implementation of [Protocol Buffers](https://developers.google.com/protocol-buffers) in TypeScript, -suitable for web browsers and Node.js. +suitable for web browsers and Node.js, created by [Buf](https://buf.build). + +## What are Protocol Buffers? -For example, the following definition: +In a nutshell, Protocol Buffers have two main functions: -```protobuf -message Person { - string name = 1; - int32 id = 2; // Unique ID number for this person. - string email = 3; +- They are a language for writing schemas for your data. +- They define a binary format for serializing your data. + +These two independent traits functions work together to allow your project and everyone who interacts with it to define messages, fields, and service APIs in the exact same way. In a practical sense as it relates to **Protobuf-ES**, this means no more disparate JSON types all over the place. Instead, you define a common schema in a Protobuf file, such as: + +```proto +message User { + string first_name = 1; + string last_name = 2; + bool active = 3; + User manager = 4; + repeated string locations = 5; + map projects = 6; } ``` -Is compiled to an ECMAScript class that can be used like this: +And it is compiled to an ECMAScript class that can be used like this: ```typescript -let pete = new Person({ - name: "pete", - id: 123 +let user = new User({ + firstName: "Homer", + lastName: "Simpson", + active: true, + locations: ["Springfield"], + projects: { SPP: "Springfield Power Plant" }, + manager: { + firstName: "Montgomery", + lastName: "Burns", + }, }); -let bytes = pete.toBinary(); -pete = Person.fromBinary(bytes); -pete = Person.fromJsonString('{"name": "pete", "id": 123}'); +const bytes = user.toBinary(); +user = User.fromBinary(bytes); +user = User.fromJsonString('{"firstName": "Homer", "lastName": "Simpson"}'); ``` -To learn more, have a look at a complete [code example](https://github.com/bufbuild/protobuf-es/tree/main/packages/protobuf-example), -the documentation for the [generated code](https://github.com/bufbuild/protobuf-es/blob/main/docs/generated_code.md), -and the documentation for the [runtime API](https://github.com/bufbuild/protobuf-es/blob/main/docs/runtime_api.md). +The benefits can extend to any application that interacts with yours as well. This is because the Protobuf file above can be used to generate types in many languages. The added bonus is that no one has to write any boilerplate code to make this happen. [Code generators](https://www.npmjs.com/package/@bufbuild/protoc-gen-es) handle all of this for you. +Protocol Buffers also allow you to serialize this structured data. So, your application running in the browser can send a `User` object to a backend running an entirely different language, but using the exact same definition. Using an RPC framework like [Connect-Web](https://github.com/bufbuild/connect-web), your data is serialized into bytes on the wire and then deserialized at its destination using the defined schema. -### TypeScript +## Quickstart -The generated code is compatible with TypeScript **v4.1.2** or later, with the default compiler settings. +To get started generating code right away, first make sure you have [Buf](https://docs.buf.build/installation) installed. If desired, you can also use `protoc`. +1. Install the code generator and the runtime library: -### Packages + ```bash + npm install @bufbuild/protobuf @bufbuild/protoc-gen-es + ``` + +2. Create a `buf.gen.yaml` file that looks like this: + + ```yaml + # Learn more: https://docs.buf.build/configuration/v1/buf-gen-yaml + version: v1 + plugins: + - name: es + path: ./node_modules/.bin/protoc-gen-es + opt: target=ts + out: src/gen + ``` + +3. Download the [example.proto](https://github.com/bufbuild/protobuf-es/blob/main/packages/protobuf-test/extra/example.proto) into a `/proto` directory: + + ```bash + mkdir proto + curl https://raw.githubusercontent.com/bufbuild/protobuf-es/main/packages/protobuf-test/extra/example.proto > proto/example.proto + ``` + +4. Generate your code: + + ```bash + buf generate proto + ``` + +You should now see a generated file at `src/gen/example_pb.ts` that contains a class named `User`. From here, you can begin to work with your schema. + +## Packages -- [@bufbuild/protoc-gen-es](https://www.npmjs.com/package/@bufbuild/protoc-gen-es): - Provides the code generator plugin `protoc-gen-es` ([source](packages/protoc-gen-es)). - The code it generates depends on `@bufbuild/protobuf`. - [@bufbuild/protobuf](https://www.npmjs.com/package/@bufbuild/protobuf): - The runtime library for the code generator plugin `protoc-gen-es` ([source](packages/protobuf)). + Provides the runtime library, containing base types, generated well-known types, and core functionality. ([source code](packages/protobuf)). +- [@bufbuild/protoc-gen-es](https://www.npmjs.com/package/@bufbuild/protoc-gen-es): + Provides the code generator plugin `protoc-gen-es`. The code it generates depends on `@bufbuild/protobuf`. ([source code](packages/protoc-gen-es)). - [@bufbuild/protoplugin](https://www.npmjs.com/package/@bufbuild/protoplugin): - Helps to create your own code generator plugin ([source](packages/protoplugin)). + Helps to create your own code generator plugin. The code it generates depends on `@bufbuild/protobuf`. ([source code](packages/protoplugin)). + +## Documentation -### FAQ +* [Code example](packages/protobuf-example) - Example code that uses protocol buffers to manage an address book. +* [Generated Code](docs/generated_code.md) - How to generate, and what code precisely is generated for any given protobuf definition. +* [Runtime API](docs/runtime_api.md) - A detailed overview of the features provided by the library `@bufbuild/protobuf`. +* [FAQ](docs/faq.md) - Frequently asked Questions. +* [Migrating to Protobuf-ES](docs/migrating.md) - Shows the changes you'll need to switch your existing code base. +* [Writing Plugins](docs/writing_plugins.md) - An overview of the process of writing a plugin using `@bufbuild/protoplugin`. -For a list of frequently asked questions, see our [FAQ documentation](docs/faq.md). +## Ecosystem + +* [connect-web](https://github.com/bufbuild/connect-web): + TypeScript clients for web browsers, based on Protobuf-ES. +* [connect-web-integration](https://github.com/bufbuild/connect-web-integration): + Example projects using Connect-Web with various JS frameworks and tooling +* [Buf Studio](https://studio.buf.build/): Web UI for ad-hoc RPCs + + +## TypeScript + +The generated code is compatible with TypeScript **v4.1.2** or later, with the default compiler settings. -### Copyright +## Copyright The [code to encode and decode varint](packages/protobuf/src/google/varint.ts) is Copyright 2008 Google Inc., licensed under BSD-3-Clause. diff --git a/docs/migrating.md b/docs/migrating.md index 5f16a3e6c..122cc638d 100644 --- a/docs/migrating.md +++ b/docs/migrating.md @@ -5,23 +5,51 @@ The following guides show the changes you'll need to switch your existing code b [from `protobuf-javascript`](#from-protobuf-javascript) or [from `protobuf-ts`](#from-protobuf-ts) to Protobuf-ES. +- [Feature Matrix](#feature-matrix) +- [From protobuf-javascript](#from-protobuf-javascript) + - [Generating Code](#generating-code) + - [Field Access](#field-access) + - [Optional Fields](#optional-fields) + - [Well-Known Types](#well-known-types) + - [Wrapper Fields](#wrapper-fields) + - [Map Fields](#map-fields) + - [Repeated Fields](#repeated-fields) + - [Oneof Groups](#oneof-groups) + - [Message Constructors](#message-constructors) + - [Serialization](#serialization) + - [Enumerations](#enumerations) + - [`toObject()`](#toobject) +- [From protobuf-ts](#from-protobuf-ts) + - [Generating Code](#generating-code-1) + - [Well-Known Types](#well-known-types-1) + - [Wrapper Fields](#wrapper-fields-1) + - [Serialization](#serialization-1) + - [Message Constructors](#message-constructors-1) + - [Cloning](#cloning) + - [Message Type Guards](#message-type-guards) + - [Reflection](#reflection) + - [Dynamic Messages](#dynamic-messages) + + +# Feature Matrix + | Feature | Protobuf-ES | protobuf-javascript | protobuf-ts | |------------------------|-------------|---------------------|-------------| -| Initializers | ✔️ | | ✔️️ | -| Plain properties | ✔️ | | ✔️️ | -| `instanceof` | ✔️ | ✔️ | ️️ | -| JSON format | ✔️ | | ✔️ | -| Binary format | ✔️ | ✔️ | ✔️ | -| TypeScript | ✔️ | ️ | ✔️ | -| Standard module system | ✔️ | | ✔️ | -| Tree shaking | ✔️ | ️ | ✔️ | -| Reflection | ✔️ | | ✔️ | -| Dynamic messages | ✔️ | ️ | ✔️ | -| Wrappers unboxing | ✔️ | | | -| Comments | ✔️ | | ✔️ | -| Deprecation | ✔️ | | ✔️ | -| proto2 syntax | ✔️ | ✔️ | | -| proto2 extensions | ️ | ✔️ | | +| Initializers | ✅ | ❌ | ✅ | +| Plain properties | ✅ | ❌ | ✅ | +| `instanceof` | ✅ | ✅ | ️️❌ | +| JSON format | ✅ | ❌ | ✅ | +| Binary format | ✅ | ✅ | ✅ | +| TypeScript | ✅ | ️❌ | ✅ | +| Standard module system | ✅ | ❌ | ✅ | +| Tree shaking | ✅ | ️❌ | ✅ | +| Reflection | ✅ | ❌ | ✅ | +| Dynamic messages | ✅ | ️❌ | ✅ | +| Wrappers unboxing | ✅ | ❌ | ❌ | +| Comments | ✅ | ❌ | ✅ | +| Deprecation | ✅ | ❌ | ✅ | +| proto2 syntax | ✅ | ✅ | ❌ | +| proto2 extensions | ️❌ | ✅ | ❌ | # From protobuf-javascript diff --git a/docs/runtime_api.md b/docs/runtime_api.md index 10cfa74b2..d3946ea89 100644 --- a/docs/runtime_api.md +++ b/docs/runtime_api.md @@ -35,10 +35,13 @@ For the following examples, we will use the following message definition [exampl syntax="proto3"; package docs; -message Example { - string foo = 1; - bool bar = 2; - Example baz = 3; +message User { + string first_name = 1; + string last_name = 2; + bool active = 3; + User manager = 4; + repeated string locations = 5; + map projects = 6; } ``` @@ -47,18 +50,18 @@ message Example { You can create an instance with the `new` keyword: ```typescript -const message = new Example(); +const user = new User(); ``` For convenience, constructors accept an initializer object: ```typescript -new Example({ - foo: "hello", - bar: true, - baz: { // you can simply pass an initializer object for this message field - foo: "world", +new User({ + firstName: "Homer", + active: true, + manager: { // you can simply pass an initializer object for this message field + lastName: "Burns", }, }); ``` @@ -74,9 +77,9 @@ with a default value are class properties with a default value: ```typescript /** - * @generated from field: string foo = 1; + * @generated from field: string firstName = 1; */ -foo = ""; +firstName = ""; ``` Protobuf fields map to default values as follows: @@ -99,17 +102,17 @@ Fields translate to plain properties. You can set and get field values with simp access: ```javascript -message.foo = "hello"; -message.baz = new Example(); +user.firstName = "Homer"; +user.manager = new User(); -message.foo; // "hello" -message.baz?.bar; // false +user.firstName; // "Homer" +user.manager?.active; // false ``` You can also use a destructuring assignment: ````javascript -let {foo, bar} = message; +let {firstName, lastName} = user; ```` @@ -122,10 +125,13 @@ grouped into an object property. With the following oneof group: ```diff -message Example { - string foo = 1; - bool bar = 2; - Example baz = 3; +message User { + string first_name = 1; + string last_name = 2; + bool active = 3; + User manager = 4; + repeated string locations = 5; + map projects = 6; + oneof result { + int32 number = 4; + string error = 5; @@ -157,27 +163,27 @@ result: To select a field, simply replace the `result` object: ```typescript -message.result = {case: "number", value: 123}; -message.result = {case: undefined}; +user.result = {case: "number", value: 123}; +user.result = {case: undefined}; ``` To query a oneof group, you can use if-blocks: ```typescript -if (message.result.case === "number") { - message.result.value; // a number +if (user.result.case === "number") { + user.result.value; // a number } ``` Or a switch statement: ```typescript -switch (message.result.case) { +switch (user.result.case) { case "number": - message.result.value; // a number + user.result.value; // a number break; case "error": - message.result.value; // a string + user.result.value; // a string break; } ``` @@ -195,7 +201,7 @@ While a shallow copy of a message can be created by using the spread operator wi message constructor, it is also possible to create a _deep_ clone of a message: ```typescript -example.clone(); +user.clone(); ``` ### Comparing messages @@ -204,14 +210,14 @@ We provide instance methods as well as static methods to test if two messages of are equal: ```typescript -example.equals(example); // true -example.equals(null); // false -Example.equals(example, example); // true -Example.equals(example, null); // false +user.equals(user); // true +user.equals(null); // false +User.equals(user, user); // true +User.equals(user, null); // false ``` ```typescript -Example.typeName; // docs.Example +User.typeName; // docs.User ``` ### Serializing messages @@ -219,15 +225,15 @@ Example.typeName; // docs.Example Serializing to the binary format is very straight-forward: ```typescript -const bytes: Uint8Array = example.toBinary(); -Example.fromBinary(bytes); +const bytes: Uint8Array = user.toBinary(); +User.fromBinary(bytes); ``` Serializing to the JSON format is equally simple: ```typescript -const json = example.toJson(); -Example.fromJson(json); +const json = user.toJson(); +User.fromJson(json); ``` But the result will be a [JSON value][src-json-value] – a primitive JavaScript that can @@ -236,8 +242,8 @@ convenience, we also provide methods that include the stringify step: ```typescript -const json = example.toJsonString(); -Example.fromJsonString(json); +const json = user.toJsonString(); +User.fromJsonString(json); ``` Note that the JSON format is great for debugging, but the binary format is more resilient @@ -340,15 +346,15 @@ import { Any } from "@bufbuild/protobuf"; import { Timestamp } from "@bufbuild/protobuf"; // Pack a message: -let any = Any.pack(message); -any.typeUrl; // type.googleapis.com/docs.Example +let any = Any.pack(user); +any.typeUrl; // type.googleapis.com/docs.User // Check what an Any contains: -any.is(Example); // true +any.is(User); // true any.is(Timestamp); // false // Unpack an Any by providing a blank instance: -message = new Example(); +message = new User(); any.unpackTo(message); // true let ts = new Timestamp(); @@ -395,10 +401,13 @@ We use the `bigint` primitive to represent 64-bit integral types, because JavaSc For the following field definitions: ```diff -message Example { - string foo = 1; - bool bar = 2; - Example baz = 3; +message User { + string first_name = 1; + string last_name = 2; + bool active = 3; + User manager = 4; + repeated string locations = 5; + map projects = 6; + uint64 ulong = 4; + int64 long = 5; } @@ -407,11 +416,11 @@ message Example { You can use `bigint` as expected: ```typescript -example.ulong = 123n; -example.long = -123n; +user.ulong = 123n; +user.long = -123n; -example.ulong + 1n; // 124n -example.long + 1n; // -122n +user.ulong + 1n; // 124n +user.long + 1n; // -122n ``` ### `bigint` in unsupported environments @@ -431,8 +440,8 @@ import { protoInt64 } from "@bufbuild/protobuf"; let input: string | number | bigint = "123"; -example.long = protoInt64.parse(input); -example.ulong = protoInt64.uParse(input); +user.long = protoInt64.parse(input); +user.ulong = protoInt64.uParse(input); ``` If you want to perform arithmetic on `bigint` fields, you will need to use a @@ -451,14 +460,17 @@ Such a type can actually be created at run time. We can take a peek at the [gene code](../packages/protobuf-test/src/gen/ts/extra/example_pb.ts) to get some insights: ```typescript -class Example extends Message { +class User extends Message { //... static readonly runtime = proto3; - static readonly typeName = "docs.Example"; + static readonly typeName = "docs.User"; static readonly fields: FieldList = proto3.util.newFieldList(() => [ - { no: 1, name: "foo", kind: "scalar", T: 9 /* ScalarType.STRING */ }, - { no: 2, name: "bar", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, - { no: 3, name: "baz", kind: "message", T: Example }, + { no: 1, name: "first_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 2, name: "last_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 3, name: "active", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, + { no: 4, name: "manager", kind: "message", T: User }, + { no: 5, name: "locations", kind: "scalar", T: 9 /* ScalarType.STRING */, repeated: true }, + { no: 6, name: "projects", kind: "map", K: 9 /* ScalarType.STRING */, V: {kind: "scalar", T: 9 /* ScalarType.STRING */} }, ]); ``` @@ -470,17 +482,20 @@ We can observe three properties here: This is actually all the information we need to re-create this message type at run time: ```typescript -const Example = proto3.makeMessageType( - "docs.Example", +const User = proto3.makeMessageType( + "docs.User", () => [ - { no: 1, name: "foo", kind: "scalar", T: 9 /* ScalarType.STRING */ }, - { no: 2, name: "bar", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, - { no: 3, name: "baz", kind: "message", T: Example }, + { no: 1, name: "first_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 2, name: "last_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 3, name: "active", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, + { no: 4, name: "manager", kind: "message", T: User }, + { no: 5, name: "locations", kind: "scalar", T: 9 /* ScalarType.STRING */, repeated: true }, + { no: 6, name: "projects", kind: "map", K: 9 /* ScalarType.STRING */, V: {kind: "scalar", T: 9 /* ScalarType.STRING */} }, ], ); ``` -The resulting `Example` is completely equivalent to the generated TypeScript class. In fact, +The resulting `User` is completely equivalent to the generated TypeScript class. In fact, this exact piece of code is generated with the plugin option `target=js`, because if saves us quite a bit of code size. @@ -508,7 +523,7 @@ compiler: const registry = createRegistryFromDescriptors( readFileSync("image.bin") ); -const Example = registry.findMessage("doc.Example"); +const User = registry.findMessage("doc.User"); ``` ### Iterating over message fields @@ -523,10 +538,13 @@ function walkFields(message: AnyMessage) { } } -walkFields(message); -// field foo: abc -// field bar: true -// field baz: undefined +walkFields(user); +// field firstName: Homer +// field lastName: Simpson +// field active: true +// field manager: undefined +// field locations: SPRINGFIELD +// field projects: {"SPP":"Springfield Power Plant"} ``` Note that the example does not handle oneof groups. Please consult the sources code @@ -548,28 +566,28 @@ users provide message data, consider accepting `PartialMessage`, so that user simply give an object literal with only the non-default values they want. Note that any `T` is assignable to `PartialMessage`. -For example, let's say you have a protobuf `message Foo`, and you want to provide a +For example, let's say you have a protobuf `message User`, and you want to provide a function to your users that processes this message: ```ts -export function sendExample(example: PartialMessage) { +export function sendUser(user: PartialMessage) { // convert partial messages into their full representation if necessary - const e = example instanceof Example ? example : new Example(example); + const u = user instanceof User ? user : new User(user); // process further... - const bytes = e.toBinary(); + const bytes = u.toBinary(); } ``` All three examples below are valid input for your function: ```ts -sendExample({foo: "abc"}); +sendUser({firstName: "Homer"}); -const e = new Example(); -e.foo = "abc"; -sendExample(e); +const u = new User(); +u.firstName = "Homer"; +sendUser(u); -sendExample(new Example()); +sendUser(new User()); ``` @@ -582,10 +600,13 @@ In contrast to `PartialMessage`, `PlainMessage` requires all properties to be provided. For example: ```typescript -let plain: PlainMessage = { - foo: "abc", - bar: false, - baz: undefined, +let plain: PlainMessage = { + firstName: "Homer", + lastName: "Simpson", + active: true, + manager: undefined, + locations: [], + projects: {} }; ``` @@ -602,8 +623,8 @@ If you want to handle messages of unknown type, the type [`AnyMessage`][src-any- provides a convenient index signature to access fields: ```typescript -const anyMessage: AnyMessage = example; -example["foo"]; +const anyMessage: AnyMessage = user; +user["firstName"]; ``` Note that any message is assignable to `AnyMessage`. diff --git a/packages/protobuf-example/README.md b/packages/protobuf-example/README.md index dc3a29eab..eed01f8c2 100644 --- a/packages/protobuf-example/README.md +++ b/packages/protobuf-example/README.md @@ -1,6 +1,6 @@ -# Code example +# Protobuf Example -This directory contains example code that uses protocol buffers to manage an +This directory contains example code that uses Protocol Buffers to manage an address book. The script [add-person.ts](./src/add-person.ts) adds a new person to an address book, prompting the user to input the person's information. The script [list-people.ts](./src/list-people.ts) lists people already in the @@ -59,6 +59,6 @@ protoc -I . --es_out=src/gen --es_opt=target=ts --plugin=protoc-gen-es=./node_mo ``` Don't forget to run `npm run build` to compile TypeScript to JavaScript, so that -Node will understand it. You do not need TypeScript to use `protobuf-es`. Just +Node will understand it. You do not need TypeScript to use **Protobuf-ES**. Just set the plugin option `target=js` if you prefer plain JavaScript, or `target=js+dts` if you prefer JavaScript with TypeScript declaration files. diff --git a/packages/protobuf-test/README.md b/packages/protobuf-test/README.md index e71fc6d09..ab536d299 100644 --- a/packages/protobuf-test/README.md +++ b/packages/protobuf-test/README.md @@ -1,9 +1,9 @@ # Tests -This package provides test coverage for @bufbuild/protobuf with Jest. +This package provides test coverage for `@bufbuild/protobuf` with Jest. We also generate code for many of the unit test proto files that are part of -github.com/protocolbuffers/protobuf. They cover many edge cases for both code +`github.com/protocolbuffers/protobuf`. They cover many edge cases for both code generation and serialization. Many test cases are run several times, once with the generated TypeScript code, @@ -24,9 +24,7 @@ If any compilation error is encountered, the process will exit immediately. The versions list is compiled of: - the earliest TypeScript version we support (4.1.2). -- the earliest version we support with `skipLibChecks` set to `false`. -- the current release candidate -- the latest patch release of all minor versions in between +- the latest patch release of all minor versions up to the current release. Our compatibility tests use the default settings for `tsconfig.json` that are generated by running `tsc --init` (with a few minor exceptions). As a result, diff --git a/packages/protobuf-test/extra/example.proto b/packages/protobuf-test/extra/example.proto index e02774698..637ca2cd8 100644 --- a/packages/protobuf-test/extra/example.proto +++ b/packages/protobuf-test/extra/example.proto @@ -15,8 +15,11 @@ syntax = "proto3"; package docs; -message Example { - string foo = 1; - bool bar = 2; - Example baz = 3; +message User { + string first_name = 1; + string last_name = 2; + bool active = 3; + User manager = 4; + repeated string locations = 5; + map projects = 6; } diff --git a/packages/protobuf-test/src/gen/js/extra/example_pb.d.ts b/packages/protobuf-test/src/gen/js/extra/example_pb.d.ts index 892c0c750..ead3f1639 100644 --- a/packages/protobuf-test/src/gen/js/extra/example_pb.d.ts +++ b/packages/protobuf-test/src/gen/js/extra/example_pb.d.ts @@ -20,36 +20,51 @@ import type { BinaryReadOptions, FieldList, JsonReadOptions, JsonValue, PartialM import { Message, proto3 } from "@bufbuild/protobuf"; /** - * @generated from message docs.Example + * @generated from message docs.User */ -export declare class Example extends Message { +export declare class User extends Message { /** - * @generated from field: string foo = 1; + * @generated from field: string first_name = 1; */ - foo: string; + firstName: string; /** - * @generated from field: bool bar = 2; + * @generated from field: string last_name = 2; */ - bar: boolean; + lastName: string; /** - * @generated from field: docs.Example baz = 3; + * @generated from field: bool active = 3; */ - baz?: Example; + active: boolean; - constructor(data?: PartialMessage); + /** + * @generated from field: docs.User manager = 4; + */ + manager?: User; + + /** + * @generated from field: repeated string locations = 5; + */ + locations: string[]; + + /** + * @generated from field: map projects = 6; + */ + projects: { [key: string]: string }; + + constructor(data?: PartialMessage); static readonly runtime: typeof proto3; - static readonly typeName = "docs.Example"; + static readonly typeName = "docs.User"; static readonly fields: FieldList; - static fromBinary(bytes: Uint8Array, options?: Partial): Example; + static fromBinary(bytes: Uint8Array, options?: Partial): User; - static fromJson(jsonValue: JsonValue, options?: Partial): Example; + static fromJson(jsonValue: JsonValue, options?: Partial): User; - static fromJsonString(jsonString: string, options?: Partial): Example; + static fromJsonString(jsonString: string, options?: Partial): User; - static equals(a: Example | PlainMessage | undefined, b: Example | PlainMessage | undefined): boolean; + static equals(a: User | PlainMessage | undefined, b: User | PlainMessage | undefined): boolean; } diff --git a/packages/protobuf-test/src/gen/js/extra/example_pb.js b/packages/protobuf-test/src/gen/js/extra/example_pb.js index 836cec52e..bb3e4da3d 100644 --- a/packages/protobuf-test/src/gen/js/extra/example_pb.js +++ b/packages/protobuf-test/src/gen/js/extra/example_pb.js @@ -19,14 +19,17 @@ import { proto3 } from "@bufbuild/protobuf"; /** - * @generated from message docs.Example + * @generated from message docs.User */ -export const Example = proto3.makeMessageType( - "docs.Example", +export const User = proto3.makeMessageType( + "docs.User", () => [ - { no: 1, name: "foo", kind: "scalar", T: 9 /* ScalarType.STRING */ }, - { no: 2, name: "bar", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, - { no: 3, name: "baz", kind: "message", T: Example }, + { no: 1, name: "first_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 2, name: "last_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 3, name: "active", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, + { no: 4, name: "manager", kind: "message", T: User }, + { no: 5, name: "locations", kind: "scalar", T: 9 /* ScalarType.STRING */, repeated: true }, + { no: 6, name: "projects", kind: "map", K: 9 /* ScalarType.STRING */, V: {kind: "scalar", T: 9 /* ScalarType.STRING */} }, ], ); diff --git a/packages/protobuf-test/src/gen/ts/extra/example_pb.ts b/packages/protobuf-test/src/gen/ts/extra/example_pb.ts index 69fa1fb3b..4cdd218e7 100644 --- a/packages/protobuf-test/src/gen/ts/extra/example_pb.ts +++ b/packages/protobuf-test/src/gen/ts/extra/example_pb.ts @@ -20,51 +20,69 @@ import type { BinaryReadOptions, FieldList, JsonReadOptions, JsonValue, PartialM import { Message, proto3 } from "@bufbuild/protobuf"; /** - * @generated from message docs.Example + * @generated from message docs.User */ -export class Example extends Message { +export class User extends Message { /** - * @generated from field: string foo = 1; + * @generated from field: string first_name = 1; */ - foo = ""; + firstName = ""; /** - * @generated from field: bool bar = 2; + * @generated from field: string last_name = 2; */ - bar = false; + lastName = ""; /** - * @generated from field: docs.Example baz = 3; + * @generated from field: bool active = 3; */ - baz?: Example; + active = false; - constructor(data?: PartialMessage) { + /** + * @generated from field: docs.User manager = 4; + */ + manager?: User; + + /** + * @generated from field: repeated string locations = 5; + */ + locations: string[] = []; + + /** + * @generated from field: map projects = 6; + */ + projects: { [key: string]: string } = {}; + + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); } static readonly runtime = proto3; - static readonly typeName = "docs.Example"; + static readonly typeName = "docs.User"; static readonly fields: FieldList = proto3.util.newFieldList(() => [ - { no: 1, name: "foo", kind: "scalar", T: 9 /* ScalarType.STRING */ }, - { no: 2, name: "bar", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, - { no: 3, name: "baz", kind: "message", T: Example }, + { no: 1, name: "first_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 2, name: "last_name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 3, name: "active", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, + { no: 4, name: "manager", kind: "message", T: User }, + { no: 5, name: "locations", kind: "scalar", T: 9 /* ScalarType.STRING */, repeated: true }, + { no: 6, name: "projects", kind: "map", K: 9 /* ScalarType.STRING */, V: {kind: "scalar", T: 9 /* ScalarType.STRING */} }, ]); - static fromBinary(bytes: Uint8Array, options?: Partial): Example { - return new Example().fromBinary(bytes, options); + static fromBinary(bytes: Uint8Array, options?: Partial): User { + return new User().fromBinary(bytes, options); } - static fromJson(jsonValue: JsonValue, options?: Partial): Example { - return new Example().fromJson(jsonValue, options); + static fromJson(jsonValue: JsonValue, options?: Partial): User { + return new User().fromJson(jsonValue, options); } - static fromJsonString(jsonString: string, options?: Partial): Example { - return new Example().fromJsonString(jsonString, options); + static fromJsonString(jsonString: string, options?: Partial): User { + return new User().fromJsonString(jsonString, options); } - static equals(a: Example | PlainMessage | undefined, b: Example | PlainMessage | undefined): boolean { - return proto3.util.equals(Example, a, b); + static equals(a: User | PlainMessage | undefined, b: User | PlainMessage | undefined): boolean { + return proto3.util.equals(User, a, b); } } diff --git a/packages/protobuf-test/src/iterating-fields.test.ts b/packages/protobuf-test/src/iterating-fields.test.ts index a77024bf2..6bbdd508d 100644 --- a/packages/protobuf-test/src/iterating-fields.test.ts +++ b/packages/protobuf-test/src/iterating-fields.test.ts @@ -14,30 +14,49 @@ import { describe, expect, test } from "@jest/globals"; import type { AnyMessage } from "@bufbuild/protobuf"; -import { Example } from "./gen/ts/extra/example_pb.js"; +import { User } from "./gen/ts/extra/example_pb.js"; /* eslint-disable @typescript-eslint/no-unsafe-member-access,@typescript-eslint/no-unsafe-assignment,@typescript-eslint/restrict-template-expressions */ describe("iterating fields", function () { test("works as expected", function () { const r = walkFields( - new Example({ - foo: "abc", - bar: true, - baz: undefined, + new User({ + firstName: "John", + lastName: "Smith", + active: true, + manager: { + firstName: "Jane", + lastName: "Jones", + }, + locations: ["PIT", "GER"], + projects: { + PES: "Protobuf-ES", + CWB: "Connect-Web", + }, }) ); - expect(r.length).toBe(3); - expect(r[0]).toBe("field foo: abc"); - expect(r[1]).toBe("field bar: true"); - expect(r[2]).toBe("field baz: undefined"); + expect(r.length).toBe(6); + expect(r[0]).toBe("field firstName: John"); + expect(r[1]).toBe("field lastName: Smith"); + expect(r[2]).toBe("field active: true"); + expect(r[3]).toBe( + 'field manager: {"firstName":"Jane","lastName":"Jones","active":false,"locations":[],"projects":{}}' + ); + expect(r[4]).toBe("field locations: PIT,GER"); + expect(r[5]).toBe( + 'field projects: {"PES":"Protobuf-ES","CWB":"Connect-Web"}' + ); }); }); function walkFields(message: AnyMessage): string[] { const r: string[] = []; for (const fieldInfo of message.getType().fields.byNumber()) { - const value = message[fieldInfo.localName]; + let value = message[fieldInfo.localName]; + if (fieldInfo.kind === "message" || fieldInfo.kind === "map") { + value = JSON.stringify(value); + } r.push(`field ${fieldInfo.localName}: ${value}`); } return r; diff --git a/packages/protobuf-test/src/readme.test.ts b/packages/protobuf-test/src/readme.test.ts new file mode 100644 index 000000000..24980d68a --- /dev/null +++ b/packages/protobuf-test/src/readme.test.ts @@ -0,0 +1,100 @@ +// Copyright 2021-2022 Buf Technologies, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { describe, expect, test } from "@jest/globals"; +import { User } from "./gen/ts/extra/example_pb.js"; + +function createUser(): User { + // Using an object in the constructor + const user = new User({ + firstName: "Homer", + lastName: "Simpson", + active: true, + locations: ["Springfield"], + projects: { SPP: "Springfield Power Plant" }, + manager: { + // you can simply pass an initializer object for this nested message field + firstName: "Montgomery", + lastName: "Burns", + }, + }); + + return user; +} + +function verifyUser(user: User) { + /* eslint-disable @typescript-eslint/no-non-null-assertion */ + expect(user.firstName).toEqual("Homer"); + expect(user.lastName).toEqual("Simpson"); + expect(user.active).toBeTruthy(); + expect(user.locations).toEqual(["Springfield"]); + expect(user.projects).toEqual({ SPP: "Springfield Power Plant" }); + expect(user.manager).toBeDefined(); + expect(user.manager!.firstName).toEqual("Montgomery"); + expect(user.manager!.lastName).toEqual("Burns"); +} + +describe("README examples work as illustrated", () => { + test("setting via property", () => { + const user = new User(); + user.firstName = "Homer"; + user.lastName = "Simpson"; + user.active = true; + user.locations = ["Springfield"]; + user.projects = { + SPP: "Springfield Power Plant", + }; + + const mgr = new User(); + mgr.firstName = "Montgomery"; + mgr.lastName = "Burns"; + + user.manager = mgr; + + verifyUser(user); + }); + test("setting via constructor", () => { + // Using an object in the constructor + const user = createUser(); + + verifyUser(user); + }); + test("binary serialization roundtrip", () => { + const user = createUser(); + + const bytes = user.toBinary(); + // ... + const deserialized = User.fromBinary(bytes); + + verifyUser(deserialized); + }); + test("json serialization roundtrip", () => { + const user = createUser(); + + const json = user.toJson(); + // ... + const deserialized = User.fromJson(json); + + verifyUser(deserialized); + }); + test("json string serialization roundtrip", () => { + const user = createUser(); + + const str = JSON.stringify(user); + // ... + const deserialized = User.fromJsonString(str); + + verifyUser(deserialized); + }); +}); diff --git a/packages/protobuf/README.md b/packages/protobuf/README.md index b7c6a25f7..87475899d 100644 --- a/packages/protobuf/README.md +++ b/packages/protobuf/README.md @@ -5,35 +5,39 @@ This package provides the runtime library for the code generator plugin ## Protocol Buffers for ECMAScript -A complete implementation of [Protocol Buffers](https://developers.google.com/protocol-buffers) -in TypeScript, suitable for web browsers and Node.js. -Learn more at [github.com/bufbuild/protobuf-es](https://github.com/bufbuild/protobuf-es). +A complete implementation of [Protocol Buffers](https://developers.google.com/protocol-buffers) in TypeScript, +suitable for web browsers and Node.js. +**Protobuf-ES** is intended to be a solid, modern alternative to existing Protobuf implementations for the JavaScript ecosystem. It is the first project in this space to provide a comprehensive plugin framework and decouple the base types from RPC functionality. -It is a complete implementation of [Protocol Buffers](https://developers.google.com/protocol-buffers) -in TypeScript, suitable for web browsers and Node.js. +Some additional features that set it apart from the others: -For example, the following definition: +- ECMAScript module support +- First-class TypeScript support +- Generation of idiomatic JavaScript and TypeScript code. +- Generation of [much smaller bundles](packages/protobuf-bench) +- Implementation of all proto3 features, including the [canonical JSON format](https://developers.google.com/protocol-buffers/docs/proto3#json). +- Implementation of all proto2 features, except for extensions and the text format. +- Usage of standard JavaScript APIs instead of the [Closure Library](http://googlecode.blogspot.com/2009/11/introducing-closure-tools.html) +- Compatibility is covered by the protocol buffers [conformance tests](packages/protobuf-conformance). +- Descriptor and reflection support -```protobuf -message Person { - string name = 1; - int32 id = 2; // Unique ID number for this person. - string email = 3; -} +## Installation + +```bash +npm install @bufbuild/protobuf ``` -Is compiled to an ECMAScript class that can be used like this: +## Documentation -```typescript -let pete = new Person({ - name: "pete", - id: 123 -}); +To learn how to work with `@bufbuild/protobuf` check out the docs for the [Runtime API](https://github.com/bufbuild/protobuf-es/blob/main/docs/runtime_api.md) +and the [generated code](https://github.com/bufbuild/protobuf-es/blob/main/docs/generated_code.md). -let bytes = pete.toBinary(); -pete = Person.fromBinary(bytes); -pete = Person.fromJsonString('{"name": "pete", "id": 123}'); -``` +Official documentation for the Protobuf-ES project can be found at [github.com/bufbuild/protobuf-es](https://github.com/bufbuild/protobuf-es). + +For more information on Buf, check out the official [Buf documentation](https://docs.buf.build/introduction). + +## Examples + +A complete code example can be found in the **Protobuf-ES** repo [here](https://github.com/bufbuild/protobuf-es/tree/main/packages/protobuf-example). -Learn more at [github.com/bufbuild/protobuf-es](https://github.com/bufbuild/protobuf-es). diff --git a/packages/protoc-gen-es/README.md b/packages/protoc-gen-es/README.md index dda483aa0..1c4e98661 100644 --- a/packages/protoc-gen-es/README.md +++ b/packages/protoc-gen-es/README.md @@ -1,38 +1,11 @@ # @bufbuild/protoc-gen-es -The code generator for Protocol Buffers for ECMAScript. For example, the -following definition: - -```protobuf -message Person { - string name = 1; - int32 id = 2; // Unique ID number for this person. - string email = 3; -} -``` - -Is compiled to an ECMAScript class that can be used like this: - -```typescript -let pete = new Person({ - name: "pete", - id: 123 -}); - -let bytes = pete.toBinary(); -pete = Person.fromBinary(bytes); -pete = Person.fromJsonString('{"name": "pete", "id": 123}'); -``` - -Learn more about the project at [github.com/bufbuild/protobuf-es](https://github.com/bufbuild/protobuf-es). - +The code generator plugin for Protocol Buffers for ECMAScript. Learn more about the project at [github.com/bufbuild/protobuf-es](https://github.com/bufbuild/protobuf-es). ## Installation -`protoc-gen-es` is a code generator plugin for Protocol Buffer compilers, -like [buf](https://github.com/bufbuild/buf) and [protoc](https://github.com/protocolbuffers/protobuf/releases). -It generates base types - messages and enumerations - from your Protocol Buffer -schema. The generated code requires the runtime library [@bufbuild/protobuf](https://www.npmjs.com/package/@bufbuild/protobuf). +`protoc-gen-es` generates base types - messages and enumerations - from your Protocol Buffer +schema. The generated code requires the runtime library [@bufbuild/protobuf](https://www.npmjs.com/package/@bufbuild/protobuf). It is compatible with Protocol Buffer compilers like [buf](https://github.com/bufbuild/buf) and [protoc](https://github.com/protocolbuffers/protobuf/releases). To install the plugin and the runtime library, run: diff --git a/packages/protoplugin-example/README.md b/packages/protoplugin-example/README.md index 9a5fed93e..c72eb1862 100644 --- a/packages/protoplugin-example/README.md +++ b/packages/protoplugin-example/README.md @@ -3,12 +3,9 @@ This directory contains an example plugin, which shows how to work with the plugin framework. It also contains a separate webpage which shows the generated files working with a remote server. -The code generation logic for the actual plugin is located in the following file: +The code generation logic for the actual plugin is located in [`protoc-gen-twirp-es.ts`](src/protoc-gen-twirp-es.ts). -- [`protoc-gen-twirp-es.ts`](src/protoc-gen-twirp-es.ts) - -The sample plugin generates a [Twirp](https://twitchtv.github.io/twirp/docs/spec_v7.html) client from service -definitions in Protobuf files. The Twirp client uses base types generated from `protobuf-es`. +The sample plugin generates a [Twirp](https://twitchtv.github.io/twirp/docs/spec_v7.html) client from service definitions in Protobuf files. The Twirp client uses base types generated from [`@bufbuild/protobuf-es`](https://www.npmjs.com/package/@bufbuild/protoc-gen-es). From the project root, first install and build all required packages: diff --git a/packages/protoplugin-test/README.md b/packages/protoplugin-test/README.md index 2e7d88a19..1a2e59041 100644 --- a/packages/protoplugin-test/README.md +++ b/packages/protoplugin-test/README.md @@ -1,3 +1,3 @@ # Tests -This package provides test coverage for @bufbuild/protoplugin with Jest. +This package provides test coverage for `@bufbuild/protoplugin` with Jest. diff --git a/packages/protoplugin/README.md b/packages/protoplugin/README.md index 98893e721..eedb31025 100644 --- a/packages/protoplugin/README.md +++ b/packages/protoplugin/README.md @@ -3,13 +3,13 @@ This package helps to create your own code generator plugin using the Protobuf-ES plugin framework. -Protobuf-ES is a complete implementation of [Protocol Buffers](https://developers.google.com/protocol-buffers) in TypeScript, suitable for web browsers and Node.js. +**Protobuf-ES** is a complete implementation of [Protocol Buffers](https://developers.google.com/protocol-buffers) in TypeScript, suitable for web browsers and Node.js. In addition to a full Protobuf runtime library, it also provides a code generator -`protoc-gen-es`, which utilizes a plugin framework to generate base types from +[`protoc-gen-es`](https://www.npmjs.com/package/@bufbuild/protoc-gen-es), which utilizes a plugin framework to generate base types from your Protobuf schema. It is fully compatible with both Buf and protoc compilers. -And now, you can write your own Protobuf-ES compatible plugins using this same +And now, you can write your own **Protobuf-ES** compatible plugins using this same plugin framework with the `@bufbuild/protoplugin` package. With `@bufbuild/protoplugin`, you can generate your own TypeScript code tailored @@ -29,4 +29,6 @@ TypeScript and your own compiler options. With `protoplugin`, you have all the tools at your disposal to produce ECMAScript-compliant code. +## Usage + Get started now with our [plugin documentation](https://github.com/bufbuild/protobuf-es/blob/main/docs/writing_plugins.md).