From ab51c88d94cd4c7ebccebed1986b6201a0055914 Mon Sep 17 00:00:00 2001
From: chris48s <chris48s@users.noreply.github.com>
Date: Sat, 4 Mar 2023 19:44:08 +0000
Subject: [PATCH] update docs following token pool changes (#8933)
* update docs following token pool changes
* Update CONTRIBUTING.md
Co-authored-by: Caleb Cartwright <calebcartwright@users.noreply.github.com>
---------
Co-authored-by: Caleb Cartwright <calebcartwright@users.noreply.github.com>
Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
---
CONTRIBUTING.md | 17 +++++++++---
doc/code-walkthrough.md | 6 ++---
doc/production-hosting.md | 57 +++++++++++++++++----------------------
3 files changed, 42 insertions(+), 38 deletions(-)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 83574f1555..549a3162c0 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -134,9 +134,20 @@ Prettier before a commit by default.
When adding or changing a service [please write tests][service-tests], and ensure the [title of your Pull Requests follows the required conventions](#running-service-tests-in-pull-requests) to ensure your tests are executed.
When changing other code, please add unit tests.
-To run the integration tests, you must have Redis installed and in your PATH.
-Use `brew install redis`, `yum install redis`, etc. The test runner will
-start the server automatically.
+The integration tests are not run by default. For most contributions it is OK to skip these unless you're working directly on the code for storing the GitHub token pool in postgres/redis.
+
+To run the integration tests:
+
+- You must have Redis installed and in your PATH. Use `brew install redis`, `apt-get install redis`, etc. The test runner will start the server automatically.
+- You must also have PostgreSQL installed. Use `brew install postgresql`, `apt-get install postgresql`, etc.
+- Set a connection string either with an env var `POSTGRES_URL=postgresql://user:pass@127.0.0.1:5432/db_name` or by using
+ ```yaml
+ private:
+ postgres_url: 'postgresql://user:pass@127.0.0.1:5432/db_name'
+ ```
+ in a yaml config file.
+- Run `npm run migrate up` to apply DB migrations
+- Run `npm run test:integration` to run the tests
[service-tests]: https://github.com/badges/shields/blob/master/doc/service-tests.md
diff --git a/doc/code-walkthrough.md b/doc/code-walkthrough.md
index 0561d9664f..b1f94c496a 100644
--- a/doc/code-walkthrough.md
+++ b/doc/code-walkthrough.md
@@ -45,14 +45,14 @@ The tests are also divided into several parts:
7. [The service tests themselves][service tests] live integration tests of the
services, and some mocked tests
1. `*.tester.js` in subfolders of [`services`][services]
-8. Integration tests of Redis-backed persistence code
- 1. [`core/token-pooling/redis-token-persistence.integration.js`][redis-token-persistence.integration]
+8. Integration tests of PostgreSQL-backed persistence code
+ 1. [`core/token-pooling/sql-token-persistence.integration.js`][sql-token-persistence.integration]
9. Integration tests of the GitHub authorization code
1. [`services/github/github-api-provider.integration.js`][github-api-provider.integration]
[service-test-runner]: https://github.com/badges/shields/tree/master/core/service-test-runner
[service tests]: https://github.com/badges/shields/blob/master/doc/service-tests.md
-[redis-token-persistence.integration]: https://github.com/badges/shields/blob/master/core/token-pooling/redis-token-persistence.integration.js
+[sql-token-persistence.integration]: https://github.com/badges/shields/blob/master/core/token-pooling/sql-token-persistence.integration.js
[github-api-provider.integration]: https://github.com/badges/shields/blob/master/services/github/github-api-provider.integration.js
Our goal is to reach 100% coverage of the code in the
diff --git a/doc/production-hosting.md b/doc/production-hosting.md
index fd8f11629f..9fb66593f6 100644
--- a/doc/production-hosting.md
+++ b/doc/production-hosting.md
@@ -14,49 +14,41 @@ Production hosting is managed by the Shields ops team:
[operations issues]: https://github.com/badges/shields/issues?q=is%3Aissue+is%3Aopen+label%3Aoperations
[ops discord]: https://discordapp.com/channels/308323056592486420/480747695879749633
-| Component | Subcomponent | People with access |
-| ----------------------------- | ------------------------------- | --------------------------------------------------------------- |
-| shields-io-production | Full access | @calebcartwright, @chris48s, @paulmelnikow |
-| shields-io-production | Access management | @calebcartwright, @chris48s, @paulmelnikow |
-| Compose.io Redis | Account owner | @paulmelnikow |
-| Compose.io Redis | Account access | @paulmelnikow |
-| Compose.io Redis | Database connection credentials | @calebcartwright, @chris48s, @paulmelnikow, @pyvesb |
-| Raster server | Full access as team members | @paulmelnikow, @chris48s, @calebcartwright, @platan |
-| shields-server.com redirector | Full access as team members | @paulmelnikow, @chris48s, @calebcartwright, @platan |
-| Cloudflare (CDN) | Account owner | @espadrine |
-| Cloudflare (CDN) | Access management | @espadrine |
-| Cloudflare (CDN) | Admin access | @calebcartwright, @chris48s, @espadrine, @paulmelnikow, @PyvesB |
-| Twitch | OAuth app | @PyvesB |
-| Discord | OAuth app | @PyvesB |
-| YouTube | Account owner | @PyvesB |
-| GitLab | Account owner | @calebcartwright |
-| GitLab | Account access | @calebcartwright, @chris48s, @paulmelnikow, @PyvesB |
-| OpenStreetMap (for Wheelmap) | Account owner | @paulmelnikow |
-| DNS | Account owner | @olivierlacan |
-| DNS | Read-only account access | @espadrine, @paulmelnikow, @chris48s |
-| Sentry | Error reports | @espadrine, @paulmelnikow |
-| Metrics server | Owner | @platan |
-| UptimeRobot | Account owner | @paulmelnikow |
-| More metrics | Owner | @RedSparr0w |
+| Component | Subcomponent | People with access |
+| ----------------------------- | --------------------------- | --------------------------------------------------------------- |
+| shields-io-production | Full access | @calebcartwright, @chris48s, @paulmelnikow |
+| shields-io-production | Access management | @calebcartwright, @chris48s, @paulmelnikow |
+| Raster server | Full access as team members | @paulmelnikow, @chris48s, @calebcartwright, @platan |
+| shields-server.com redirector | Full access as team members | @paulmelnikow, @chris48s, @calebcartwright, @platan |
+| Cloudflare (CDN) | Account owner | @espadrine |
+| Cloudflare (CDN) | Access management | @espadrine |
+| Cloudflare (CDN) | Admin access | @calebcartwright, @chris48s, @espadrine, @paulmelnikow, @PyvesB |
+| Twitch | OAuth app | @PyvesB |
+| Discord | OAuth app | @PyvesB |
+| YouTube | Account owner | @PyvesB |
+| GitLab | Account owner | @calebcartwright |
+| GitLab | Account access | @calebcartwright, @chris48s, @paulmelnikow, @PyvesB |
+| OpenStreetMap (for Wheelmap) | Account owner | @paulmelnikow |
+| DNS | Account owner | @olivierlacan |
+| DNS | Read-only account access | @espadrine, @paulmelnikow, @chris48s |
+| Sentry | Error reports | @espadrine, @paulmelnikow |
+| Metrics server | Owner | @platan |
+| UptimeRobot | Account owner | @paulmelnikow |
+| More metrics | Owner | @RedSparr0w |
## Attached state
Shields has mercifully little persistent state:
-1. The GitHub tokens we collect are saved on each server in a cloud Redis
- database. They can also be fetched from the [GitHub auth admin endpoint][]
- for debugging.
+1. The GitHub tokens we collect are stored in a fly.io postgres database
2. The server keeps the [resource cache][] in memory. It is neither
persisted nor inspectable.
-[github auth admin endpoint]: https://github.com/badges/shields/blob/master/services/github/auth/admin.js
[resource cache]: https://github.com/badges/shields/blob/master/core/base-service/resource-cache.js
## Configuration
-To bootstrap the configuration process,
-[the script that starts the server][start-shields.sh] sets a single
-environment variable:
+To bootstrap the configuration of non-secret settings, we set a single environment variable:
```
NODE_CONFIG_ENV=shields-io-production
@@ -71,7 +63,8 @@ files:
contains non-secrets which are checked in to the main repo.
- [`default.yml`][default.yml]. This file contains defaults.
-[start-shields.sh]: https://github.com/badges/ServerScript/blob/master/start-shields.sh#L7
+Secrets are supplied directly as environment vars.
+
[config]: https://github.com/lorenwest/node-config/wiki/Configuration-Files
[local-shields-io-production.yml]: ../config/local-shields-io-production.template.yml
[shields-io-production.yml]: ../config/shields-io-production.yml
--
GitLab