# FactSpectra Assurance: install in your AWS account > The product name comes from `planproof/brand.py`. If the name there changes, update the > `Description` in `cloudformation.yaml` and this README to match. FactSpectra Assurance tests your Amazon Q in Connect assistant against your own plan documents and reports wrong, wrong-plan, and made-up answers, with page-level proof. Everything runs in **your** AWS account: your documents, the questions, and the answers never leave it (unless you choose the Anthropic API option for Claude, described below). ## Before you start: HTTPS or internal The dashboard is never served over plain HTTP on the internet. Pick one: - **Internet-facing (default):** reachable from `AllowedCidr` only, over HTTPS only. You need an ACM certificate for a name you control, e.g. `assurance.yourplan.com`, in the same region: ACM console > **Request certificate** > public, DNS validation. It is free and is usually issued within minutes of adding the validation record. Pass its ARN as `CertificateArn`. The stack refuses to create an internet-facing dashboard without a certificate. - **Internal:** set `LoadBalancerScheme=internal`. The load balancer has only private IPs in the stack's VPC (`10.80.0.0/16`). Your people reach it over VPC peering, Transit Gateway, Direct Connect or a client VPN, and `AllowedCidr` is your internal range (e.g. `10.0.0.0/8`). The certificate is optional here but still recommended. Without one, traffic is HTTP inside your private network. `AllowedCidr` must be `/8` or narrower; `0.0.0.0/0` and other very broad ranges are rejected. ## One-click launch (AWS Marketplace) 1. Subscribe to the product in AWS Marketplace, choose **Continue to Configuration**, pick the fulfillment option **CloudFormation (ECS Fargate)**, then pick the **region where your Amazon Connect instance and Q in Connect assistant live**. 2. Choose **Launch CloudFormation**. The stack page opens with the template and the container image already filled in. 3. Fill in **Access**: - **AllowedCidr**: your office or VPN egress range, e.g. `203.0.113.0/24`, or your own address as `x.x.x.x/32` (find it at ). - **CertificateArn** (internet-facing) or **LoadBalancerScheme** = `internal` (see above). 4. Tick **I acknowledge that AWS CloudFormation might create IAM resources** and choose **Create stack**. 5. After about 5 to 8 minutes the stack reaches `CREATE_COMPLETE`. Create a DNS CNAME from your certificate's name to the host in the **DashboardUrl** output, then sign in (next section). The app finds your Q in Connect assistant and uses Claude through Amazon Bedrock in your account. You don't need to configure anything else. CLI equivalent (for testing outside Marketplace): ```bash aws cloudformation create-stack --stack-name factspectra \ --template-body file://deploy/cloudformation.yaml \ --capabilities CAPABILITY_IAM \ --parameters ParameterKey=AllowedCidr,ParameterValue=203.0.113.0/24 \ ParameterKey=CertificateArn,ParameterValue=arn:aws:acm:us-east-1:111122223333:certificate/... \ ParameterKey=ContainerImage,ParameterValue= ``` ## Signing in An admin password is generated at install and kept in Secrets Manager. Nobody types it into the template, and it never appears in the task definition. 1. Open the **AdminPasswordConsoleUrl** output, choose **Retrieve secret value**, and copy `password`. From a terminal you can run the **AdminPasswordCommand** output instead. 2. Open `https://` and sign in with that password. To change the password, edit the secret value (keep the JSON shape `{"password": "..."}`), then restart the service: ECS console > service > **Update** > **Force new deployment**. Login sessions are signed with a second generated secret (`PLANPROOF_SESSION_SECRET`). A new value for that secret signs everyone out. ### Parameters | Parameter | Default | What it does | |---|---|---| | `AllowedCidr` | none (required) | Who can reach the load balancer. Must be `/8` to `/32`. | | `LoadBalancerScheme` | `internet-facing` | `internet-facing` (HTTPS required) or `internal` (private IPs only). | | `CertificateArn` | blank | ACM certificate. Required for `internet-facing`. With it, HTTP redirects to HTTPS. | | `ClaudeProvider` | `auto` | `auto` uses the Anthropic API key if the secret holds one and Bedrock otherwise. `anthropic` always uses the key. `bedrock` always uses Bedrock. | | `AnthropicApiKey` | blank | Stored in Secrets Manager only. Leave it blank to use Bedrock. | | `BedrockModelId` | `anthropic.claude-opus-5-5` | The Claude model (or inference profile, e.g. `us.anthropic...`) on Bedrock. | | `QConnectAssistantId` | blank | Set this only if the region has more than one Q in Connect assistant. | | `TaskSize` | `small` | `small` is 1 vCPU / 2 GB, `medium` is 2 vCPU / 4 GB, `large` is 4 vCPU / 8 GB. | | `ContainerImage` | Marketplace image | Leave this as provided. | ## Security defaults - **Network:** the load balancer accepts connections only from `AllowedCidr`. App tasks accept traffic only from the load balancer, and EFS accepts traffic only from the tasks. - **Transport:** TLS 1.2/1.3 only (`ELBSecurityPolicy-TLS13-1-2-2021-06`). The buckets deny non-TLS requests, and EFS mounts require TLS. - **App login:** the generated admin password plus signed `HttpOnly`, `SameSite=Strict` cookies. - **Container:** runs as uid 1000 (non-root) with a read-only root filesystem and all Linux capabilities dropped. The only writable paths are `/data` (EFS) and `/tmp` (task ephemeral storage). - **Audit:** ALB access logs go to their own bucket (`AccessLogBucket` output, kept 365 days). ECS Container Insights is on, and app logs go to CloudWatch Logs. - Single sign-on (Cognito or your IdP on the ALB) is not part of the default install. ## What gets created | Resource | Purpose | |---|---| | VPC with 2 public subnets, an internet gateway and an S3 gateway endpoint | Isolated network for the app. No NAT gateway, to keep costs down. | | Application Load Balancer + security groups | Dashboard entry point, restricted to `AllowedCidr`; access logs to S3. | | ECS cluster (Container Insights on), Fargate service (1 task), task definition | Runs the container (`uvicorn planproof.api:app`, port 8080, health check `/api/health`). Tasks have a public IP for outbound API calls only. Inbound traffic is allowed only from the load balancer. | | Documents S3 bucket (SSE-S3, Block Public Access, versioned, TLS-only, old versions expire after 90 days) | Plan documents. The app syncs new or changed PDFs from it. **Kept when the stack is deleted.** | | Access-log S3 bucket (SSE-S3, private, TLS-only, 365-day expiry) | ALB access logs. **Kept when the stack is deleted.** | | EFS file system (encrypted at rest and in transit, IAM-authorized access point, daily AWS Backup) | `/data` in the container: the SQLite database (journal mode `DELETE`, which is safe on NFS) and uploaded PDFs. **Kept when the stack is deleted.** | | 3 Secrets Manager secrets | Anthropic API key (optional, `{"ANTHROPIC_API_KEY": "..."}`), admin password (`{"password": "..."}`, generated), session-signing key (generated). | | IAM task role (least privilege) | Listed below. | | IAM execution role | Pulls the image (including from the Marketplace registry), writes logs, and injects the session-signing key. | | CloudWatch Logs group | App logs, kept for 30 days. | **Task role permissions.** IAM uses the service prefix `wisdom` for Q in Connect. - Q in Connect: `ListAssistants` (discovery). `CreateSession`, `SendMessage` and `GetNextMessage` on assistants and sessions in this account and region. Nothing else. - S3: `ListBucket` and `GetObject` on **the documents bucket only** (read-only). - Secrets Manager: `GetSecretValue` on **its own Anthropic key and admin password secrets only**. - EFS: mount and write through its own access point only. - Bedrock (left out when `ClaudeProvider=anthropic`): `bedrock:InvokeModel` and `InvokeModelWithResponseStream` on `anthropic.*` foundation models and inference profiles only. Environment passed to the container: `AWS_REGION`, `PLANPROOF_DATA=/data`, `PLANPROOF_DB=/data/planproof.db`, `PLANPROOF_SQLITE_JOURNAL=DELETE`, `PLANPROOF_DOCS_BUCKET`, `PLANPROOF_ADMIN_SECRET_ARN`, `PLANPROOF_ANTHROPIC_SECRET_ARN`, `PLANPROOF_CLAUDE_PROVIDER`, `PLANPROOF_BEDROCK_MODEL`, optionally `QCONNECT_ASSISTANT_ID`, and `PLANPROOF_SESSION_SECRET` (injected from its secret). The app reads the Anthropic key and the admin password from Secrets Manager while it runs, so a key you add later is picked up within about 10 minutes. ## How the Q in Connect assistant is found and asked At startup the app calls `ListAssistants` in the stack's region: - **Exactly one assistant:** the app uses it. This is the normal case and needs no configuration. - **None:** the dashboard says no assistant was found. Create one in Amazon Connect, then restart the service (ECS console > service > **Update** > **Force new deployment**). - **More than one:** the app does not guess. Set `QConnectAssistantId` (update the stack) to choose one. For each test question the app opens a new session (`CreateSession`; for a plan-scoped run it adds a tag filter `plan_id=`, which you can edit for each document on the Documents page), sends the question (`SendMessage`), and polls `GetNextMessage` until no `nextMessageToken` is returned. The final message is graded. Q's holding line ("Let me check that for you.") is never graded as an answer. Throttling, service-unavailable and network errors are retried automatically: first by the AWS SDK (adaptive retry mode, up to 10 attempts), then by restarting the conversation with jittered backoff. If a question still fails, or Q doesn't finish answering in time, it is recorded as an **error** and not as a wrong answer. Errored questions don't count toward any score, and you can resume the run to retry them. Permission or configuration problems, such as a missing IAM permission or a wrong assistant ID, are reported right away and are not retried. ## Claude: Amazon Bedrock or the Anthropic API Claude writes the test questions from your documents and grades the assistant's answers. - **Amazon Bedrock (default, no key):** calls are signed with the task role and billed to your AWS account, and data stays in AWS. Make sure the model in `BedrockModelId` is available to the account: Bedrock console > **Model access**. Some Claude models need a one-time access request, and the first call may need an AWS Marketplace model subscription, which an administrator accepts once in the Bedrock console. To use a cross-region inference profile, set `BedrockModelId` to its ID (e.g. `us.anthropic.`). - **Anthropic API:** pass `AnthropicApiKey` at install, or put the key in the secret later: ```bash aws secretsmanager put-secret-value --secret-id \ --secret-string '{"ANTHROPIC_API_KEY":"sk-ant-..."}' ``` The app picks the key up within about 10 minutes, or right away after a new deployment. Requests go to `api.anthropic.com`. For a key-only setup, set `ClaudeProvider=anthropic`. The stack then also removes the Bedrock permissions. - **Switching to Bedrock:** update the stack with `ClaudeProvider=bedrock`, or leave it on `auto` and set the secret value back to `{"ANTHROPIC_API_KEY":""}`. Switching only changes where Claude runs. Your data and results stay where they are. ## Optional: private subnets (no public IPs on tasks) The default install gives the task a public IP for outbound calls, which avoids NAT costs. Inbound traffic is still blocked. If your policy forbids public IPs on workloads, change the template as follows (a copy you maintain; contact support for a reviewed variant): 1. Add two private subnets (e.g. `10.80.10.0/24`, `10.80.11.0/24`) with their own route table, and move the EFS mount targets and the ECS service's `Subnets` to them. Set `AssignPublicIp: DISABLED`. With `LoadBalancerScheme=internal`, put the load balancer there too. 2. Give the private route table outbound access in one of these ways: - **NAT gateway** (about $33 per month per AZ plus data processing). This is the simplest option, and the only one that works with the Anthropic API provider. - **VPC interface endpoints, no internet path at all**. Add `ecr.api`, `ecr.dkr`, `logs`, `secretsmanager`, `wisdom` (Q in Connect) and `bedrock-runtime`, plus the existing S3 gateway endpoint on the private route table. Enable private DNS and allow 443 from the task security group. Each interface endpoint costs about $7.50 per AZ per month. Use `ClaudeProvider=bedrock`. 3. Narrow the task security group's 443 egress to the endpoint security group (endpoints) or keep `0.0.0.0/0` (NAT). ## Rough monthly cost (us-east-1, default `small`, running 24x7) | Item | Approx. per month | |---|---| | Fargate task, 1 vCPU / 2 GB | ~$36 | | Application Load Balancer (light traffic) | ~$20 | | Public IPv4 addresses (internet-facing ALB x2 + task; internal: task only) | ~$11 (~$4 internal) | | Container Insights (one service) | ~$3 | | Secrets Manager (3 secrets) | ~$1.20 | | EFS, S3 (documents + access logs), CloudWatch Logs (a few GB) | ~$2 | | **Infrastructure total** | **~$70 to $75** | These estimates don't include usage-based charges, which depend on how many documents and test runs you have: - **Claude** (Bedrock or Anthropic API): question generation runs once per document, plus one grading call for each test question in each run. - **Amazon Q in Connect**: billed by AWS for each assistant interaction under your Connect pricing. To save money between test cycles, set the ECS service's desired count to 0. Data on EFS and S3 is kept. ## Upgrades AWS Marketplace publishes new image versions. Update the stack with the new `ContainerImage`. The service stops the old task before it starts the new one, because SQLite has a single writer. Expect about 1 to 2 minutes of downtime. Data on EFS is kept. ## Teardown 1. Delete the stack: CloudFormation console > stack > **Delete**, or `aws cloudformation delete-stack --stack-name factspectra`. 2. The **documents bucket, the access-log bucket and the EFS file system are kept on purpose** so that deleting the stack cannot destroy data. To remove them as well: ```bash # buckets: the documents bucket is versioned, so use the S3 console (select bucket > Empty > # Delete), which removes all object versions; `aws s3 rb --force` alone fails while old versions remain. # EFS (mount targets are already removed with the stack) aws efs delete-file-system --file-system-id ``` AWS Backup recovery points for the EFS file system stay in the default backup vault until they expire or you delete them. 3. The secrets are deleted with the stack, but Secrets Manager may hold them for a 7 to 30 day recovery window. The admin password secret has a fixed name (`/admin-password`). To reinstall under the **same stack name** during that window, remove it first: `aws secretsmanager delete-secret --secret-id /admin-password --force-delete-without-recovery`. ## Building the image (vendor) `Dockerfile` installs exact versions from `requirements.txt`. For release builds, pin the base image by digest (the Dockerfile header has the commands): `docker build --build-arg BASE_IMAGE=python:3.12.12-slim-trixie@sha256: .` ## Validating the template (developers) `deploy/validate.sh` parses the YAML (with CloudFormation short tags), checks that references resolve, runs `cfn-lint` if it is installed, and runs `aws cloudformation validate-template` if the AWS CLI has credentials. It does not create any resources.