June logbook

2026-06-25 to 2026-07-05: leave

2026-06-23 to 24

gsul alias, handed to Jim
- on return Ant has hard coded all environments ARNs (tmp during migration)
- has been merged to main, need to enable in all envs, rollout and get dependent project changes merged.

2026-06-22

psu cdk: gsul alias

test:

deploy alias from aea-6515-lambda-alias

$ aws lambda get-alias --function-name arn:aws:lambda:eu-west-2:591291862413:function:psu-pr-3196-GetStatusUpdates --name dev
{
    "AliasArn": "arn:aws:lambda:eu-west-2:591291862413:function:psu-pr-3196-GetStatusUpdates:dev",
    "Name": "dev",
    "FunctionVersion": "$LATEST",
    "Description": "Alias for GetStatusUpdates Lambda (gsul)",
    "RevisionId": "11a5a937-e820-47e1-b985-8ef351547ec7"
}

run aws lambda get-alias and PfP PR test
deploy alias from aea-6515-stateful-only
repeat cli and PfP PR test

2026-06-19

rubber duck
- regression tests
  - now working! Ant supposes transient issue
- gsul lambda alias
- query from Darren via Jack:
  - a prescription created via FHIR but potentially being cancelled via HL7?
    - Matt referenced ‘reset history’ button in EPSAT
    - Jack to confirm ‘which button pressed?’

gsul lambda alias

use level 2 (Alias, not CFAlias)

test change of ARN on repeat deploy

had to delete old stack as 2 diff alias w same name (makes sense)

first clean deploy https://github.com/NHSDigital/eps-prescription-status-update-api/actions/runs/27819309137/job/82335645883?pr=3196 v2

Outputs:
PsuApiFrontDoorStack.GetStatusUpdatesAliasArn = arn:aws:lambda:eu-west-2:591291862413:function:psu-pr-3196-GetStatusUpdates:dev
PsuApiFrontDoorStack.GetStatusUpdatesAliasName = dev
Stack ARN:
arn:aws:cloudformation:eu-west-2:591291862413:stack/psu-api-pr-3196-frontdoor/c1399480-6bcd-11f1-b92c-06f9329ad133

second run (trivial change):

Outputs:
PsuApiFrontDoorStack.GetStatusUpdatesAliasArn = arn:aws:lambda:eu-west-2:591291862413:function:psu-pr-3196-GetStatusUpdates:dev
PsuApiFrontDoorStack.GetStatusUpdatesAliasName = dev
Stack ARN:
Arn:aws:cloudformation:eu-west-2:591291862413:stack/psu-api-pr-3196-frontdoor/c1399480-6bcd-11f1-b92c-06f9329ad133

rerun PfP (after manually deleting stack) https://github.com/NHSDigital/prescriptionsforpatients/actions/runs/27770275546?pr=2760 v4
check with cli:

$ aws lambda get-alias --function-name arn:aws:lambda:eu-west-2:591291862413:function:psu-pr-3196-GetStatusUpdates --name dev 
{
    "AliasArn": "arn:aws:lambda:eu-west-2:591291862413:function:psu-pr-3196-GetStatusUpdates:dev",
    "Name": "dev",
    "FunctionVersion": "$LATEST",
    "Description": "Alias for GetStatusUpdates Lambda (gsul)",
    "RevisionId": "11a5a937-e820-47e1-b985-8ef351547ec7"
}

maybe complete the runbook migration just for own understanding?
new laptop setup
- can’t run powershell setup script as not signed!
- run powershell as admin has to be done as right click in start menu?
- wsl --install Ubuntu-24.04
- use Jim’s new fetch_repo.sh in eps-repo-status

2026-06-18

merge and release empty stacks
PR for #2 for review
- tried a query to find version before $LATEST, none there?
  - a SAM characteristic?
- instead publish CF output for $LATEST from SAM exposed under resourcePrefix
- seems good? deployment happens, execution happens, regression tests fail
regression tests also fail on release cut -> auth failure
- related to APIM changes?
- Maybe need test app reconnected (after APIM name change (removal of -api)? Or new secret?

2026-06-17

deploy empty stacks (normal release)

ready for re-review

set up gsul alias in psu cdk front door stack (normal release)

set copilot on this

change pfp/cpt to use alias (normal release)

set copilot on this

implement all the api stuff in stateless cdk stack. Hard code the resource arns into the code that stateless stack needs (normal release)
implement new style proxygen deployment pointing to stateless cdk stack - remove old proxygen deployment (normal release)

remove stateless stuff from old cloudformation stack (normal release)
migration prep in cloudformation stack (removing exports, changing retention on resources to migrate, removing resources from cloudformation stack) - this can all be done in several normal releases
import resources that were in cloudformation into cdk stack - this is the only manual step
deploy cdk stateful stack - should be no changes following import - normal releases
tidy up cdk stacks - eg - change hard coded lookups put in place earlier to use values from stateful stack
remove cloudformation deployment steps

2026-06-16

rubber duck
- empty stack definition: Ant wants to add front door
  - surely the point of the alias is to have a stable name?
  - match to environmenti? eg dev in dev, int in int…
- .js qualified imports
- Jack / Darren convo
psu migration
- empty stack without lambda alias but with front door stack
- new copilot instructions pr
new laptop arrived

2026-06-15

dependabots
- PSU: all good, Jim identified issue with merge to main
PSU empty stack
- needs to adopt destructive changes approach from Matt
- need to introduce lambda alias for gsul
- still have issue with proxygen deploy, Ant says bin it.
  - solved by fixing api name (remove -api suffix)
PSU migration
- potentially need to provide formal export constants for dependencies between stateless and stateful
  - https://nhsdigital-platforms.slack.com/archives/D09EEC98VKM/p1781510707579809

2026-06-12

duck
- ant wants pre-req PR to minimised DF changes like KMS * expansions
PRs merged:
- Jim’s PSU proxygen migration (pre-req for cdk empty stack)
- PfP python bump to 3.14
- PfP delete stacks
PRs reviewed
- PSU CDK empty stack
  - but… couple of findings to resolve, notably proxygen deployment test
  - and also needed to merge some stack changes (resourcePrefix)
PSU PRs
- kicked off rebasing, currently 26 open

2026-06-11

6515: migration
- continuing the runbook creation started Weds pm
- almost completed a clean run, but unclear why Table nested stacks did not go

2026-06-10

6515: empty stack
- merged Jim’s APIM change into empty-stack but SAM proxygen still fails
- investigated, cannot find a difference
- try a new PR?
- no idea why but a new build due to apparently unrelated set -euo pipefile change caused it to start working https://github.com/NHSDigital/eps-prescription-status-update-api/pull/3126

6515: migration

search for resources imported from account-resources

needed to patch for renamed account resources

-      Fn.importValue("account-resources:LambdaAccessSecretsPolicy")
+      Fn.importValue("secrets-cdk:IAM:LambdaAccessSecretsPolicy:Arn")
-      Fn.importValue("account-resources:LambdaAccessSecretsPolicy")
+      Fn.importValue("secrets-cdk:IAM:LambdaAccessSecretsPolicy:Arn")
-    const cloudWatchKmsKeyArn = Fn.importValue("account-resources:CloudwatchLogsKmsKeyArn")
+    const cloudWatchKmsKeyArn = Fn.importValue("account-resources-cdk-uk:KMS:CloudwatchLogsKmsKey:Arn")
-        destinationArn: Fn.importValue("lambda-resources:SplunkDeliveryStream"),
-        roleArn: Fn.importValue("lambda-resources:SplunkSubscriptionFilterRole")
+        destinationArn: Fn.importValue("account-resources-cdk-uk:Firehose:SplunkDeliveryStream:Arn"),
+        roleArn: Fn.importValue("account-resources-cdk-uk:IAM:SplunkSubscriptionFilterRole:Arn")
-    const trustStoreBucketArn = Fn.importValue("account-resources:TrustStoreBucket")
+    const trustStoreBucketArn = Fn.importValue("account-resources-cdk-uk:Bucket:TrustStoreBucket:Arn")

successful run: https://github.com/NHSDigital/eps-prescription-status-update-api/actions/runs/27273699033/job/80550722341?pr=3127
remaining imports from outside PSU should be fine
- eps-route53-resources:EPS-domain (eps_environment_route53.yml:52) and
- eps-route53-resources:EPS-ZoneID (eps_environment_route53.yml:57)

2026-06-09

APIM
- merge to main failed on proxygen publish
  - confusion about maybe APIM, then we figured it was another difference from cpt
  - Ant hacked the cdk-utils and eventually we got merged
- fix delete stacks workflow with the same role change used elsewhere
- never did get local proxygen publish working
- Jim did not merge PSU PR so my empty stack PR is still hanging

2026-06-08

APIM

welcome back to a multiple page down msg from Ant not answering the question but just ‘do it like this’.

reversed the make changes, back to the same stage as Friday

Error in deployment script: Error: Error calling lambda lambda-resources-ProxygenPTLInstancePut: {"errorType":"Error","errorMessage":"General error","trace":["Error: General error","    at proxygenErrorHandler (/packages/proxygen/src/helpers.ts:134:14)","    at qle (/packages/proxygen/src/proxygenInstancePut.ts:37:5)","    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)","    at async Cle (/node_modules/@middy/core/index.js:196:24)","    at async BufferedInvokeProcessor.i (/node_modules/@middy/core/executionModeStandard.js:21:15)","    at async BufferedInvokeProcessor.processInvoke (file:///var/runtime/index.mjs:1092:22)","    at async _Runtime.processSingleConcurrent (file:///var/runtime/index.mjs:1178:7)","    at async _Runtime.start (file:///var/runtime/index.mjs:1165:7)","    at async ignition (file:///var/runtime/index.mjs:1634:5)"]}
at invokeLambda (/__w/prescriptionsforpatients/prescriptionsforpatients/node_modules/@nhsdigital/eps-deployment-utils/src/specifications/invokeLambda.ts:19:11)
at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
at async deployApi (/__w/prescriptionsforpatients/prescriptionsforpatients/node_modules/@nhsdigital/eps-deployment-utils/src/specifications/deployApi.ts:94:3)
at async main (/__w/prescriptionsforpatients/prescriptionsforpatients/packages/specification/scripts/deploy_api.ts:11:5)

npm error Lifecycle script deploy-api failed with error: ```

that 'General error' is coming out of the proxygen lambda and when I look up it's internal logging the cause is once again

```
"Error: secretOrPrivateKey must be an asymmetric key when using RS512\n    at Object.x1.exports (/node_modules/jsonwebtoken/sign.js:130:22)\n    at createSignedJWT (/packages/proxygen/src/signingHelpers.ts:30:25)\n    at Qz (/packages/proxygen/src/helpers.ts:53:21)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async qle (/packages/proxygen/src/proxygenInstancePut.ts:26:25)\n    at async Cle (/node_modules/@middy/core/index.js:196:24)\n    at async BufferedInvokeProcessor.i (/node_modules/@middy/core/executionModeStandard.js:21:15)\n    at async BufferedInvokeProcessor.processInvoke (file:///var/runtime/index.mjs:1092:22)\n    at async _Runtime.processSingleConcurrent (file:///var/runtime/index.mjs:1178:7)\n    at async _Runtime.start (file:///var/runtime/index.mjs:1165:7)",
```

finally deciphered the correct combination of params to get deploy and publish to pass
Jason L
- set up spine access: 4x TOPT for livea,liveb,refa,refb
- needed to be [re-]granted VPN access, upgraded software 5.3.2 to 5.3.3
- needed to be granted Logbook access
- TODO: read policies and email security

2026-06-05

APIM

regression tests failure viewing prescriptions
apigee showing ancient PRs (eg 242,254)
- TODO: use as proxygen cli / cdk utils test cases?
Ant’s direction is cdk utils without lambda now that PTL is separate from Prod
created and tested make targets for proxygen_deploy and proxygen_publish
worked locally once I lodged a new MTLS secret (another make target)
mtls secret cannot be lodged from pull request: https://github.com/NHSDigital/eps-cdk-utils/blob/fd0a8dffe48fadcdb36f1d4e13c8cb0dc8e6d9df/packages/deploymentUtils/src/specifications/deployApi.ts#L75
```
if (!isPullRequest) {
  console.log("Store the secret used for mutual TLS to AWS using Proxygen proxy lambda")
  ...
}
```

got to the stage that I had a clear error to read a secret from the AWS lambda (left msg on slack):

State of play on PfP: I have both deploy and publish working locally (to get faster feedback) _and_ using the TS approach that is working for CPT but of course that is different to running in GH.
On running in GH I get the following error in the lambda-resources-ProxygenPTLInstancePut
    "stack": "Error: secretOrPrivateKey must be an asymmetric key when using RS512\n    at Object.y1.exports (/node_modules/jsonwebtoken/sign.js:130:22)\n    at createSignedJWT (/packages/proxygen/src/signingHelpers.ts:30:25)\n    at Jz (/packages/proxygen/src/helpers.ts:53:21)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Ule (/packages/proxygen/src/proxygenInstancePut.ts:26:25)\n    at async Ale (/node_modules/@middy/core/index.js:196:24)\n    at async BufferedInvokeProcessor.i (/node_modules/@middy/core/executionModeStandard.js:21:15)\n    at async BufferedInvokeProcessor.processInvoke (file:///var/runtime/index.mjs:1092:22)\n    at async _Runtime.processSingleConcurrentI think you mentioned having deployed secrets to all enviroments except prod @Anthony Brown?
Which would explain what I am seeing because locally I am using the dev account lambda but in GH it is the lambda in the prod account. How's my hypothesis? (edited)

2026-06-04

APIM
- Ant agrees lambda is overkill now
  - previous concern was
- notified of secrets approved 11:30
- iterate the workflow, frustratingly slowly: 10 mistakes took till 16:30
- QC failures due to SQ 403 at least twice
- APIM steps passing, regression tests failing

2026-06-03

APIM-adjustment for PfP
- figure out the role separation for proxygen deployment
  “for all PTL proxygen operations you call APIM PTL proxygen apis APART FROM PUBLISH, which still uses the APIM PROD proxygen api”
- TODO: update deployment utils to get the correct KID value from CF exports as private key already does (I think)
  - copilot has investigated this
- unwrapping the TS layers, could just do this:
```
aws lambda invoke --profile proxygen-deployment --function-name "${instance_put_lambda}" --cli-binary-format raw-in-base64-out --payload file://payload.json out.txt > response.json
aws lambda invoke --profile proxygen-publish --function-name "${spec_publish_lambda}" --cli-binary-format raw-in-base64-out --payload file://payload.json out.txt > response.json
```
- why not just use proxygen cli? https://nhsd-confluence.digital.nhs.uk/spaces/APM/pages/804495095/Proxygen+CLI+user+guide

2026-06-02

release cut
- issue with FHIR facade, qc fixed by rerun, tag release failed x2
- added an artificial PR and re-ran the FHIR facade, got it thru
- added the 2 missing ones manually: eps-storage-resources & eps-data-extract
- TODO: PR to add new repos to script
standup
- mentioned to Pete need to write to Wales about PfP v2 (Danielle back Thu)
helped Jim get notifications workflow merged
watch Ant do account resources cdk change
- key is to mod SAM stateless to import static ARN string from CDK statefulr even while SAM continues to manage it
review rainbow deployments for Pete
prep APIM-required adjustments to PfP

2026-06-01

review update notification config workflow https://github.com/NHSDigital/eps-prescription-status-update-api/pull/3108
fill in missing PfP splunk reports for Pete
6515:
- review if its built a stack, whether ci/release look like they might work
- why no environment support in the typescript like svelte/angular do?
planning
- 6633: augment test strategy
decision log
- proactive contact to wales, copy Jen. Discuss on tango w Pete tomorro in 1st instance
release cut
- issue with FHIR facade on qc

2026-05-29

PfP cert update: 26 days to complete non-proxygen testing!
- check with Danielle next week
PfP CDK
- adopt new delete stack code
- splunk reports need updating, change requested of Ant
6515:
- PR 3126:
  - turned out to be a huge rewrite
    - update for private repos
    - reorder stateless second
    - dodgy copilot manipulation of numbers (30->29, 731->730)
    - remove sam stack name passed to stateless (copilot?)
    - add run regression tests
    - add deploy apigee

2026-05-28

complete investigation of cancel prescription
psu cdk
- had to close 2969 due to secret
- PR 3126: prepped empty stack pr, nearly ready, self-review tomorrow
- PR 3127: prepped stateful only pr, possibly already broken