2026-09-30

CDK recovery plan

https://nhsdigital-platforms.slack.com/archives/C0ARWQQ09GU/p1790782824197889

rollback to cd v1.19.2-beta - merge to main branch delete stacks in dev, ref, qa take release cut from main release to dev, ref, qa, int, prod

2026-09-29

2026-09-28

  • EPS triage https://nhsd-confluence.digital.nhs.uk/spaces/APIMC/pages/822183849/Epsilon+Triage+Rota
    • fhir facade: 25 PRs but now building
    • pfp: 7 PRs, updated 2992, revisit, approve and merge
    • PSU: merge to main failure
      • sandbox: CPSU API timeout, retry
      • main: both APIs: 504 Gateway Time-out, retry
    • cpt ui:
      • merge to main failures on several recent releases
        • 2x reg tests (dependabots) => retry
        • 6937: cloudfront cert issue => retry
        • 6706: “Changeset name must past this regex: [a-zA-Z][-a-zA-Z0-9]|arn:[-a-zA-Z0-9:/]”

2026-09-25

Prune INC0425050

https://nhsd-confluence.digital.nhs.uk/spaces/SPINE/pages/7373095/KOP-101+Prescriptions+ChangeLogPruner

  1. ssh into live 2a:
  • prune prescription 2E4634-B83033-A98F2Y
.... -i DUMMY
cannt connect to localhost 8888

  1. ssh nto live 2b

    python pruneprescription.py -scn 5 -p 2E4634-B83033-A98F2Y -i DUMMY many PORX_IN999000UK01, also PORX_IN080101UK31

  • need to send to jen and shaun about …01
  • continue until list is reasnable size (eg <10 entries mostly diff interaction ids)

python pruneprescription.py -scn 5 -p B7F987-B83033-A98F0Z -i DUMMY

  • also needs to be notified: PORX_IN999000UK01 python pruneprescription.py -scn 5 -p C5DD2D-B83033-A98F14 -i DUMMY
  • also notify due to PORX_IN999000UK01

python pruneprescription.py -scn 5 -p CB4858-E85058-AC5EAA -i DUMMY python pruneprescription.py -scn 5 -p AB6A15-M84627-68CC9P -i DUMMY

Can prune b and a from b

python pruneprescription.py -p 2E4634-B83033-A98F2Y -i DUMMY python pruneprescription.py -p 2D5CBA-M88015-C3D12Y -i DUMMY python pruneprescription.py -p CB4858-E85058-AC5EAA -i DUMMY python pruneprescription.py -p AB6A15-M84627-68CC9P -i DUMMY python pruneprescription.py -p B7F987-B83033-A98F0Z -i DUMMY python pruneprescription.py -p C5DD2D-B83033-A98F14 -i DUMMY

  • repeat for INC0424090
  • INC0424292
  • INC0424648

kop-69

ServiceNow will tell you the errors found by KOP-69 but not the resolution

  • log into wallix (a or b according to which is currently live)
  • select opsadmin@local@live2[a|b]-access-msgprocessor-1:SSH
  • run the curl generated by kop69.py
curl "http://localhost:7001/reQueueHandler?overrideSuppressResponse=true&overridePersistDuration=true&inboundOutbound=inbound&internalIDToFetch=20260924134131184243_1E569A_3027570739" &&
curl "http://localhost:7001/reQueueHandler?overrideSuppressResponse=true&overridePersistDuration=true&inboundOutbound=inbound&internalIDToFetch=20260924114817855491_31050E_3027570739"

result:

{“Status”: true, “Output”: {“internalID”: “20260924134131184243_1E569A_3027570739”, “info”: “queued”}}{“Status”: true, “Output”: {“internalID”: “20260924114817855491_31050E_3027570739”, “info”: “queued”}}

  • repeat for INC0424074 (2 records)
  • INC0424289 (1 record)
  • INC0424635 (3 records requeued)
  • INC0425040 (4 records requeued)
  • INC0425386 (4 records requeued)

2026-09-24

2026-09-23

  • show n tell
    • show decision log?
    • show new endpoint in swagger editor
    • show sample request-response
  • planning
    • cdk fix is available says AWS
    • Matt suggests Th morning to try the migration
  • 7015 event stream
    • started reading up on event streams and exploring some CDK
  • 6934:couple of short chats w Tej

2026-09-22

  • rubber duck
    • talk Tej thru testing cdk-utils change
  • release
    • got Bence to do the secrets deployment
    • finally got thru most of the deployments (overrun by few mins)
    • but CPT API failed: “Environment ‘prod’ is disallowed for healthcare-worker-prescription-tracker. Get in touch with the API Management team if this should be changed.”
    • were able to confirm that UI is not dependent via Tom so continued with that later on
  • Got Sirisha thru the aws-configure / aws login
    • next on to writing the query results to the bucket
  • tracing reg tests failure on state machine change with Tej
    • turns out to be double escaping

2026-09-21

2026-09-18

  • 6667: Change is made butEPSAT tests are failing, prepped this but late, so decided to hold for 🦆🦆🦆

    Over the past couple of days, I’ve been seeing timeouts in EPSAT tests across several PRs, including this one that is no more than a change to error messages, largely in OAS: https://github.com/NHSDigital/electronic-prescription-service-api/actions/runs/35337519178/job/105587651183?pr=5187 Is this common? I’d appreciate pointers on how to investigate.

  • 6767: goal is consistency but cancel permits telecom to be undefined - what do we think about that?
    • Matt confirmed there are business reasons that telecom may reasonably be omitted from cancel (because you can refer back to the original creation where it is mandatory)
    • hence go for the simplest change
    • however, EPSAT tests are failing here too.
  • pre-meet on cancel whole prescription decision log, added a couple of items

2026-09-17

  • familiarise self with Adam’s reg test changes
  • 6055: reg test?
  • review for Sirisha - Already done by Ant
  • 6667: cancel prescription error codes
  • 6767: telecom consistency between create and cancel prescriptions

2026-09-16

2026-09-15

2026-09-14

  • PSU CDK
    • certain amount of expected consternation from the team
    • agreed not to escalate
    • need to find out why PSU stack is missing in dev (tables are not)
  • Dependabots
  • 6662: cancel all
    • drafted endpoint, handed schemas to copilot to do
  • decision call
    • sites data:
      • optum (emis): 35.37%
      • boots: 12.55%
      • positive: 11.57%
      • ClanWilliam (big in NI): 10.84%
      • Invatech: 9.41
      • Cegedim: 9.38%
      • Unknown: 7.20
      • Apotec: 3.68
      • Pharmacy2U: 0.01 (but 70-100k scripts per day; embedded in tesco)

2026-09-11

  • PSU CDK migration
    • post migration: if destructive changes then run cdk diff job and remove existing job
    • raised case: 178913604700038
      • I reviewed the event history for that stack, your reading is correct and there have been no further updates to it after that point, including after the refactor was created. Nothing modified the stack between the refactor’s creation and your execution attempt, so the validation error does not reflect an actual update.
      • This matches a known issue with the stack refactor validation, where an internal consistency check can incorrectly report a stack as changed and block execution even when the stack has not been modified. Our service team has identified the cause and is rolling out a fix, with a current target of 2026-09-25.

2026-09-10

  • PSU CDK
    • agreed to do CF refactor and cdk orphan + import in single session, estimated slightly over 1h
    • need to resolve tag difference on import: separate aws cli or incorporate in patch
      • incorporate in patch => re-test on clean branch
      • prepped clean patches and commit them
    • also agreed to remove all SAM in the post-migrate PR merge

2026-09-09: roadmap day in Leeds

2026-09-08

  • CDK
    • Discovered that use of CDK TableV2 is necessarily AWS::DynamoDB::GlobalTable whereas existing SAM is AWS::DynamoDB::Table
    • duck decided to look at what is required to get both done
    • confirmed needs an additional (and separate) migration step
  • SPR meeting, see below

SPR

CommunicationRequest

  • Additional element at header level with no cross-ref with MedicationRequest and certainly no link to item

  • HL7 allows patient information at each item

  • Matt proposes that we use medication request > supporting information to create the missing link

  • At least one supplier has proposed using Comm Req > about to ref the bundle prescription id

  • CommunicationRequest

    • payload
      • contentString
      • contentReference
      • about
      • subject
      • requester - prescibing org resource (to align with prev assumption in Med Req)
      • recipient
      • status - hard code to active as is required

PractitionerRole

Need to store just enough to get practitioner info

  • identifier - inc.
  • code - exc
  • practitioner (seperate object)
    • array of identifiers
      • retaining sds-user-id should be enough? (relies on Healthcare worker API that is ‘being built’)

MedicationDispense

  • MedicationRequest
    • extensions
      • prescriptionNonDispensingReason - exc.
      • prescriptionStatus - exc. - should be composed from item level status
      • repeatInformation - exc. - eRD info could be gleaned from item level
    • statusReason - inc.
    • category - exc.
    • medication - inc. - should always be Actual medical product packs, occasional exceptions mean we need it in addition to Med Req
    • subject - inc.
    • context - exc. - not used
    • supporting info - exc. - not used
    • performer - inc. - same convo about references
    • location - exc.
    • authorizingPrescription - inc. ref to med req
    • type - inc. used to store item status
    • quantity - inc. - same as above, some variation from med req is poss.
    • daysSupply - exc. - not currently used, maybe FAME
    • whenPrepared - exc.
    • whenHandedOver - inc.
    • dosageInstruction - inc.
    • others below whenHandedOver excluded except dosageInstruction

Conclusion

  • create example (create prescription, dispense, PfP?)
  • meeting with PDM to ask their opinion about dehydrated or fully hydrated resources

2026-09-07

  • CDK
    • make an early start and do a dry run
      • cdk warning:

        Confirmed upstream: this is a known, currently open bug — aws/aws-cdk#38709 (“TableV2 constructor still triggers TableGrantsProps#policyResource/#encryptedResource deprecation warnings — regression from #38399’s partial fix”), with several in-progress fix PRs (#38573, #38184, #37467) not yet merged/released.

        • open bug — aws/aws-cdk#38709 (“TableV2 constructor still triggers TableGrantsProps#policyResource/#encryptedResource deprecation warnings — regression from #38399’s partial fix”), with several in-progress fix PRs (#38573, #38184, #37467) not yet merged/released.
  • review decision log for pete https://nhsd-confluence.digital.nhs.uk/spaces/APIMC/pages/1492751877/D345+-+EPS+FHIR+API+Whole+Prescription+Cancellation

2026-09-04

  • FHIR validator
    • help various dependabot PRs thru
    • no spring boot fix for 3 critical vulnerabilities as of 3.5.16
  • dependabot
    • fhir facade: down to 15 (some dependabots still there)
    • account resources: down to 7
  • CDK
    • still didn’t get to redo dry run
  • 6662:
    • questions for Matt
      • start from spec
      • how to fix the fact that code cannot distinguish invoking endpoint?
      • looks like spine could handle multiple cancellations

2026-09-03

  • CDK
    • redo dry run
  • D344 (eRD test) reviewed
  • dependabot
    • account resources start the day at 19
    • fhir facade: still at 30 :-/

6662 cancel all

Discussed with Pete

  • Subsequent cancellation is when prescription is already ‘With Pharmacy’ and has to be returned to spine, then (subsequently) cancelled.

  • new endpoint could be simply to avoid pulling the rug from under existing suppliers

  • is the data item defining a cancel at bundle or item level?

    • yes the coding for cancel is at the MedicationRequest level

  • is cancel single item a valid case? (yes by pharmacy but that is different)

    • would appear yes from OAS

Digging deeper:

  • Need new endpoint because old one can only return single response. Also, request can have >1 MedicationRequest in the spec but validation prevents this in reality. bundle-validator.ts function verifyCancellationBundle contains:

      const medicationRequests = getMedicationRequests(bundle)
if (medicationRequests.length !== 1) {
  validationErrors.push(errors.medicationRequestNumberIssue)
}

  • New endpoint continues to use ‘prescription-order-update’ because these are predefined but operation id and bookmark in psec need to be unique TODO: is this ok?

  • checked the spine expectations:

  • Need to catch up with Phil on FAME work so we don’t tread on each other continuously.

2026-09-02

6662: cancel all

Request flow is:

  1. Route entry: process.ts:38

  2. Bundle-to-Spine wrapper: index.ts:7

  3. Cancellation branch selection: factory.ts:106

  4. Actual cancellation request construction: cancellation.ts:24 select first item only to be cancelled

    Matt suggested a new endpoint but could also simply process all here (also need to check validation, what conditions are rejected?). Could a bundle without MedicationRequest signify ‘cancel all’? Would spine cope with that?

2026-09-01

  • 6055:
    • got the test passing in dev (no MTLS)
    • added the role inc. policy to invoke, created temp role PsuRegressionTestRoleLocal
      • test now failing on preceeding step “I advance the clock to beyond the post-dated time”
      • apparently not related to role changes as fails if I stash those changes
  • reporting spike
    • approx 150k notifications in 24h to 15:00 Tues => sounds about right (pete suggested 200k on midweek but yesterday was bank holiday)
    • Sirisha checking if can schedule
  • eRDs
    • Tom provided screenshots, Pete happy to write up
    • Ideally don’t put invalid FHIR out in the wild (therapyCoding fix needed App side)
  • dependabot
    • PSU: 20 PRs, down to 9 by end of day
    • account resources: 27 PRs
      • retried an APIM deployment and it just fails on the Azure step
        • try @dependabot recreate and see what tomorrow brings

role to test lambda invoke

# set -euo pipefail

# adjust as needed
export AWS_PROFILE=eps-dev
export AWS_REGION=eu-west-2
ROLE_NAME=PsuRegressionTestRoleLocal

ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
CALLER_ARN="$(aws sts get-caller-identity --query Arn --output text)"

# Convert assumed-role ARN -> IAM role ARN so trust policy is valid
if [[ "$CALLER_ARN" == arn:aws:sts::*:assumed-role/* ]]; then
  ROLE_FROM_SESSION="$(echo "$CALLER_ARN" | cut -d/ -f2)"
  TRUST_PRINCIPAL="arn:aws:iam::${ACCOUNT_ID}:role/${ROLE_FROM_SESSION}"
else
  TRUST_PRINCIPAL="$CALLER_ARN"
fi

cat > /tmp/psu-trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowLocalCallerAssumeRole",
      "Effect": "Allow",
      "Principal": { "AWS": "${TRUST_PRINCIPAL}" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

cat > /tmp/psu-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvokeGsul",
      "Effect": "Allow",
      "Action": ["lambda:InvokeFunction"],
      "Resource": [
        "arn:aws:lambda:${AWS_REGION}:${ACCOUNT_ID}:function:psu*-get-status-updates*",
        "arn:aws:lambda:${AWS_REGION}:${ACCOUNT_ID}:function:psu-api*GetStatusUpdates*:*"
      ]
    },
    {
      "Sid": "ResolveGsulArnFromCloudFormation",
      "Effect": "Allow",
      "Action": [
        "cloudformation:DescribeStacks",
        "cloudformation:ListExports",
        "cloudformation:ListStacks"
      ],
      "Resource": "*"
    }
  ]
}
EOF

# create if missing; otherwise update trust
if aws iam get-role --role-name "${ROLE_NAME}" >/dev/null 2>&1; then
  aws iam update-assume-role-policy \
    --role-name "${ROLE_NAME}" \
    --policy-document file:///tmp/psu-trust.json
else
  aws iam create-role \
    --role-name "${ROLE_NAME}" \
    --assume-role-policy-document file:///tmp/psu-trust.json
fi

# upsert inline policy
aws iam put-role-policy \
  --role-name "${ROLE_NAME}" \
  --policy-name PsuRegressionTestPolicyLocal \
  --policy-document file:///tmp/psu-policy.json

# set env var used by regression tests
export PSU_ROLE_ARN="arn:aws:iam::${ACCOUNT_ID}:role/${ROLE_NAME}"
echo "PSU_ROLE_ARN=${PSU_ROLE_ARN}"

# quick validation
aws sts assume-role --role-arn "${PSU_ROLE_ARN}" --role-session-name psu-local-test >/dev/null
echo "Role assume test passed"

2026-08-28

  • PDM
    • currently only silver service, working on getting platinum (this FY)
    • Jane ? going to mat leave soon, Michael Sinclair taking over
  • CPSU problem flagged by Kayal
    • likely due to eRD test data, disable feature flag (manually)
      • initally thought, can’t do that as only CDK param, not SSM param
      • but of course it’s an env var on the lambda🤦
  • 6908: retries
    • add evidence and close
  • 6857: eRD
    • release revised build to Kayal in qa initially
    • then to int

2026-09-01

  • CDK
    • redo dry run
  • 6055:
    • see if the AI has a plan