2026-09-01

  • CDK
    • redo dry run
  • 6055:
    • see if the AI has a plan

2026-08-28

  • PDM
    • currently only silver service, working on getting platinum (this FY)
    • Jane ? going to mat leave soon, Michael Sinclair taking over
  • CPSU problem flagged by Kayal
    • likely due to eRD test data, disable feature flag (manually)
      • initally thought, can’t do that as only CDK param, not SSM param
      • but of course it’s an env var on the lambda🤦
  • 6908: retries
    • add evidence and close
  • 6857: eRD
    • release revised build to Kayal in qa initially
    • then to int

Panic attack on eRDs not showing

CPSU has With Pharmacy as well as RTC

  1. find PSU dynamo db transactions: 2 but both RTC
index=app_prescriptions_qa source="aws:loggroup:/aws/lambda/psu-*UpdatePrescriptionStatus" F98A42-A83008-775565
| spath path=message.command.input.TransactItems{0}.Put.Item.PrescriptionID.S output=PrescriptionId
| spath path=message.command.input.TransactItems{0}.Put.Item.PatientNHSNumber.S output=PatientNhsNumber
| spath path=message.command.input.TransactItems{0}.Put.Item.Status.S output=Status
| spath path=message.command.input.TransactItems{0}.Put.Item.ApplicationName.S output=ApplicationName
| spath path=message.timestamp output=rawTime
| spath path="message.x-request-id" output=x_request_id
| eval time_iso=strftime(strptime(rawTime,"%Y-%m-%dT%H:%M:%S.%3NZ"),"%Y-%m-%dT%H:%M:%S.%3NZ")
| table PrescriptionId PatientNhsNumber time_iso Status ApplicationName x_request_id
  1. New release made for the change to the eRD works :-/

2026-08-27

Problem: find all prescriptions not sent updates, determine if have been retried

  1. Affected request ids (775)

Between 25 Aug 26 10:17 and 11:08 BST

index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-3-beta-UpdatePrescriptionStatus" "message.level"=ERROR
| search message.message="Failed to push prescriptions to the notifications SQS"
| rename "message.nhsd-request-id" as request_id
| table request_id
  1. Request ids for successful notify requests
index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-5-beta-UpdatePrescriptionStatus"
| search message.message="Successfully sent a batch of prescriptions to the SQS"
| rename "message.nhsd-request-id" as request_id
| table request_id
  1. All prescription updates received

Problem with this is that it can only be run for very short periods (like 15 mins) before hitting 100Mb disk limit.

index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-*-beta-UpdatePrescriptionStatus"
| search message.message="Transitioning item status."
| eval iso_dt=strftime(_time, "%Y-%m-%dT%H:%M:%S.%3N%z")
| table iso_dt message.x-request-id message.prescriptionID message.nhsNumber message.newStatus message.newTerminalStatus message.previousStatus message.previousTerminalStatus
  1. successful notify requests (variation on 2 above) showing prescription id

Promising, gives > 20K records from 25th Aug 11:08 to 27th 18:00

index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-5-beta-UpdatePrescriptionStatus"
| search message.message="Transitioning item status."
| rename "message.x-request-id" as request_id
| rename "message.prescriptionID" as prescription_id
| search [
    search index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-5-beta-UpdatePrescriptionStatus"
    | search message.message="Successfully sent a batch of prescriptions to the SQS"
    | rename "message.nhsd-request-id" as request_id
    | fields request_id
  ]
| table request_id prescription_id message.nhsNumber message.newStatus message.newTerminalStatus message.previousStatus message.previousTerminalStatus
  1. correlate succssful notifications with problematic prescription ids

This gives 0 records having scanned over 100 million events between 25 Aug 11:08 to 27 18:00 as well as the warning:

[subsearch]: Subsearch produced 10000 results, truncating to maxout [subsearch_maxout] 10000.

I doubt we’re going to get any sense out of this route.

index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-*-beta-UpdatePrescriptionStatus"
| search message.message="Transitioning item status."
| rename "message.x-request-id" as request_id
| rename "message.prescriptionID" as prescription_id
| search prescription_id IN ("PRESCRIPTION_ID")
| search request_id [
    search index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-5-beta-UpdatePrescriptionStatus"
    | search message.message="Successfully sent a batch of prescriptions to the SQS"
    | rename "message.nhsd-request-id" as request_id
    | fields request_id
  ]
| table request_id prescription_id message.nhsNumber message.newStatus message.newTerminalStatus message.previousStatus message.previousTerminalStatus
  1. find requests with retries

Finds 2,094 from the point we started the release on 25th to 27 Aug 18:00

index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-*-beta-UpdatePrescriptionStatus"
| search message.message="Transitioning item status."
| eval iso_dt=strftime(_time, "%Y-%m-%dT%H:%M:%S.%3N%z")
| eventstats count as request_count by message.x-request-id
| where request_count > 1
| table iso_dt message.x-request-id message.prescriptionID message.nhsNumber message.newStatus message.newTerminalStatus message.previousStatus message.previousTerminalStatus request_count

restrict to prescriptions that are of interest gives 2,117 retries across all suppliers in the range of 2-4 retries.

index=app_prescriptions_prod source="AWS:LogGroup:/aws/lambda/psu-api-stateless-v1-18-*-beta-UpdatePrescriptionStatus"
| search message.message="Transitioning item status."
| search message.prescriptionID IN ("PRESCRIPTION_ID")
| eval iso_dt=strftime(_time, "%Y-%m-%dT%H:%M:%S.%3N%z")
| eventstats count as request_count by message.x-request-id
| where request_count > 1
| table iso_dt message.x-request-id message.prescriptionID message.nhsNumber message.newStatus message.newTerminalStatus message.previousStatus message.previousTerminalStatus request_count

2026-07-26

  • show and tell
    • PARS replacing SARS
      • eps library approach, not yet in place
    • test tooling discussion prompted thoughts of how it is hard to interact with PSU

PSU tooling use case

  1. TUI app that accepts params
  • aws profile
  • period(default to last five minutes),
  • max count: defaults to 5
  1. Sample prescription updates during period limit number to max count but include total count
  2. Where updates are RTC / RTC Partial, report the update status
  3. Present a table of results inc. result index number
  4. Await user input: options to include: q (quit), r (resample), (n) resend notification (additionally requires result index)

Seems promising, but can’t decrypt prod without Teams. Need way to create test prescription and send update.

spr - eps store

  • 3 mnths prescription data already terabytse
    • suggests eps data store will only be summary (model on FHIR vsn of app reqts?)
  • PDM is basically a FHIR document store

    • carries but does not validate our profiles

    • Matt: do we just hold the references to prescribe, dispenser, patient in pdm fhir docs

      • existing sources ODS for prescribers, dispensers etc, pds for patient, healthcare ??? api too
      • ODS (but not DOHS) keeps obsolete records (pharmacy that closed)

    • Ant

      • If hold reference only, will we see patient as is not as was in the moment
      • PDS should be able to store history (name changes etc)

    • Being data driven about this: trading larger redundant records vs compute to look up / rehydrate each time

    • 2 prescription line items of 20 pills and 10 pills, could be fullfilled as 10, 10, 10

      • need to record prescribed line items and dispense items
        • could prescribe cocodamol as separate codeine and paracetamol both tracked back to same prescription line item.
        • need longitudinal view not only of what was prescribed by what patient received.
        • justification? could

    • Rob: initially ignore claims, but understand how easy it would be to do.

    • Matt: could update the medication request with tracking status (rather than storing separately)

      • benefit: making PfP requests is faster / simpler
      • What is service level of PDM? and eventual consistency or instant? if eventual, how long?

    • is it useful to store EPS status? (cancelled, expiry, others?)

      • we need something as not all dispensers are onboarded to NPPTS (and may never be?)

spr data model

  • medication request info
    • extensions
      • number of prescriptions issued: for erds
      • authorisation expiry date (for erd)
      • responsible practitioner :
        • currently can have multiple (locum, registrar etc)
        • (Oracle want to do this for 2ndary care). Could we use prescribing org?
      • nhsbs prescription type (nonsense!!)
        • take first half only?
        • map to category?
      • controlledDrug
        • schedule
      • dispensingInformation
        • dispenseStatus: needed?
      • prescriptionStatusHistory:
        • NPPTS status (embedded?)
        • full prescription status
    • identifier
      • may need new extension that holds presciption and issue number for erds
    • status: this is where we put cancelled
    • statusReason
    • intent
    • category: leave (NHS specific), outpatient, community, discharge … not mapped to HL7
    • medication
      • could we replace Snowmed with DM+D (they are already DM+D codes)
      • could replace separate to this work (certainly on the way out, should be ok on the way in)
    • subject: only nhs number
    • requester: the person (or org?) signing; suggest practitioner role and practicioner org
    • performer: ~~?~~inc. (used for pfp, should really be dispenseRequest performer)
      • intention is admin
      • pfp uses as the actual dispenser rather than nominated one (stored in dispense request performer)
    • basedOn (used for erd)
    • groupIdentifier (holds precription id)
    • courseOfTherapyType
    • note
    • doseageInstruction
    • dispenseRequest
      • extension
        • performerSiteType - exclude
      • initialFill - not supported - exclude
      • dispenseInterval - exc. not currently supported (fhir expects fixed interval hence can’t use for eRD) - ex
      • validityPeriod - inc. - use for post dated scripts (diff to post dated notifications) - filtered out > 28 in future in PfP (spine?), Rob would like to see how many post dated scripts actually used (Rob says should be discouraged)
      • numberOfRepeatsAllowed - include - PfP max 6 in API is wrong!
      • quantity - inc.
      • expectedSupplyDuration - inc. eRD, currently in days, FAME could be smaller
      • performer - inc. ODS of nominated pharmacy
      • substitution - inc. - policy says currently always false - (few exceptions like sugar free, cfc free), US allows substitution of anything deemed professionally appropriate
      • priorPrescription - exc. - could use once have single item prescribing - could use for previous eRD issue (compound prescription id and issue) as part of SPR
      • detectedIssue - exc.
      • eventHistory - exc.

2026-07-25

2026-08-24

  • 6857
    • try to get PfP v1.11.3-beta deployed
      • confirmation of changed link by App team
        • need to raise change request with Notify
    • is PfP v1.12.0-beta unblocked from FHIR facade issues? Release to int.
      • released to int but issue in qa that Kayal is not seeing the test data
         $ curl  --header "nhsd-nhslogin-user: P9:9436300034" --header "x-request-id: $(cat /proc/sys/kernel/random/uuid)"  https://${stack_name}.qa.eps.national.nhs.uk/Bundle
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
vscode ➜ /workspaces/prescriptionsforpatients (main) $ echo $stack_name
pfp-api-v1-12-0-beta
  • PSU CDK
    • migrate in dev, inc merge of 3346

2026-08-21

2026-08-20

  • PSU CDK
    • agreed to defer tightening permissions will after CDK migration complete
    • dry run
      • ran into validation error after [accidentally] triggering unmigrated builds by pushing doc changes
        • start clean migration on new branch
          • SQS went fine
          • issues with tables
            • some refining of rename and move scripts

2026-08-19

  • 6857
    • merged PR
    • release needs to wait for this week’s release to clear int
  • PSU CDK
    • completed SQS test migration on PR branch
    • prep similar steps for tables
    • identified that permissions are still looser than ideal (eg kms:* in Messaing.ts)

2026-08-18

  • 6857
    • spine setup
      • had to kill iBoss (start > services > idsa > click stop)
    • actually hard-coded data will work
    • prepped PR
  • 6822
    • review with Jono
  • PSU CDK: parked temporarily

2026-08-17

2026-08-14

  • PSU CDK migration
    • adoption of params merged
    • release cut to replace Jack’s taken on Tuesday
    • testing will be blocked by eRD experiment
    • regression tests failing!
      • retry failed
      • test locally?
  • PfP
    • disable eRD hack to re-enable testing
  • 6857
    • top priority, discuss in planning on Monday

2026-08-13

  • Find sso_role_name for ~/.aws/config
    TOKEN="$(jq -r '.accessToken' ~/.aws/sso/cache/*.json | head -n1)"
aws sso list-account-roles \
  --account-id 434629240718 \
  --access-token "$TOKEN" \
  --query 'roleList[].roleName' \
  --output text | tr '\t' '\n'

2026-08-10,11,12: Vacation

2026-08-07

2026-08-06

  • Cegedim incident
    • prod rollout completed
  • Splunk reports update
    • Ant made the proposed change, Pete confirms arrival of reports today
    • I caught up the missing 4th and 5th data (20 separate queries) manually
  • PSU CDK migration
    • new cut (1.18.3-beta) to add the new parameters and secrets
      • fix grype!
      • script to copy params and secrets

2026-08-06

Cegedim incident

  • APIM unblocked by 13:15
  • Tom confirmed test now passing
  • deploying to int
  • prod rollout blocked by GitHub actions outage

PSU CDK migration

  • proposed to
    1. deploy additional parameters and secrets
    2. adopt the new parameters and secrets
    3. do migration for messaging and then separately for tables
    • agreed at rubber duck
  • could do resource renames

2026-08-05

Cegedim incident

PSU CDK migration

Error for “StackRefactorId”: “2e380eae-ed04-4c7b-8df8-80ff562615e0”

The resource PrescriptionStatusUpdatesTable defined in the resources mapping did not move from its original stack psu-pr-3374-Tables-KFOVFV9B9W3U. Review your stack template desired state.\nThe resource PrescriptionNotificationStatesTableV1 defined in the resources mapping did not move from its original stack psu-pr-3374-Tables-KFOVFV9B9W3U. Review your stack template desired state.\nThe resource PrescriptionStatusUpdatesKMSKey defined in the resources mapping did not move from its original stack psu-pr-3374-Tables-KFOVFV9B9W3U. Review your stack template desired state.\nThe resource PrescriptionNotificationStatesKMSKey defined in the resources mapping did not move from its original stack psu-pr-3374-Tables-KFOVFV9B9W3U. Review your stack template desired state.\nThe resource PrescriptionStatusUpdatesKMSKeyAlias defined in the resources mapping did not move from its original stack psu-pr-3374-Tables-KFOVFV9B9W3U. Review your stack template desired state.\nThe resource PrescriptionNotificationStatesKMSKeyAlias defined in the resources mapping did not move from its original stack psu-pr-3374-Tables-KFOVFV9B9W3U. Review your stack template desired state.\nThe resource BlockedSiteODSCodesParameter defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource EnabledSiteODSCodesParameter defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource EnabledSupplierApplicationIDsParameter defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource EnableNotificationsExternal defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource EnableNotificationsInternal defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource NotifyAPIBaseURLParameter defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource NotifyRoutingPlanIDParameter defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource TestPrescriptionsParameter1 defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource TestPrescriptionsParameter2 defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource TestPrescriptionsParameter3 defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource TestPrescriptionsParameter4 defined in the resources mapping did not move from its original stack psu-pr-3374-Parameters-1QO5KXDIKZ3SW. Review your stack template desired state.\nThe resource SQSSaltSecret defined in the resources mapping did not move from its original stack psu-pr-3374-Secrets-1ENBPHCVRNDJ. Review your stack template desired state.\nThe resource PSUNotifyAPIKeySecret defined in the resources mapping did not move from its original stack psu-pr-3374-Secrets-1ENBPHCVRNDJ. Review your stack template desired state.\nThe resource PSUNotifyApplicationIDSecret defined in the resources mapping did not move from its original stack psu-pr-3374-Secrets-1ENBPHCVRNDJ. Review your stack template desired state.\nThe resource PSUNotifyKIDSecret defined in the resources mapping did not move from its original stack psu-pr-3374-Secrets-1ENBPHCVRNDJ. Review your stack template desired state.\nThe resource PSUNotifyPrivateKeySecret defined in the resources mapping did not move from its original stack psu-pr-3374-Secrets-1ENBPHCVRNDJ. Review your stack template desired state.\nThe resource PSUSecretsKMSKey defined in the resources mapping did not move from its original stack psu-pr-3374-Secrets-1ENBPHCVRNDJ. Review your stack template desired state.\nThe resource PSUSecretsKMSKeyAlias defined in the resources mapping did not move from its original stack psu-pr-3374-Secrets-1ENBPHCVRNDJ. Review your stack template desired state.\nThe resource NHSNotifyPrescriptionsSQSQueue defined in the resources mapping did not move from its original stack psu-pr-3374-Messaging-4GAS9IGUR5AE. Review your stack template desired state.\nThe resource NHSNotifyPrescriptionsDeadLetterQueue defined in the resources mapping did not move from its original stack psu-pr-3374-Messaging-4GAS9IGUR5AE. Review your stack template desired state.\nThe resource PostDatedNotificationsSQSQueue defined in the resources mapping did not move from its original stack psu-pr-3374-Messaging-4GAS9IGUR5AE. Review your stack template desired state.\nThe resource PostDatedNotificationsDeadLetterQueue defined in the resources mapping did not move from its original stack psu-pr-3374-Messaging-4GAS9IGUR5AE. Review your stack template desired state.\nThe resource NotificationSQSQueueKMSKey defined in the resources mapping did not move from its original stack psu-pr-3374-Messaging-4GAS9IGUR5AE. Review your stack template desired state.\nThe resource NotificationSQSQueueKMSKeyAlias defined in the resources mapping did not move from its original stack psu-pr-3374-Messaging-4GAS9IGUR5AE. Review your stack template desired state."

Root cause: CloudFormation refactor validates nested stack templates at their S3 URLs

When create-stack-refactor sees a resource mapping where Source.StackName = psu-pr-3374-Tables-KFOVFV9B9W3U, it downloads the template for that nested stack from its original S3 URL and checks whether the moved resources are absent from the revised source. They aren’t absent — we never provided a revised template for that nested stack — so it fails with “did not move from its original stack”.

The StackDefinitions we provided only covered:

psu-pr-3374 (parent SAM) → source-revised.template.yaml psu-api-pr-3374-stateful (CDK) → destination-revised.template.json CloudFormation also requires revised templates for the four nested stacks that are the actual sources: psu-pr-3374-Tables-, psu-pr-3374-Parameters-, psu-pr-3374-Secrets-, psu-pr-3374-Messaging-.

But the nested stack templates are not straightforward to revise, because — as the Tables stack shows — they contain resources that depend on what we’re moving:

So providing a revised nested template that removes the moved resources leaves broken Ref/Fn::GetAtt references in the resources that stay.

The additional information is all in the describe output — Status: CREATE_FAILED with that long StatusReason list is the full extent of what CF reports. There is no deeper log.

The path forward has two realistic options:

Option 1 — Include revised nested stack templates in StackDefinitions Download each of the 4 nested stacks via get-template, strip the moved resources, upload revised versions to S3, and add them to StackDefinitions. The broken intra-template references would need to be tolerated or worked around (e.g. by also moving the dependent AutoScaling/IAM resources into CDK). This is significantly more complex.

Option 2 — Use CloudFormation resource import instead of stack refactor Rather than moving ownership via refactor:

Remove the SAM nested stacks (Tables, Parameters, Secrets, Messaging) from the SAM parent template — CloudFormation will retain the physical resources because of DeletionPolicy: Retain (or add it first) Import the retained physical resources directly into the CDK stack using aws cloudformation create-change-set –change-set-type IMPORT This avoids the nested stack template problem entirely. The cost is it’s a two-step process (SAM update to drop nested stacks → CDK import) rather than atomic.

2026-08-04

Cegedim incident

  • 500 from CPSU endpoint

  • name of state machine handler has to be ‘handler’

  • no regression tests :-/

  • most of the day spent making release

2026-08-03

  • not on spine triage
  • PSU CDK
    • release to prod done!
    • new cut for Apotec rollout
      • APIM failure to qa and ref -> retry
    • stateful migration rehearsal

2026-07-31