2026-07-31

2026-07-30

  • spine triage
    • prescriptions not found in pfp
      • can scan psu notify but not prescriptions
      • look at psu logs to find notification and prescription
      • look at secondary indexes for query by nhsnumber (cli not console perhaps)
    • pharmacy x
      • see if fixed by Matt’s change or something different
      • may be transient?
    • potentially suspicious use of CPT
      • code/eps/cpt/cpt-patients-by-session-id.spl
      • code/eps/cpt/cpt-prescriptions-by-session-id.spl
  • PSU CDK
    • in qa, kayal to inform when ready for int
    • stateful migration prep

2026-07-29

  • show n tell
    • quick comparison of cdk import vs cf refactor
  • PSU CDK
    • fixed failed reg tests
      • allure showed it was check prescriptions but response body told us it was mtls
      • eventually remembered the makefile proxygen_put_mtls_secret done for PfP
      • similar solution made for PSU
  • PfP and CPT PRs to use reusable policy -> not priority
  • stateful stack

2026-07-28

  • spine triage
    • KOP-69 4.3
    • ?need to install AWS client, check SSH access
      • may be unnecessary?
    • KOP101 needs a while loop to avoid having to keep rerunning the pruneprescription.py
  • PSU CDK
    • yesterday’s changes for singular schedule and backup restore rule have created a destructive change
      • turns out it would hapen in every build because rule itself in front door but references lambda in stateless
        • could do something similar to gsul (param to hold the arn to invoke)
        • agreed w Matt to drop backup restore rule because it is minimal (describe table and check > 1 row)

2026-07-27: day off

2026-07-24

  • PSU CDK
    • proxygen
      • change tests
  • enforced Windows reboot
    • ugh, all wsl settings wiped out - another hour I’ll never get back!
    • Confirmed findings:
      1. This was not a firewall block pattern.
      2. sshd was running in WSL, but WSL keep-alive was missing after reboot.
      3. Scheduled task existed but was configured to not run / stop on battery.
      4. SSH config had duplicate Port 2222 directives in both:
        • /etc/ssh/sshd_config
        • /etc/ssh/sshd_config.d/10-wsl-port.conf

2026-07-23

  • PSU CDK
    • proxygen merge issue
      • unblocked (correct name is suffixed -api, potentially changed and changed back by APIM)
      • also skip spec publication on PR
      • post dated regression tests fail!
        • tests expect to not have -api suffix
        • effectively the deploymentUtils couple the Proxygen lambda lookup name (prescription-status-update-api) with the Apigee URL path prefix (prescription-status-update),
        • the so choice is to change the tests to support the coupling or change deployment utils to break the coupling
    • stateful stack
      • destructive change (one remaining): ParametersGetNotificationsParameterPolicyE5CF0AF0
  • CPT state machine issue

2026-07-22

  • PSU CDK
    • Ant wants to hold PSU release until CPT fixed even though fix is in CPT and dependent on what is already in prod
    • merged proxygen switch
      • merge to main failed after PR passed!
    • stateful stack
      • validation error disappeared by creating a new PR :-(
      • now failing:
        • destructive changes flagged by diff to main
          • refactoring had created new policy names - fixed those, more to do…
        • reg tests
          • Check the API gateway and lambda logs for why check prescription is not working
  • Proxy briefing
    • 68 GPs currently, expanding to 250
    • cross-GP not available at the moment (anticipated more necessary for proxy to older people)
  • lunch and learn
    • make tests parallel - sounds easy, but
      • use value generators to get your own nhs number, prescription etc
      • don’t hard code dates (10 years in the future expiry is not safe!)
    • just run prescriptions sub-system in jenkins while testing
      • eg many intermittent tests are in clinicals
      • feature branch does not run slow tests , but develop will, so be sure to test before merge
      • jenkins faster than local for big test suites but single tests will be faster locally as jenkins has to do full build
    • spine process manager in VS code easier than spine_management.sh, but can be tempremental if use too quickly
    • beware Jira fields that spine requires that are optional on our boards (see slides / ask Jack)

2026-07-21

  • PSU CDK
    • merged 2/3 stateless PRs
    • complete? work on getting stateful stack
      • hit by SAM release failing : ‘Validation error’?
  • Apotec enablement
    • reviewed the removal of individual enablement and addition of system level enablement
    • good, but don’t merge yet

2026-07-20

  • PSU CDK
  • decision log
    • D334 - Should the sandbox environments on EPS APIs be decommissioned?
      • Matt highlighted that sandbox is a defined term by APIM, so offering something better (returning non-hard-coded values) would not meet their checkbox.
      • APIM own the text that all APIs offer a sandbox
    • D335 - Should processing to validate a proxy’s request to access a patient’s prescription data data be completed in the PfP API?
      • option 1 looks like all requesters are responsible
        • as Matt points out this is risky to trust upstream that we do not own
      • option 2 PfP (and by implication all other services that offer proxy access) are responsible
      • option 3 centralise
        • this is the only way to solve the m requesters x n APIs problem
  • AEA-6795: CPT API getting gsul results serialised as string instead of JSON

    Note: Assign and Output steps occur in parallel. If you choose to transform data during variable assignment, that transformed data will not be available in the Output step. You must reapply the JSONata transformation in the Output step. which is why I internalised it as “best use outputs to keep scope as local as possible” but I’m sure there are uses for assign – REF: https://docs.aws.amazon.com/step-functions/latest/dg/transforming-data.html

2026-07-16, 2026-07-17

  • PSU CDK
    • proxygen branch failing regression tests
      • [Ant] there should be two products per API in Apigee - one for level 3 (uses machine auth) and one for level 0 (uses api key auth)
        • so reg tests are using level 3?
      • PR
      • take stock of issue deploying eps-prescription-status-update-api (PSU) with CDK.
        • the regression tests are failing with 504 to checkprescriptionstatusupdates
          • this is most easily seen in Allure
        • try to narrow down to Apigee or AWS issue
          get_my_prescriptions has a way to connect to the AWS resources bypassing proxygen by setting the product to pws_aws.

I want to do the same to produce a version of the psu test in send_status_update.feature. Initially just running the single test tagged `@update` would be enough

so while `product=psu env=internal-dev  PULL_REQUEST_ID=pr-3215 tags=update make run-tests` fails I'd like to see if `product=psu-aws env=internal-dev  PULL_REQUEST_ID=pr-3215 tags=update make run-tests` would pass or also fail
          • new test passes on main, fails on pr, points to API Gateway auth not Apigee

            • Find both API Gateway IDs

              export AWS_REGION=eu-west-2
export AWS_PROFILE=eps-dev
aws apigateway get-rest-apis \
  | jq '.items[] | select(.name | test("psu"; "i")) | {name, id}'

            • List resources

              export AGW_ID=4gqbh730zl # psu-api-pr-3215-stateless-apigw",
export AGW_ID=6ta63oo9wi # name: psu-apigw
aws apigateway get-resources --rest-api-id $AGW_ID \
  | jq '.items[] | {path, id}'

            • Check the auth type on a resource method (e.g., the GET on checkprescriptionstatusupdates)

              export RES_ID=cp4e2r # PR "path": "/checkprescriptionstatusupdates"
export RES_ID=otw6em # main
aws apigateway get-method \
--rest-api-id $AGW_ID \
--resource-id $RES_ID \
--http-method GET \
| jq '{authorizationType, authorizerId}'

            • check the resource policy attached to the API itself (not individual methods)

              aws apigateway get-rest-api --rest-api-id $AGW_ID \
  | jq '.policy | @uri | @html | gsub("'"; "'")' \
  | python3 -c "import sys,urllib.parse,json; print(json.dumps(json.loads(urllib.parse.unquote(sys.stdin.read().strip('\"'))), indent=2))"

              this concludes there’s no significant differences (laregly SAM vs CDK)

            • check for WAF (I know there isn’t one, but check)

              aws wafv2 get-web-acl-for-resource \
  --resource-arn "arn:aws:apigateway:eu-west-2::/restapis/4gqbh730zl/stages/prod"

              no return body, return code = 0

            • check what stages exist to rule out that as an error:

              aws apigateway get-stages --rest-api-id 4gqbh730zl 
  | jq '[.item[].stageName]'

              prod was indeed correct.

            • check paths exposed

              aws apigateway get-resources --rest-api-id 6ta63oo9wi --region eu-west-2 | jq '[.items[] | {path, id, methods: (.resourceMethods // {} | keys)}] | sort_by(.path)' > apigw-resources-dev.json 
aws apigateway get-resources --rest-api-id 4gqbh730zl --region eu-west-2   | jq '[.items[] | {path, id, methods: (.resourceMethods // {} | keys)}] | sort_by(.path)' > apigw-resources-pr.json

              diff is only by id

            • check domain path mapping

              aws apigateway get-base-path-mappings \
  --domain-name psu-pr-3215.dev.eps.national.nhs.uk
{
  "items": [
    {
      "basePath": "(none)",
      "restApiId": "o2xhzquuog",
      "stage": "prod"
    }
  ]
}

              AH-HA! This is the SAM managed gateway on the PR when it should be the CDK managed one

            • redirect domain to new (CDK) gateway (temporary fix)

              aws apigateway update-base-path-mapping \
  --domain-name psu-pr-3215.dev.eps.national.nhs.uk \
  --base-path "(none)" \
  --patch-operations \
    op=replace,path=/restapiId,value=4gqbh730zl \
    op=replace,path=/stage,value=prod

              CDK fix:

              new apigw.BasePathMapping(this, 'BasePathMapping', {
  domainName: customDomain,
  restApi: yourCdkApi,
  // basePath: undefined  ← maps to root (none)
});

            • API gateway direct test (product=psu-aws env=internal-dev PULL_REQUEST_ID=pr-3215 tags=update make run-tests) still fails

            • Find the PFP stable dev API Gateway ID

              aws apigateway get-rest-apis \
  | jq '.items[] | select(.name | test("pfp"; "i")) | {name, id}'
...
{
  "name": "pfp-apigw",
  "id": "pj35kpn5g3"
}

              Get base path mappings to confirm which one is used

              aws apigateway get-base-path-mappings \
  --domain-name pfp.dev.eps.national.nhs.uk
{
  "items": [
      {
          "basePath": "(none)",
          "restApiId": "pj35kpn5g3",
          "stage": "prod"
      }
  ]
}

              Get authorizers for PFP_AWS

              export AGW_PFP=pj35kpn5g3
aws apigateway get-authorizers \
  --rest-api-id $AGW_PFP \
  --region eu-west-2 \
  | jq '.items[] | {id, name, type, authorizerUri}'

              None, and none for CDK managed PSU either.

            • move on to checking the lambdas, first PFP-AWS (pj35kpn5g3)

              aws apigateway get-resources --rest-api-id pj35kpn5g3 --region eu-west-2 \
  | jq '.items[] | {path, id, methods: (.resourceMethods // {} | keys)}'
for example, `/Bundle` has id `znlp2c`

# Get the integration (lambda ARN) for resource
export RES_ID=znlp2c
aws apigateway get-integration \
  --rest-api-id pj35kpn5g3 \
  --resource-id $RES_ID \
  --http-method GET \
  --region eu-west-2 \
  | jq '{type, uri}' 
URI produces arn:aws:apigateway:eu-west-2:states:action/StartSyncExecution, telling us it's a state machine

            • ABANDONED THIS THREAD FOR ALIGNING DOMAIN ROUTING BETWEEN STATELESS AND SPEC

            • modify stateless stack to avoid ‘-stateless’ appearing in the domain name

              • made commit 87f7bcda in eps-prescription-status-update-api (psu)
              • check domain aligned to cdk api gateway
                aws apigateway get-base-path-mappings --domain-name psu-api-pr-3215.dev.eps.national.nhs.uk --region eu-west-2 responds with the expected API gateway id (4gqbh730zl) and
              • Then smoke endpoint:
                curl -i https://psu-api-pr-3215.dev.eps.national.nhs.uk/checkprescriptionstatusupdates?... responds with `HTTP/2 200
date: Thu, 16 Jul 2026 17:48:06 GMT
content-type: application/json
content-length: 6962
x-amzn-requestid: 71557942-7005-47ad-a229-4240939a115f
lastevaluatedkey-taskid: 8f8f4052-f269-4bf7-83fd-ffd6efff3f8e
x-amz-apigw-id: AnDleG48rPEEOdw=
cache-control: no-cache
lastevaluatedkey-prescriptionid: 4CCCB7-A83008-43DA89
x-amzn-trace-id: Root=1-6a591955-6c308ae67902022611d33e9

            • and yet the tests still fail (now 404 rather than 504)

              • splunk shows gateway communicating with lambda (getting 404)
                https://nhsdigital.splunkcloud.com/en-GB/app/nhse_eps_all_sh_all_viz/search?q=search%20index%3Dapp_prescriptions_dev%20source%3D%22AWS%3ALogGroup%3A%2Faws%2Fapigateway%2Fpsu-api-pr-3215-apigw%22%20%7C%20spath%20%22message.resourcePath%22%20%7C%20search%20%22message.resourcePath%22!%3D%22%2Ffavicon.ico%22%20%7C%20spath%20%22message.resourcePath%22%20%7C%20search%20%22message.resourcePath%22!%3D%22%2F.git%2Fconfig%22%20%7C%20search%20%22message.resourcePath%22!%3D%22*php*%22%20%7C%20spath%20%22message.resourcePath%22%20%7C%20search%20%22message.resourcePath%22!%3D%22%2F%22%20%7C%20spath%20%22message.resourcePath%22%20%7C%20search%20%22message.resourcePath%22!%3D%22%2F_environment%22%20%7C%20spath%20%22message.resourcePath%22%20%7C%20search%20%22message.resourcePath%22!%3D%22%2Finfo%22%20%7C%20spath%20%22message.resourcePath%22%20%7C%20search%20%22message.resourcePath%22!%3D%22*.env*%22%7C%20spath%20%22message.apiId%22&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-60m%40m&latest=now&sid=1784279522.480825_28D52018-42BF-4D1D-9A2B-B80098E883C7
              • lambda cloudwatch shows 404 is genuine result from checkprescriptionstatus (no items found)
                {
  "level": "INFO",
  "message": "inputOutputLogger request",
  "timestamp": "2026-07-16T19:55:17.164Z",
  "service": "status",
  "cold_start": false,
  "function_arn": "arn:aws:lambda:eu-west-2:591291862413:function:psu-api-pr-3215-stateless-CheckPrescriptionStatusUpdates",
  "function_memory_size": "256",
  "function_name": "psu-api-pr-3215-stateless-CheckPrescriptionStatusUpdates",
  "function_request_id": "fd5ea0ed-39bf-4da6-887d-7937c6177d1a",
  "sampling_rate": 0,
  "xray_trace_id": "1-6a593725-5a36e3642946efb76d7939eb",
  "x-request-id": "adb4c471-4aa1-4638-a8b8-f6e284c21625",
  "apigw-request-id": "060133cd-6eb6-4021-af9e-4797e7e9d121",
  "request": {
      "response": {
          "statusCode": 404,
          "body": "{\"items\":[]}",
          "headers": {
              "Content-Type": "application/json",
              "Cache-Control": "no-cache"
          }
      }
  }

              }

            • look at the logs revealed some policy issues, starting with sqsSalt not being accessible

            • also had to complete the missing SQS ARNs

2026-07-14

  • PSU CDK
    • agreed to use proxygen and reg tests as test of stateless
    • merged Ant’s diff
    • working thru small irritations with proxygen deployment and sl-o-w github actions again
  • prep for end of sprint review
    • problem statement:
      • PfP and CPT share Get Status Updates Lambda (gsul)
      • to deliver zero downtime we need to allow that there may be multiple gsul deployments at the same time
      • so cannot any longer rely on $LATEST to choose the right one
    • state machine picture:
      • PfP state machine
    • sequence diagram of blue-green deployment:
      • PfP blue-green state machine

2026-07-13

  • PSU CDK

  • standup / planning
    • paul add me to unattended downloads chat
    • delegated access discussion of wht we may have to do something
      • Matt seems to imply we may be pssed info in order to make the final determination
        • maybe an additional spec entry? one that APIM have not anticipated seems quite likely
    • data reporting in power bi (ant mentions splunk is no longer strategic!)

2026-07-10

  • PSU CDK
    • merged the gsul PR
    • PfP update done, but blocked until gsul released, so hold fire
    • worked on stateless stack, potentially will work now, fix and review on monday
  • windows
    • discovered TeamViewer already installed
      • seems cannot use as locked down to certain users
        • TODO: can I add another user?

EPSAT usage / links

2026-07-09

  • PSU CDK
    • rework gsul PR to read PSU Table
    • recognised that wildcard change needed rework and readoption in PfP / CPT
      • start PfP adoption

2026-07-08

  • PSU CDK
    • picking up “PR to put gsul into stateless stack” from Jim
  • windows machine
    • got into a right state with one WSL under tstephen-nhs user and another under the ‘real’ user
    • told copilot to think more fundamentally about and it suddenly comes up with “mirrored networking”
      • this exposes wsl directly on the same IP removing need for ‘proxy hop’
      • faster and more robust
      • also allows us to remove the local user
  • aws console
    • discovered can no longer access from linux (this is the ‘gated aws’ convo that happened when I was away)
    • tried to set up RDP
      • enabled, but would not permit login
      • tried new local account, got further but told me it would log out ‘main’ user
        • maybe this would be ok on reflection?
      • reset domain password in case that had some escape chars, no dice

2026-07-06, 07

  • PSU CDK
    • tweaked plan
      1. PSU release to prod including gsul lambda (happening now)
      2. PfP and CPT release cut including adoption of #1 (happening imminently)
      3. PfP and CPT release testing and ultimately deployment to prod
      4. PR to enforce next env to wait on CDK as well as SAM
      5. PR to put gsul into stateless stack
      6. PR to implement remaining items of stateless stack (not including any SAM deletions yet)
      7. PR to adopt proxygen CDK style and disable SAM style
      8. next release cut, testing and ultimately release to prod (must be after #3)
  • new windows laptop
    • lots of messing around with SSH and config
    • did get a small PR made, signed and merged late Tuesday!