May logbook

2023-05-31

• review ADFS -> IAC impact doc
• convo with IPI about skipper
• ceip-3653:
  • convo with Ashish, happy with direction

2023-05-30

• New Relic Distributed Tracing: Tracking Across your Application Stacks
• LOATHED: Office day to discuss release automation when still waiting for Argo 7 months on
• RDP
  • commit: 3b032eb2e937b73ffb504e3f58fde3572c1cb21c needs reconciliation
  • LOATHED: SQS names not match platform class, and opaque, with no dead-letter 12 months after conversation started
  • Another new account not correctly bootstrapped LOATHED: get PR approved and still required to run terraform apply manually (https://github.com/elsevier-centraltechnology/tio-terraformcontrol-ce/pull/957 even contained a bug demonstrating the false sense of security provided)
  • Already bootstrapped! (https://global-elsevier.slack.com/archives/CDHK7D9UL/p1684490972290679), query terraformed without commit?
• Actions:
  • runbook entry for monitoring release
  • direct links for reconciler status and recent reconciler logs (on Ops page or Dashboard?)

2023-05-26 - Sick: fever then gastro

2023-05-25

• create new id-prod-use cluster

2023-05-24

• issue with isdp-dev-alpha skipper
• create new id-prod-use cluster
  • need procedure for creating cluster
    • check platform param created
    • check nr params created
    • use account api to find account name from id curl -s -H 'x-api-key: TOKEN' https://api.ce.tio.systems/accounts//151574817355 | jq
    • add both to: https://github.com/elsevier-centraltechnology/tio-terraformcontrol-ce/blob/master/183742092277/eu-west-1/logging-platform/terraform.tfvars#L45
    • create pr, get approved, merge
    • apply terraform (manually!) and of course it failed │ Error: Missing required argument │ │ The argument “account_id” is required, but was not set.
    • new 2 NR env vars (see README 2 dirs up)
• TIO town hall
  • Azure OpenAI Studio: train and export to python, curl etc

2023-05-23

• proposal for crtxctl nexxt steps
• more fiddling with bom
• cloud trail meeting
  • useful for finding the stuff aws does behind the scenes
  • lasts 90 days with [small?] chance ISDP can go back further if absolutely needed
  • stored in s3
  • account local only
  • no fine grained access controls
  • finally… appears we’re trying to debug permission boundary failures

2023-05-22: vacation day

2023-05-18

• science direct
  • Gabriel Bonsoir:
  • Frank Borg (reports to Kevin) product development side
    • wants to create project and get buy in
    • 90 microservices
  • Timeframe
    • WAG, TPR1, buy in end of May
  • Capabilities
    • need to contaierise
    • stage 0 need to flush out needed capabilities
  • Motivation
    • EC2 managed by Ansible (large and hard to manage)
    • 20 mins of pipeline for change
  • confluence/SDX/Developer Tim Overhead
  • Architect: Terry and Derek
  • Ordering:
    • 6? candidate services identified
    • some nodejs

2023-05-16

• completed vulnerability review

• how could vulnerabilities be checked more systematically?

  • read the ‘bom’
  • get the latest app release from author url
  • if it exists: get the latest helm chart from publisher / supplier url
  • compare latest to current in platform definitions
  • create a ticket to evaluate the change for compatibility because semver is not reliable
  • OR create a branch with the change (if this is doable)

• release requirements

  • a brake if something does not go well (rather than log error to NR and continue)
  • a post reconcile check

• system test requirements

  • three things: drift test, smoke test, load test
  • assigned to Luis & Ashish

2023-05-12

• ceip-3114 skipper

• prisma vulnernabilities

  • nri-bundle 5.0.4 -> 5.0.12: https://github.com/newrelic/helm-charts/compare/nri-bundle-5.0.4...nri-bundle-5.0.12
    • k8s-events-forwarder:1.36.1 : 3 : https://hub.docker.com/layers/newrelic/k8s-events-forwarder/1.41.0/images/sha256-20e9973191da132ae8f3649b1ad00bb58a7b66df8dbc66dffb21fd7b001949d2?context=explore
    • infrastructure-bundle:2.8.38
  • skipper:v0.14.6: https://github.com/zalando/skipper/compare/v0.14.6...v0.16.44
  • amazon-k8s-cni:v1.12.6-eksbuild.1: vulnerable but latest available according to https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
  • aws-ebs-csi-driver: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/compare/v1.16.0...v1.18.0
    • csi-attacher:v4.1.0-eks-1-25-latest: Part of https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html?
    • csi-node-driver-registrar:v2.7.0-eks-1-25-latest: ditto
    • csi-provisioner:v3.4.0-eks-1-25-latest: ditto
    • csi-resizer:v1.7.0-eks-1-25-latest
    • csi-snapshotter:v6.2.1-eks-1-25-latest: ditto
  • coredns: https://github.com/coredns/coredns/compare/v1.8.7...v1.10.0
  • kube-proxy:v1.23.16-minimal-eksbuild.2: vulnerable but latest according to https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html
  • livenessprobe:v2.9.0-eks-1-25-latest: https://github.com/kubernetes-csi/livenessprobe/compare/v2.9.0...v2.10.0
  • kube-secret-inject-webhook:4.1.0: latest available, needs go 1.20.3
  • aws-efs-csi-driver:v1.4.3: https://github.com/kubernetes-sigs/aws-efs-csi-driver/compare/v1.4.3...v1.5.5 (dates back to Nov ‘22)
  • vertical-pod-autoscaler@6.0.0: https://github.com/cowboysysop/charts/compare/vertical-pod-autoscaler-6.0.0...vertical-pod-autoscaler-7.0.1
    • vpa-admission-controller:0.11.0 (bom is already 0.12.0)
  • aws-node-termination-handler:v1.19.0 is the latest
  • aws-for-fluent-bit: https://github.com/aws/aws-for-fluent-bit/compare/v2.31.2...v2.31.10
    • ticket to PR upstream to catch up versions and then adopt it
  • container-collector:1308 - latest according to https://hub.docker.com/r/cloudhealth/container-collector/tags/#!
  • external-dns:v0.13.1: https://github.com/kubernetes-sigs/external-dns/compare/v0.13.1...v0.13.4
    • https://github.com/bitnami/charts/blob/main/bitnami/external-dns/Chart.yaml, chart vsn 6.20.1 equates to 0.13.4
  • kyverno:v1.7.5: https://github.com/kyverno/kyverno/compare/v1.7.5...v1.9.3
  • metrics-server:v0.6.2: https://github.com/kubernetes-sigs/metrics-server/compare/v0.6.2...v0.6.3
    • update our helm chart 1.1.2:
  • csi-node-driver-registrar:v2.6.0: https://github.com/kubernetes-csi/node-driver-registrar/compare/v2.6.0...v2.8.0
    • Liam is in the process of rolling out 2.7.0 as part of csi secrets
  • operator:v1.28.1: https://github.com/tigera/operator/compare/v1.28.1...v1.28.12
  • kube-ingress-aws-controller:v0.14.1: https://github.com/zalando-incubator/kube-ingress-aws-controller/compare/v0.14.1...v0.14.15
  • defender:defender_22_12_582: Twistlock
  • tigera:v3.24.1: -> 3.25.1: https://docs.tigera.io/calico/latest/release-notes/

• planning

  • 3114: skipper
  • 3702 (khush needs help)

2023-05-11

• ceip-3653 sbom

2023-05-10

• iso documentation
• need to figure out why cws missed (handover from Ashish)

2023-05-09

• planning

  • ceip-3672
  • PPE

• iso

2023-05-03 -05

• ISO audit prep

2023-05-02

• meetings