I've been working on a TFVC-to-Git migration for a client lately and part of that work is getting their code building on Azure Pipelines. They're on Azure DevOps Server (on-prem) and they've got a private NuGet feed in Azure Artifacts that everything depends on. So I set up a self-hosted build agent, wrote a pipeline, queued it up and...it failed. And then I fixed it and it failed again.

Here's the thing. Both of those failures looked like the same problem -- "the agent can't talk to the feed" -- but they were two completely different problems in two completely different layers. And neither of them was caused by the thing I first suspected, which was the account the agent runs as. If you're setting up a self-hosted agent against a private feed and you're stuck, there's a decent chance you're stuck on one of these.

The Short Version

• "unable to get local issuer certificate" from a pipeline task is a Node.js trust problem. Node ignores the Windows certificate store. Export your internal CA chain to a PEM file and point NODE_EXTRA_CA_CERTS at it in the agent's .env file.
• A 403 with "You need to have 'ReadPackages'" is a feed permission problem, but for the identity the restore actually presents. With NuGetAuthenticate in the pipeline, that's Project Collection Build Service, not your agent's service account. Read the task's log line, then grant that identity on the feed.
• Neither problem is caused by running the agent as Network Service. The account only changes where the environment variable goes and which identity would be used as a fallback.
• There's no az devops command for feed permissions. It's the REST API or the web UI.

The Setup

Quick sketch of the environment so the rest of this makes sense:

• Azure DevOps Server, on-prem, with an HTTPS endpoint signed by the company's internal certificate authority
• A self-hosted Windows agent (let's call it BUILD-01) running as a Windows service under the Network Service account
• An Azure Artifacts feed (let's call it PrivateFeed) that the solution restores packages from
• A pipeline that does the usual NuGetAuthenticate task, then an MSBuild step that does -restore and builds the solution

On my own workstation, everything worked. I could restore, build, and push packages to the feed with nothing but my Windows login. So I figured the agent would be the same. It was not the same.

Problem #1: "unable to get local issuer certificate"

First run. The NuGetAuthenticate task fails with:

##[error]Error: unable to get local issuer certificate

My first thought was "this is a credentials thing." It's not. That message is a TLS error and it's coming from Node.js.

A bunch of the built-in Azure Pipelines tasks are written in JavaScript and run on the Node runtime that ships with the agent. NuGetAuthenticate is one of them. And here's the thing about Node: it doesn't use the Windows certificate store. It ships with its own list of trusted root certificate authorities and it ignores whatever Windows trusts.

So on that build box, Windows trusted the company's internal CA (that's why Git worked, that's why tf.exe worked, that's why MSBuild worked). But Node had never heard of it. When the task tried to talk to https://devops.contoso.local, Node looked at the certificate, couldn't find the issuer in its own list, and bailed.

The fix is to hand Node the CA certificate chain via an environment variable called NODE_EXTRA_CA_CERTS. You need the CA chain as a PEM file. If you don't have one lying around, you can pull it straight off the server:

$tcp   = New-Object Net.Sockets.TcpClient('devops.contoso.local', 443)
$ssl   = New-Object Net.Security.SslStream($tcp.GetStream(), $false, ({ $true }))
$ssl.AuthenticateAsClient('devops.contoso.local')
$leaf  = New-Object Security.Cryptography.X509Certificates.X509Certificate2($ssl.RemoteCertificate)
$chain = New-Object Security.Cryptography.X509Certificates.X509Chain
$chain.Build($leaf) | Out-Null

# everything above the leaf: the intermediate(s) and the root
$pem = foreach ($e in $chain.ChainElements | Select-Object -Skip 1) {
    "-----BEGIN CERTIFICATE-----`n" +
    [Convert]::ToBase64String($e.Certificate.RawData, 'InsertLineBreaks') +
    "`n-----END CERTIFICATE-----"
}
$pem -join "`n" | Set-Content C:\agent\contoso-ca.pem -Encoding ascii

Now, where do you set the environment variable? This is where the Network Service part matters. If you set NODE_EXTRA_CA_CERTS as a user environment variable, the agent service will never see it because the service isn't running as you. You could set it as a system-wide variable but the cleaner way is to use the agent's .env file. It's a plain text file in the agent's root folder (next to .agent and .credentials) and the agent reads it at startup.

NODE_EXTRA_CA_CERTS=C:\agent\contoso-ca.pem

Restart the agent service and every Node-based task in every pipeline picks it up.

(Side note: if you can't find the .agent file, it's there. It's a dotfile so Explorer and dir hide it. Try Get-ChildItem C:\agent -Force.)

Was this problem caused by running as Network Service? No. Node would have failed the same way under any account. Running as a service just changed where the fix had to go.

Problem #2: 403 Forbidden, "You need to have ReadPackages"

Second run. NuGetAuthenticate succeeds this time. Then the restore fails:

error NU1301: Unable to load the service index for source https://devops.contoso.local/tfs/DefaultCollection/MyProject/_packaging/PrivateFeed/nuget/v3/index.json.
  Response status code does not indicate success: 403 (Forbidden - User 'Build\c9c98cf9-...' lacks permission to complete this action. You need to have 'ReadPackages'.)

Ok. This one actually is a permissions problem. But the interesting question is: permissions for whom?

My first instinct was "the agent runs as Network Service so I need to grant the machine account access to the feed." That's wrong. Look at what the NuGetAuthenticate task printed when it succeeded:

Setting up the credential provider to use the identity 'Project Collection Build Service (DefaultCollection)' for feeds in your organization/collection starting with:
  https://devops.contoso.local/tfs/DefaultCollection/

That's the whole answer right there. The task configures the NuGet credential provider to present the pipeline's identity -- the build service account, the same thing that System.AccessToken represents. Network Service never enters the conversation. The machine account never enters the conversation. The feed sees a request from Project Collection Build Service and that identity isn't a member of the feed.

The fix: in Azure DevOps, go to Artifacts, pick the feed, click the gear, go to Permissions, and add Project Collection Build Service (YourCollection) as a Contributor. (Reader is enough to restore; Contributor is what you'll need when the pipeline starts pushing packages, so just do it once.) You'll need to be an Owner on the feed to do this.

Now, here's a wrinkle that I think is worth understanding. If I had not used the NuGetAuthenticate task, the restore would have fallen back to Windows integrated authentication and the identity it presented would have been the machine account (CONTOSO\BUILD-01$), because that's what Network Service looks like on the network. That account would have needed the feed permission instead.

So the account the agent runs as determines the fallback identity. The NuGetAuthenticate task determines whether the fallback even gets used. Either way, the thing to do is read the log, see which identity is actually being presented, and grant that one. Don't guess.

Was it Network Service or was it the certificate?

I asked myself this after the fact because I wanted to know what to write in the client's runbook. The answer is that neither one was "the" cause.

• The certificate problem was about Node's trust store. Any account, same failure.
• The 403 was about the identity the client presented to the feed. The NuGetAuthenticate task decided that, not the service account.

Two layers. TLS trust and authorization. What made it confusing is that they surfaced at the exact same spot in the log, one right after the other, and both of them could be summarized as "the build can't get its packages."

Bonus: is there a CLI for the feed permissions?

Not really, and I looked. az devops doesn't have a command for Azure Artifacts feed permissions and the az artifacts extension is only for universal packages. What you've got is the REST API:

PATCH {collection}/{project}/_apis/packaging/feeds/{feedId}/permissions?api-version=7.1
[
  { "identityDescriptor": "<descriptor of the build service identity>", "role": "contributor" }
]

...and the annoying part is getting the identity descriptor, which is a second call to the identities API. For a one-time setup, the web UI is faster. If you're doing it for a lot of feeds, script the REST call.

Summary

If your self-hosted agent can't restore from your private Azure Artifacts feed, check these in order:

1. unable to get local issuer certificate from a task = Node.js doesn't trust your internal CA. Export the CA chain to a PEM file and set NODE_EXTRA_CA_CERTS in the agent's .env file. Restart the agent service.
2. 403 / ReadPackages on restore = the identity the restore is presenting isn't on the feed. Read the NuGetAuthenticate log to see which identity that is (it's almost certainly Project Collection Build Service, not your service account), and grant that identity Contributor on the feed.
3. Neither of these is caused by the account the agent runs as. The account only changes where the environment variable goes and which identity gets used as a fallback.

I hope this helps.

-Ben

If you're trying to get an old codebase onto a modern build pipeline, or your code is still on TFVC and the pipeline is the least of your worries, that's the kind of work I do. Let's talk.