Started
Resolved
Duration
2 days
Update timeline
<b>This incident is now resolved.</b><br> Please follow the instructions using one of the options below if any of your Enterprise Agents cannot automatically update themselves.<br> <hr /> <h3>Option 1: Add the New DNSSEC Key</h3> Adding the new DNSSEC key can restore the Agent’s connection to the ThousandEyes platform. After the Enterprise Agent reconnects, the Test Data collected since the Enterprise Agent’s last check-in should be retroactively pushed to the Test timeline as long as the Test Data remains in the Enterprise Agent’s local cache.<br> The local cache retains Test Data for up to <b>36 hours</b> after connectivity is lost; data older than that may be permanently lost. <h4>Cisco Application-Hosting (CAF) Agent</h4> Access the CLI of the Cisco device hosting the CAF Agent.<br> Open a terminal in the app container:<br> <tt>app-hosting connect <app-id> session bash</tt><br> Append the new DNSSEC key to the key file (<em>note – this is all one line</em>):<br> <tt>echo ". 172800 IN DNSKEY 257 3 8 AwEAAa96jeuknZlaeSrvyAJj6ZHv28hhOKkx3rLGXVaC6rXTsDc449/cidltpkyGwCJNnOAlFNKF2jBosZBU5eeHspaQWOmOElZsjICMQMC3aeHbGiShvZsx4wMYSjH8e7Vrhbu6irwCzVBApESjbUdpWWmEnhathWu1jo+siFUiRAAxm9qyJNg/wOZqqzL/dL/q8PkcRU5oUKEpUge71M3ej2/7CPqpdVwuMoTvoB+ZOT4YeGyxMvHmbrxlFzGOHOijtzN+u1TQNatX2XBuzZNQ1K+s2CXkPIZo7s6JgZyvaBevYtxPvYLw4z9mR7K2vaF18UYH9Z9GNUUeayffKC73PYc=" >> /usr/lib/te-agent/dns-rootserver.key</tt> <h4>Docker Agent</h4> Run the following command (<em>note – this is all one line</em>) on the Docker host, replacing <tt><CONTAINER_NAME></tt> with the Agent’s container name:<br> <tt>docker exec <CONTAINER_NAME> sh -c 'echo ". 172800 IN DNSKEY 257 3 8 AwEAAa96jeuknZlaeSrvyAJj6ZHv28hhOKkx3rLGXVaC6rXTsDc449/cidltpkyGwCJNnOAlFNKF2jBosZBU5eeHspaQWOmOElZsjICMQMC3aeHbGiShvZsx4wMYSjH8e7Vrhbu6irwCzVBApESjbUdpWWmEnhathWu1jo+siFUiRAAxm9qyJNg/wOZqqzL/dL/q8PkcRU5oUKEpUge71M3ej2/7CPqpdVwuMoTvoB+ZOT4YeGyxMvHmbrxlFzGOHOijtzN+u1TQNatX2XBuzZNQ1K+s2CXkPIZo7s6JgZyvaBevYtxPvYLw4z9mR7K2vaF18UYH9Z9GNUUeayffKC73PYc=" >> /usr/lib/te-agent/dns-rootserver.key'</tt> <h4>Linux Package Agent</h4> Append the new DNSSEC key to the key file (<em>note – this is all one line</em>):<br> <tt>echo ". 172800 IN DNSKEY 257 3 8 AwEAAa96jeuknZlaeSrvyAJj6ZHv28hhOKkx3rLGXVaC6rXTsDc449/cidltpkyGwCJNnOAlFNKF2jBosZBU5eeHspaQWOmOElZsjICMQMC3aeHbGiShvZsx4wMYSjH8e7Vrhbu6irwCzVBApESjbUdpWWmEnhathWu1jo+siFUiRAAxm9qyJNg/wOZqqzL/dL/q8PkcRU5oUKEpUge71M3ej2/7CPqpdVwuMoTvoB+ZOT4YeGyxMvHmbrxlFzGOHOijtzN+u1TQNatX2XBuzZNQ1K+s2CXkPIZo7s6JgZyvaBevYtxPvYLw4z9mR7K2vaF18UYH9Z9GNUUeayffKC73PYc=" >> /usr/lib/te-agent/dns-rootserver.key</tt><br> <b>Please note:</b> Customers will still need to complete Option 2 for a long-term fix, even if Option 1 is applied in the interim. <hr /> <h3>Option 2: Upgrade the Agent Image</h3> <h4>Cisco Application-Hosting (CAF) Agent</h4> Run the following command on the Cisco device hosting the CAF Agent:<br> <b>If you have x86 architecture:</b><br> <tt>app-hosting upgrade appid {application-name} package https://downloads.thousandeyes.com/enterprise-agent/thousandeyes-enterprise-agent-x86_64-5.2.1.cisco.tar</tt><br> <b>If you have ARM architecture:</b><br> <tt>app-hosting upgrade appid {application-name} package https://downloads.thousandeyes.com/enterprise-agent/thousandeyes-enterprise-agent-aarch64-5.2.1.cisco.tar</tt><br> <b>Please note:</b> Upgrading the Cisco App-Hosting Image (CAF) Agent image will clear the Enterprise Agent’s local Test Data cache. The Test timeline will have a data gap for the affected period, and the cached Test Data will not be backfilled. <h4>Docker Agent</h4> Use the <tt>latest</tt> image tag (or <tt>latest-agent</tt> if you are not using BrowserBot), or Docker image version <tt>0.105.0</tt> or later.<br> <tt>docker pull <name></tt> <tt>docker stop <running-agent></tt> <tt>docker rm <running-agent></tt> <tt>docker run <same options as previous launch></tt><br> <h4>Linux Package Agent</h4> Upgrade the <tt>te-agent</tt> package to the latest version (<tt>1.247.2</tt>) using the system package manager (<tt>yum</tt>, <tt>dnf</tt>, or <tt>apt</tt>, depending on your distribution).<br> For more details, see <a href="https://docs.thousandeyes.com/product-documentation/global-vantage-points/enterprise-agents/managing/upgrading-enterprise-agents">Upgrading Enterprise Agents</a>.
New updates are being pushed for Enterprise Agents. Enterprise Agents showing online are out of scope. Customers with offline Enterprise Agents should reach out to Standard ThousandEyes Support Channels to manually update configurations.
A fix has been implemented. Operations will continue to monitor performance.
We are continuing to investigate this issue.
Cloud and enterprise agents are showing failure message in DNSSEC tests due intermediate validation. The issue started at 14:00 PM UTC.