Skip to content

[Bug] Deleting an application script routing rule makes healthy providers unavailable #16548

Open

Description

@xiaobaicai66695

Pre-check

  • I am sure that all the content I provide is in English.

Search before asking

  • I had searched in the issues and found no similar issues.

I also searched for AppScriptStateRouter DELETED, script router rule delete, and No provider available script routing. #16545 covers a separate cross-service router-chain bug; it is not required to reproduce this failure.

Apache Dubbo Component

Java SDK (apache/dubbo)

Affected module: dubbo-cluster; affected router: AppScriptStateRouter.

Dubbo Version

  • Apache Dubbo Java SDK 3.3.6 from Maven Central. The error also reproduces in the unmodified 3.3 branch at commit 843b98b using a targeted regression test.
  • Amazon Corretto JDK 1.8.0_442 (bundled Nashorn JavaScript engine), Windows 11 build 26200.
  • Nacos 2.5.2 standalone as the configuration center, mapped to 127.0.0.1:18848 (HTTP) and 127.0.0.1:19848 (gRPC).
  • Curator TestingServer 5.9.0 as the ZooKeeper registry at 127.0.0.1:23281.
  • Two provider JVMs and one consumer JVM. No Spring application is needed.

Steps to reproduce this issue

  1. Start Nacos and ZooKeeper. The local Nacos command was docker run -d --name codex-dubbo-script-repro-nacos -e MODE=standalone -e NACOS_AUTH_ENABLE=false -p 127.0.0.1:18848:8848 -p 127.0.0.1:19848:9848 nacos/nacos-server:v2.5.2. Start two separate provider JVMs for the same OrderProbeService: order-v1 at 127.0.0.1:23981 and order-v2 at 127.0.0.1:23982. Both use provider application name route-chain-repro-order. The method String instance() returns the instance ID.

  2. Publish the following YAML to Nacos group dubbo, Data ID route-chain-repro-order.script-router:

    configVersion: v3.0
    scope: application
    key: route-chain-repro-order
    runtime: true
    enabled: true
    force: true
    type: javascript
    script: |
      (function route(invokers, invocation) {
          var result = new java.util.ArrayList();
          for (var i = 0; i < invokers.size(); i++) {
              if (invokers.get(i).getUrl().getPort() == 23982) {
                  result.add(invokers.get(i));
              }
          }
          return result;
      })(invokers, invocation);
  3. Start a single-reference consumer JVM using the same ZooKeeper registry and Nacos config center. Set roundrobin load balancing and retries=0 on the Order reference. Invoke instance() five times; all five calls return order-v2.

  4. In Nacos, delete that Data ID with ConfigService.removeConfig("route-chain-repro-order.script-router", "dubbo"). Confirm the Nacos HTTP config endpoint returns 404, wait five seconds for the config notification, then invoke instance() five more times on the same consumer reference.

  5. Republish the same YAML, wait five seconds, and invoke instance() five more times.

The rule lifecycle in the reproducer is:

observe("BEFORE_DELETE", order, 5);
config.removeConfig("route-chain-repro-order.script-router", "dubbo");
waitUntilNacosGetReturns404();
Thread.sleep(5000);
observe("AFTER_DELETE", order, 5);
config.publishConfig("route-chain-repro-order.script-router", "dubbo", rule);
Thread.sleep(5000);
observe("AFTER_RESTORE", order, 5);

Each observe call counts the returned provider IDs and caught exceptions. Providers remain registered and healthy throughout. The same failure was also confirmed with a second UserProbeService reference in the consumer when using the independent router-chain fix proposed in PR #16546.

What you expected to happen

Deleting the script rule should disable script filtering. While the rule is absent, both healthy Order providers should remain available for normal load balancing. Republishing the rule should restore v2-only routing. No RPC should fail solely because the rule was deleted.

What actually happened

The original Dubbo 3.3.6 single-reference run returned:

BEFORE_DELETE={order-v2=5}
NACOS_HTTP_AFTER_DELETE=404
AFTER_DELETE={ERROR:RpcException=5}
AFTER_RESTORE={order-v2=5}

The two-reference run with the separate #16545 chain fix had the same 5/5 errors after deletion. The consumer log contains java.lang.NullPointerException at AppScriptStateRouter.doRoute(AppScriptStateRouter.java:69), followed by Failed to execute router and No provider available. The failure happens in a single-reference consumer without the #16545 fix, so the chain fix is not its cause.

Anything else

The DELETED handler in AppScriptStateRouter.process sets scriptRule = null but leaves the compiled scriptRouter non-null. On the next call, doRoute passes the scriptRouter == null check and dereferences scriptRule.isValid(). DynamicDirectory.doList catches the routing exception and returns an empty candidate list, so the cluster reports that no provider is available even though both providers are healthy.

A fix should publish the rule and compiled router together as one state, and clear both on deletion. This also avoids observing mismatched state when config updates race with invocation routing. A regression test can assert routing before deletion, pass-through after deletion, and restored script routing after republishing.

I tested that approach locally against real RPCs using the original Dubbo 3.3.6 router factory and, separately, the #16545 router-chain fix. In both setups, after deletion the five calls reached both providers (order-v1=2, order-v2=3) with zero errors; after republishing, all five reached order-v2. The targeted regression test initially failed with the same NPE on unmodified 3.3 source and passed after the fix.

Do you have a (mini) reproduction demo?

  • Yes, I have a minimal reproduction demo to help resolve this issue more effectively!

The local reproducer uses separate provider and consumer JVMs, an embedded ZooKeeper server, Nacos 2.5.2, the YAML above, and the lifecycle code shown above. The provider/consumer setup is the same as the complete runnable source in #16545; the additional dynamic deletion sequence is shown here. PR #16549 includes a regression test.

Are you willing to submit a pull request to fix on your own?

  • Yes I am willing to submit a pull request on my own!

The fix and regression test are submitted in PR #16549.

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions