Unfortunately, AFAIK, this is not supported. Windows DNS natively uses GSS-TSIG, which relies on Kerberos and the Active Directory environment to securely negotiate keys and authenticate dynamic updates.
As a workaround, you can potentially try the following:
- Secure by IP Access Control Lists (ACLs) - Instead of cryptographic transaction signatures, restrict zone transfers strictly by target IP addresses. On the Windows primary zone, configure Zone Transfers to "Only to the following servers" and explicitly type the IP addresses of your non-Windows secondary nodes.
- Network-Layer Security (IPsec) - if the zone data must be authenticated or encrypted over transit networks, handle the security at the packet layer instead of the application layer.
If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.
hth
Marcin