Zerto 10 Role-based Access Controls via Keycloak

If you’re still on Zerto 9.7 or lower on the Windows Zerto Virtual Manager and have been asking for better role-based access controls (RBAC) for Zerto, then you need to get migrated over to the new Zerto Virtual Manager Appliance (ZVMA)!

About the Zerto Virtual Manager Appliance

The Linux-based Zerto Virtual Manager Appliance (ZVMA) made its debut in Zerto 9.5, and has since become the standard going forward with Zerto, as the last Windows version (of the ZVM) was 9.7. In Zerto 10, there is no Windows ZVM, so migration is now on the table and I’d highly recommend going that route to to prevent being left behind (and I will go more into detail about that in another blog post).

In addition to the underlying OS changing, came a modernization of how the ZVM has been architected. Instead of running everything as a single (or maybe a few) Windows services, Zerto has been built to run as containers on top of MicroK8s on a hardened Debian 11 virtual appliance. Please also note that because it’s Debian 11, the minimum vSphere version that supports it is vSphere 7.x.

That said – there is no separate software package to download and install; the ZVMA is now a fully-packaged OVF that you just deploy in vSphere. The best part is once it’s deployed, you’re ready to use it. This fundamental change on how Zerto has been built also introduced the ability to provide more frequent updates (quarterly) and virtually no disruption as each container can be updated independently without having to disrupt the entire functionality of the ZVM.

Now back to why you’re here…

While in the older versions of Zerto, there were some basic role-based access controls, they relied on vSphere roles, which meant that anyone who needed to log into Zerto would need to have credentials to log onto the vCenter client. This has all changed once you’ve entered the world of the Linux ZVM.

Instead of relying on vSphere permissions for each user, Zerto now has it’s own authentication services built on Keycloak (https://www.keycloak.org/), which provides you with a more secure posture when it comes to safeguarding your ability to recover from something as disruptive as a ransomware attack.

By removing the reliance on vSphere logins (which have typically been integrated to Active Directory), the chances of an elevated AD account becoming compromised will not affect Zerto’s operation because there is no dependency on those logins to get into Zerto. Not even the service account Zerto uses to manage API calls to vCenter can affect Zerto, because it’s not even managed by Zerto. While we’re on that subject, the ZVMA also supports MFA for added security. Additionally, you get to keep tighter grips on who actually has access and can log into vSphere while making sure your recovery environment stays protected/isolated.

Configure Role-based Access Controls in Zerto 10

In this section, I’ll cover what the role-based access controls looks like, what roles and permissions are involved, and how to set a user up and grant the correct roles, because when I first went through this, I didn’t find it as intuitive; so hopefully this helps if anyone reading finds themselves in a similar situation.

Note that before doing this, the assumption is that you’re already familiar with deploying the Linux Zerto Virtual Manager (OVF deployment via vCenter) and have already gone through and changed default passwords as well as paired to your vCenter. If you haven’t done that and need the information to do so, visit https://help.zerto.com for the deployment guide.

Also, this is not the guide for configuring Keycloak for any other integration such as Active Directory or Okta, for example. This is simply using accounts local to the ZVMA (in Keycloak). For other supported integration, visit the Zerto documentation at: https://help.zerto.com

Enable Roles and Permissions

Once you’ve completed the pre-requisite steps above, log onto the Zerto Management page at https://[yourZVMAIPAddress]/management. You must do this in order to leverage the Zerto Roles and Permissions through Keycloak.

  1. In the management interface, click on Security & RBAC on the left navigation bar.
  2. Enable the radio button for “No Access” under Roles & Permissions

    Enabling Roles & Permissions

Create a Keycloak User and Configure Permissions

  1. Log onto the Keycloak administration UI at https://[yourZVMAIPAddress]/auth.
  2. Once logged in, click on the realm dropdown menu and switch from master to zerto.

    Changing the realm to zerto realm in Keycloak
  3. Click on Users on the left navigation bar, and then click the Add user button.

    Add a Keycloak user to the zerto realm
  4. In the create user window, set actions as needed, such as update password (change password upon initial logon) or any other options you require. Click Create when done.

    Keycloak create user dialog
  5. You should now see the user details and several tabs across the top. Click on Role mapping.

    Role mapping in user details in Keycloak
  6. Click the Assign role button

    Assign role in Keycloak
  7. At first glance, don’t worry if you don’t see any Zerto roles. (This is what got me and wasn’t clearly identified in the documentation). Click on the filter dropdown menu on the top left, and select Filter by clients.

    Filter by clients selection in Keycloak
  8. You will now see a full list and a section tagged zerto-client. From that section, select the required roles for your user, and click the Assign button at the bottom.

    Zerto roles listed in Keycloak
  9. You will now see the role(s) assigned to the user.

    Assigned role to user in Keycloak
  10. Finally, before the user can try logging in, click on the Credentials tab at the top, and set the password.

    Set the user's password in Keycloak

Managing Zerto Roles by Using Groups

Maybe you don’t want to manage roles and permissions on a per-user basis, especially at scale. Besides, it’s a best practice to use groups for role management so you can simply add users to them down the road without having to repeat the steps above for each user.

So, if your preferred method to manage roles is by group, you can skip the steps above, and follow these steps and be on your way. Just remember, when you set users up, you still have to set the initial password and other options before they can login.

  1. If you’re not already logged into Keycloak, login at https://[yourZVMAIPAddress]/auth.
  2. Change from the master realm (dropdown on the top left) to the zerto realm.
  3. Click on Groups under the Manage section on the left
  4. Click the Create group button.

    Create a group in Keycloak
  5. Provide a name for your group and click Create

    Create a group in Keycloak
  6. Click on the group you just created.

    Group Created in Keycloak
  7. Click on the Role mapping tab at the top, and click Assign Role

    Assign Role to group in Keycloak
  8. Click on the filter dropdown and select Filter by clients.

    Filter by clients in Keycloak
  9. Scroll down the list to the area tagged zerto-client and select the role(s) you wish to apply to the group you just created. When done, click Assign.

    zerto-client roles in Keycloak
  10. Now, add members to the group (if you have previously created users – otherwise, create users and then add them to the group). Click on the Members tab, and click Add member.

    Add members to group in Keycloak
  11. Select the users to add to the group as members, and click the Add button to finish.

Summary

Managing Zerto users in Zerto 10 via Keycloak doesn’t have to be difficult. It’s quite easy, actually, especially when assigning roles at the group level. By assigning different roles to different users depending on what they need access to be able to do, you’re not only exercising better access controls with Zerto, but you are also providing better security, able to create responsibilities for others without giving them any vSphere permissions, and also reducing your own operational/administrative overhead.

Now the question is whether or not to integrate with Active Directory – that is totally up to you. I’m going to leave you with this piece of advice though. Zerto 10 was built with Keycloak to isolate authentication and provide better security when it comes to recovering from cyberthreats. By choosing not to integrate with AD, there is no other way for bad actors to access Zerto, therefore giving you a better chance at quickly turning the tables on them and recovering to a point in time before any malware/ransomware took over. Zerto 10 also introduced in-line encryption detection, so your protected workloads will have a built-in early warning system, so you’ll be able to not only react faster, but be notified before all hell breaks loose.

Let me know your thoughts in the comments, and feel free to ask me any questions about what was shared here.

I will be working on additional Zerto 10 content, so stay tuned!

Share This:

Update: Migrate VM from Hyper-V to vSphere with Pre-Installed VMware Tools (vSphere 7 and 8 Edition)

I had previously written a post in response to a problem a customer was facing with migrating from Microsoft Hyper-V to VM vSphere.

You can find that previous post here: Migrate VM from Hyper-V to vSphere with Pre-Installed VMware Tools

I am writing this as a follow-up, because while the workaround I documented still works (for vSphere 6.x VMware Tools), something with the VMware Tools had changed when vSphere 7 went GA.  Several attempts to manipulate the new .msi file proved to not work, and in the flurry of life, I hadn’t had a chance to really sit down and figure it out.  So, the workaround for “now” was to install the working 6.x version, get migrated, and then upgrade VMware Tools; and that still works, by the way.

Then one day, I was going through my blog comments someone had responded, saying they’d figured it out.  @Chris, thank you very much for sharing your find!

So, since vSphere 8 recently went GA, I figured I’d also test this procedure on VMware Tools 12, and I’m happy to say, it also works.  So here’s what’s changed from the previous post when you’re trying to do the same using VMware Tools 11 (vSphere 7) or VMware Tools 12 (vSphere 8).

What You Will Need

Before you can get started, you’ll need to get a few things.  For details on how to get these requirements, refer to the original post mentioned above. 

  • Microsoft Orca (allows you to edit .msi files) – This is part of the Windows SDK, so if you don’t have it, see the post referenced above for the link to download as well as the procedure to only install Orca.
  • VMware Tools 11 or 12
  • Visual C++ 2017 Redistributable (if you’re following the procedure to get the VMware Tools from your own system, be sure to grab the vcredist_x64.exe)

If you would like to skip editing the VMware Tools MSI, you can download already “jailbroken” versions below. 

Note: These worked in the testing I performed, and I will not be making any changes to them, supporting them, or be responsible for what you download off of the Internet.  To be absolutely sure you have complete control over what you install in your environment (ESPECIALLY IN PRODUCTION), download from trusted sources and perform the edit to the MSI yourself.

Edit VMware Tools MSI with Orca (for VMware Tools 11 and VMware Tools 12)

  1. Launch Orca
  2. Click Open, and browse to where you saved VMware Tools64.msi, select it, and click Open.

    Launch Orca and Open VMware Tools MSI

  3. In the left window pane labeled Tables, scroll down and click on CustomAction.
  4. In the right window pane, look for the line that says VM_LogStart, right-click it, and select Drop Row.
  5. When prompted, click OK to confirm.


  6. In the left window pane labeled Tables, scroll down and click on InstallUISequence.
  7. In the right window pane, look for the line that says VM_CheckRequirements. Right-click on this entry, and select Drop Row.
  8. When prompted, click OK to confirm.

    InstallUISequence > VM_CheckRequirements > Drop Row

  9. Click save on the toolbar, and close the MSI file. You can also exit Orca now.

Next Steps

Now that you’ve successfully edited the MSI file to be able to be installed on your Hyper-V Windows VMs, copy the installers (don’t forget vcredist_x64.exe) and install.  When it asks for a reboot, you can safely ignore it, because once the VM boots up in vSphere, it would have already taken care of that for you.  (One less disruption to your production Hyper-V virtual machine).

Thanks for reading! GLHF

If you found this useful and know of any others looking to do the same, please share and comment.  I’d like to hear if/how it’s helped you out! If you’d like to reach me on social media, you can also follow me and DM me on Twitter @eugenejtorres

Share This:

Reduce the Cost of Backup Storage with Zerto 8.5 and Amazon S3

When Zerto 7.0 was released with Long-Term Retention, it was only the beginning of the journey to provide what feels like traditional data protection to meet compliance/regulations for data retention in addition to the 30-day short term journal that Zerto uses for blazing fast recovery.

A few versions later, Zerto (8.5) has expanded that “local repository” to include “remote repositories” in the public cloud. Today it’s Azure blob (hot/cold), and AWS S3 (with support for Standard S3, Standard S3-IA, or Standard One Zone-IA).

And to demonstrate how to do it, I’ve created some content, which includes video and a document that walks you through the process. In the video, I even go as far as running a retention job (backup) to AWS S3, and restoring data from S3 to test the recovery experience.

The published whitepaper can be found here: https://www.zerto.com/page/deploy-configure-zerto-long-term-retention-amazon-s3/

Update: I have just completed testing with S3 Bucket Encryption using Amazon S3 key (SSE-E3), and the solution works without any changes to the IAM policy (https://github.com/gjvtorres/Zerto-LTR-IAM-Policy). There are two methods to encrypt the S3 bucket, with Amazon S3 key as the first option (recommended), and AWS Key Management Service key (SSE-KMS) as the other. I suggest taking a look at the following AWS document that provides pricing examples of both methods. According to what I’ve found, you can cut cost by up to 99% by using the Amazon S3 key. So go ahead, give it a read!

https://aws.amazon.com/kms/pricing/

Now for the fun stuff…

The first option I have is the YouTube video below (or you can watch on my YouTube channel) .

I’ve also started branching out to live streaming of some of the work I’m doing on my Twitch channel.

If you find the information useful, I’d really appreciate a follow on both platforms, and hey, enable the notifications so when I post new content or go live, you can get notified and participate. I’m always working on producing new content, and feedback is definitely helpful to make sure I’m doing something that is beneficial for the community.

So, take a look, and let me know what you think. Please share, because information’s only useful if those who are looking for it are made aware.

Cheers!

Share This:

Single vCenter, Single ZVM, and Recovering Zerto in a Failure Scenario

As a follow-up to my previous blog entry titled “Zerto Virtual Manger Outage, Replication, and Self-Healing“, which covers a ZVM failure scenario in an environment with paired ZVMs and two vCenters, I also decided to test and document what I found to be a useful solution to being able to recover from a failed ZVM in an environment where there is only one vCenter and one instance of Zerto Virtual Replication installed.  Granted, this is generally not a recommended deployment topology due to potentially having a single point of failure, this type of deployment does exist, and this should provide a suitable solution to allow recovery.

The following has been successfully tested in my lab, which is a vSphere environment, but I also do anticipate that this solution can also be carried over to a Hyper-V environment; which I’m hoping to test soon.

Since my lab originally consisted of two vCenters and two ZVMs, I first had to tear it down to become a single vCenter and single ZVM environment for the test.  Here is what I did, should you want to test this on your own before deciding whether or not you want to actually deploy it in your environment.

Disclaimer: 

Once again, this is not generally a recommended configuration, and there are some caveats similar to the referenced blog entry above, but with that said, this will allow you to be able to recover if you have Zerto deployed in your environment as described above.

Considerations

Please note that there may be some things to look out for when using this solution because of how the journal contains data until the checkpoints have been committed to the replica disk:

  • Journal disk being added at the time of a ZVM failure
  • VRA installation, new VRA installation at the time of a ZVM failure
  • Changes made to protected VMs (VMDK add) may not be captured if coinciding with a ZVM failure
  • VPG settings changed at the time of a ZVM failure, such as adding/removing a VM from a VPG

 

Based on additional testing I’ve done, it makes best sense to keep the journal size of the VPG protecting the ZVM as short as possible because any changes that occur to the ZVM (any of the above) will first go to the journal before aging out and being committed to the replica disk.  If those changes don’t commit to the disk, they will not appear in the UI when the ZVM is recovered using this method.

This was found by creating a VPG to protect another set of workloads, and then 10 minutes later, running through the recovery steps for the ZVM.  What I didn’t account for here is the FIFO (first-in-first-out) nature of the journal.  Because the change I had made resided within journal for the protected ZVM, it did not get a chance to age out to disk.  Recovering from the replica did not include the new VPG.

As a result, the recommendation for journal history when protecting the ZVM would be 1 hour (the minimum) – meaning your RPO for the ZVM will be 1 hour.

Setup the Test Environment

Before you can test this, you will need to configure your lab environment for it.  The following assumes your lab consists of two vCenters and 2 installations of Zerto Virtual Replication.  If your lab only has 1 vCenter, simply skip the “lab recovery site” section and move to the “lab protected site” steps.

In lab recovery site:

  1. Delete all existing VPGs
  2. Delete VRAs (via the ZVM UI)
  3. Un-pair the two ZVR sites (in the sites tab in ZVM UI)
  4. Remove hosts from recovery site vCenter

In lab protected site:

  1. [Optional] Create a new cluster, and add the hosts you removed from your recovery site.
  2. Deploy VRAs any hosts you’ll be using in for the test.
  3. Configure VPGs.

Protect the ZVM using Zerto

One thing I’ve wondered about that I finally got around to testing is actually protecting the ZVM itself using ZVR.  I’m happy to say, it appears to work just fine.  After all, Zerto does not make use of agents, snapshots, or disrupt production for that matter, as the technology basically replicates/mirrors block writes from the protected to the recovery site after they’re acknowledged via the virtual replication appliances, not touching the protected workload.

Protecting the ZVM is as simple as protecting any other application, via a VPG (Virtual Protection Group).  While you can likely protect the ZVM via storage snaps and replication, you’re still not going to get an RPO anywhere close to what Zerto itself can provide, which is typically in seconds – many cases single-digit seconds.  What this means, is that your amount of data loss, in the case of the ZVM, will likely be in minutes, even shorter if you can automate the recovery portion of this solution via scripting.

So, a few things to make this solution easier when creating the VPG to protect the ZVM:

  1. When selecting your default recovery server for the VPG that protects the ZVM, select a host, as opposed to a cluster.  This allows you to easily locate the VRA responsible for protecting the ZVM.  Further on through this article, you’ll see why.
  2. Select a specific datastore for recovery.  You can select a datastore cluster, but for the same reasons as above, selecting a specific datastore allows you to easily locate the disk files for the “recovery replica” of the ZVM in the event of a failure.

    Replication Settings - VPG Creation Wizard

  3. Select the production network/portgroup that houses the production IP space for the ZVM (Recovery tab of VPG creation wizard).  We will not be changing the IP address.Recovery Tab - VPG Creation Wizard
  4. Do not change the IP address for failover/move or test (in the NICs tab of the VPG creation wizard).

    NIC Settings - VPG Creation Wizard

Once you’ve created the VPG, allow initial sync to complete.  As you can see below, I now have a VPG containing the ZVM.  Please note that I’m protecting only the ZVM because I am using the embedded SQL CE database.  Using an external SQL server for the ZVR database will require additional planning.  Once initial sync has been completed, you’re ready to begin the actual failure test and recovery.

VPG List - Protecting ZVM

Simulate a Failure of the Primary ZVM

In order to test the recovery, we will need to simulate a failure of the Primary ZVM.

  1. Power off the ZVM.  Optionally, you can also go as far as deleting it from disk.  Now you know there’s no coming back from that scenario.  The ZVM will be gone.

Recover the ZVM Using the Replica

If you remember form the blog post linked at the beginning of this one, even if the ZVM is down, the VRAs are still replicating data.  Knowing that, the VRA in the recovery site (in this case on the recovery host) will have a lock on the VMDK(s) for the ZVM.  That is why I mentioned it would be good to know what host you’re replicating the ZVM to.

  1. IMPORTANT: Before you can start recovering, you will need to shutdown the VRA on the host specified for recovery.  Doing so will ensure that any lock on the VMDK(s) for the replica will be released.
  2. Once the VRA has been shutdown, open the datastore browser and move or copy the VMDK(s) out of the VRA folder to another folder.  By doing this, you’re making sure that if that VRA comes back up before you can delete the VPG protecting the ZVM, there will not be a conflict/lock.  If you select to copy the files, rather than move them, then you can use the existing replica as a pre-seed to re-protect the ZVM.
  3. Create a new VM using the vSphere client.
  4. Select to create a Custom virtual machine.

    Create VM - Custom

  5. Provide a name for the VM that doesn’t already exist in vCenter if you did not delete the original “failed” ZVM.  This ensures there won’t be a naming conflict.
  6. Select the datastore where you copied the replica VMDK(s) to.
  7. Select the Virtual Machine Version.  In this case, you can leave the default, which will be the latest version supported by vSphere version.

    Create VM - vHW Version

  8. Select the OS version for the ZVM.

    Create VM - OS Version

  9. Select the number of vCPUs required. (Match what the original ZVM had)
  10. Select the amount of memory to allocate to the VM. (Match what the original ZVM had)
  11. Select the PortGroup and Adapter type and make sure it’s set to connect at power on.  This should match the original.  My original ZVM had been configured with VMXNET3, so that’s what I selected.
  12. Select the SCSI controller to use.  Again, try to match the original.  Mine was LSI Logic SAS.
  13. On the Select a Disk screen, select Use an existing virtual disk.

    Create VM - Select Existing Disk

  14. Browse to the location of the ZVM replica’s VMDK(s) you copied, and select the disk and click OK.

    Create VM - Select existing disk file

  15. Leave the advanced options at default.
  16. On the summary screen, click Finish.
  17. When the creation is completed, power on the VM, open the console, and watch it boot up.  At this point, DO NOT power on the VRA that you previously shutdown.  There will be some cleanup, especially if you did not copy the VMDK(s) to another location.

Power on new VM created using existing disk.

Clean-up

Once the recovered ZVM has booted up, go ahead and log in to the Zerto UI.  Don’t be alarmed that everything is red.  This is because the ZVM is coming up from being down for a while, and it needs to run some checks, and get re-situated with the VRAs and begin creating new checkpoints again.  Once that process completes, as we saw in the previous blog article (referenced at the beginning of this one), things will start to go green and into a “Meeting SLA” state.

  1. Click on the VPGs tab.
  2. Locate the VPG previously created to protect the ZVM, and delete it.  If you want to retain the original replica disks as a pre-seed, make sure you select the checkbox labeled Keep the recovery disks at the peer site.  Please note that because the VRA that was protecting this VPG is still down, you may need to click delete again, and force the deletion of the VPG.

    Delete VPG - Preserve recovery disks.

  3. Once the VPG is deleted, go ahead and power on the VRA you previously shutdown.

Verify ZVR Functionality

Now that we’ve cleaned up and powered the VRA back up, you can verify that ZVR is working again, and the ZVM is performing its duty of creating and tracking checkpoints in the journal again.  You can do this by starting to initiate a failover test and clicking to see what checkpoints are available, or by attempting to recover a file from the journal from any one of the VPGs.

Validate checkpoint functionality

(Above) you can see when the ZVM went down, and when it started creating and tracking checkpoints again.

Validate JFLR

(Above) Restored a file from the Journal.

Summary

While this is not an optimal/recommended configuration, through testing and validation, we have seen that even in a single ZVM, single vCenter environment, being able to recovery the platform that is providing your resiliency services is completely possible.  Granted, there will be some data loss (RPO) on the ZVM itself, despite being down for time between the failure and the recovery, Zerto Virtual Replication is clearly able to pick up where it left off, and resume protection of your environment.

If you found this to be useful, please share, comment, and let me if you’ve tried this for yourself!

Share This: